Streaming responses

[lbam]

In this line...

django-vectortiles/vectortiles/postgis/mixins.py

Line 58 in 86f0087

return row.tobytes() if row else b""

the entire MVT response from PostGIS is converted to a bytes object. That means it has to be in memory twice, once as a memoryview and once as a bytes. I'm wondering if this is necessary, since Django performs the same conversion. If the conversion were not there, a caller could intercept the memoryview and make a StreamingHttpResponse out of it, so that it only needs to be in memory once.

[StefanBrand]

Good point, it looks as if the tobytes call could possibly be removed. I quickly ran the tests with this change and they run fine. Are you interested in contributing a PR?

[lbam]

I could discuss a PR with my employer, but I'd first need to have #36 resolved.

[lbam]

I've also done some more research and found that this is not the bottleneck, memory-wise. The tile is sent over the wire by PostgreSQL as ASCII hex. psycopg reads the entire tile into memory in that format, then decodes it into a separate buffer, so at that point the memory use is 3x the size of the tile. Fixing this issue would therefore not reduce the peak memory use. For that, the ST_AsMVT query would somehow have to chunk the resulting bytea in PostgreSQL so this library can receive it piecemeal and sent the chunks to the client in a streaming response.

[submarcos]

I could discuss a PR with my employer, but I'd first need to have #36 resolved.

This point is resolved. I'm working on a 1.0 refactor and release. If you contribute a PR I can implement it in 0.2.1 and 1.0 versions