Python : Generating a gzip string by returning chunks

Question

I am going to ask for something I could not find on Stackoverflow. I am doing some Django, and I've recently discovered that I can stream the HTTP output using a generator. The output of a page is perfect for a normal case, however I wanted to stream the page output using GZip compression.

I've tried using the simple zlib.compress function, to no avail. The function generates small gzip files.

I want return small chunks of data as they are processed, as a string. Those chunks should form the content of a Gzipped file. How one would do this ? Thanks.

score 2 · Accepted Answer · answered Apr 26 '11 at 22:39

2

use zlib.compressobj([level]) and Compress.compress(string) and Compress.flush([mode]) to finish

import zlib
def compress(chunks):
    c = zlib.compressobj()
    for chunk in chunks:
       yield c.compress(chunk)
    yield c.flush(zlib.Z_FINISH)

answered Apr 26 '11 at 22:39

Dan D.

73,243
15
104
123

Very quick answers as usual on Stackoverflow. Thank you very much for this. The code seems good but does not work, the browser returns an error with the generated page. – Steve K Apr 26 '11 at 23:27
have the proper headers for deflate been set? – Dan D. Apr 26 '11 at 23:29
Hmm I don't know actually. I've set the 'Content-Encoding' to 'gzip' : `response['Content-Encoding'] = "gzip"`. Not setting this header returns garbled data as expected. – Steve K Apr 26 '11 at 23:35
the issue is that this doesn't yield a gzip but a deflate encoding so you'd need to set `response['Content-Encoding'] = "deflate"` – Dan D. Apr 27 '11 at 00:50
It's perfect, Dan. The browser, unfortunately, waits for the output to be complete. [..One minute passes..] `yield c.flush(zlib.Z_SYNC_FLUSH)` when you want to flush the output to the browser solves the problem. Thank you very much. – Steve K Apr 27 '11 at 01:38

score 0 · Answer 2 · edited May 23 '17 at 11:47

0

Compressing the content is a property of your webserver and not the framework. If you are using apache, you can use mod_deflate. I hope you are not referring to the simple server given by the django for testing purposes.If you are referring do that, then simply looking into code for the term gzip and see if it does any sort of compression at all.

Also, if you are thinking of compression in the app server, be warned and it is better to move this job to the web server.

edited May 23 '17 at 11:47

Community

1
1

answered Apr 26 '11 at 22:32

Senthil Kumaran

54,681
14
94
131

Thank you so much for your quick response. I actually use nginx which serves as a proxy to the django code. In this configuration I'm not sure asking nginx to gzip the output would work : I have not tested that (my bad) but it would make sense if it waited for the output to finish, and that is not what I want. – Steve K Apr 26 '11 at 23:19

Python : Generating a gzip string by returning chunks

2 Answers2