Is it possible to use gzip compression with Server-Sent Events (SSE)?

Question

I would like to know if it is possible to enable gzip compression for Server-Sent Events (SSE ; Content-Type: text/event-stream).

It seems it is possible, according to this book: http://chimera.labs.oreilly.com/books/1230000000545/ch16.html

But I can't find any example of SSE with gzip compression. I tried to send gzipped messages with the response header field Content-Encoding set to "gzip" without success.

For experimenting around SSE, I am testing a small web application made in Python with the bottle framework + gevent ; I am just running the bottle WSGI server:

@bottle.get('/data_stream')
def stream_data():
    bottle.response.content_type = "text/event-stream"
    bottle.response.add_header("Connection", "keep-alive")
    bottle.response.add_header("Cache-Control", "no-cache")
    bottle.response.add_header("Content-Encoding", "gzip")
    while True:
        # new_data is a gevent AsyncResult object,
        # .get() just returns a data string when new
        # data is available
        data = new_data.get()
        yield zlib.compress("data: %s\n\n" % data)
        #yield "data: %s\n\n" % data

The code without compression (last line, commented) and without gzip content-encoding header field works like a charm.

EDIT: thanks to the reply and to this other question: Python: Creating a streaming gzip'd file-like?, I managed to solve the problem:

@bottle.route("/stream")
def stream_data():
    compressed_stream = zlib.compressobj()
    bottle.response.content_type = "text/event-stream"
    bottle.response.add_header("Connection", "keep-alive")
    bottle.response.add_header("Cache-Control", "no-cache, must-revalidate")
    bottle.response.add_header("Content-Encoding", "deflate")
    bottle.response.add_header("Transfer-Encoding", "chunked")
    while True:
        data = new_data.get()
        yield compressed_stream.compress("data: %s\n\n" % data)
        yield compressed_stream.flush(zlib.Z_SYNC_FLUSH)

Yes it's possible; browsers support that; the framework needs to be clever though -- you want the compression code to flush the chunk after each (?) event (or after a short time) to ensure timely event delivery. — Dima Tisnek, Feb 09 '22 at 02:01

score 7 · Accepted Answer · answered May 24 '14 at 11:45

7

TL;DR: If the requests are not cached, you likely want to use zlib and declare Content-Encoding to be 'deflate'. That change alone should make your code work.

If you declare Content-Encoding to be gzip, you need to actually use gzip. They are based on the the same compression algorithm, but gzip has some extra framing. This works, for example:

import gzip
import StringIO
from bottle import response, route
@route('/')
def get_data():
    response.add_header("Content-Encoding", "gzip")
    s = StringIO.StringIO()
    with gzip.GzipFile(fileobj=s, mode='w') as f:
        f.write('Hello World')
    return s.getvalue()

That only really makes sense if you use an actual file as a cache, though.

answered May 24 '14 at 11:45

otus

5,572
1
34
48

Thanks for your explanation. Indeed, changing the Content-Encoding to deflate helps a bit: the first message is processed on the client side. But only the first :( Do you have any idea why? Thanks in advance – mguijarr May 24 '14 at 22:00
Are you trying to call compress independently for each data block? I don't think that will work. All the data should be in a single compressed stream. That means gzip with its streaming interface may actually be the way to go. However, I would need to see more of your code to offer specific pointers. – otus May 25 '14 at 06:54
Thanks a million ; finally it works! I edited my question to tell what I had to change. – mguijarr May 25 '14 at 08:39
NP. I forgot zlib objects can flush, that's a very useful pattern to remember. – otus May 25 '14 at 09:04
How would this work if not using bottle? using Flask-SSE for example. Can't seem to get it to actually compress the stream data. – Source Matters Mar 14 '23 at 19:53

score 2 · Answer 2 · answered Jun 06 '15 at 05:49

There's also middleware you can use so you don't need to worry about gzipping responses for each of your methods. Here's one I used recently.

https://code.google.com/p/ibkon-wsgi-gzip-middleware/

This is how I used it (I'm using bottle.py with the gevent server)

from gzip_middleware import Gzipper
import bottle
app = Gzipper(bottle.app())
run(app = app, host='0.0.0.0', port=8080, server='gevent')

For this particular library, you can set w/c types of responses you want to compress by modifying the DEFAULT_COMPRESSABLES variable for example

DEFAULT_COMPRESSABLES = set(['text/plain', 'text/html', 'text/css',
'application/json', 'application/x-javascript', 'text/xml',
'application/xml', 'application/xml+rss', 'text/javascript',     
'image/gif'])

All responses go through the middleware and get gzipped without modifying your existing code. By default, it compresses responses whose content-type belongs to DEFAULT_COMPRESSABLES and whose content-length is greater than 200 characters.

Is it possible to use gzip compression with Server-Sent Events (SSE)?

2 Answers2

Linked