How to compress/minimize size of JSON/Jsonify with Flask in Python?

Question

I'm sending a huge JSON string (with jsonify in Flask) to my webpage pretty often, so I would like to reduce the data. The easiest option is probably to remove all line breaks and space characters, but just to give you an example of this:

Normal jsonify: 361KB
Removing all line breaks and space characters: 118KB (wow).
Zip the original file: 35KB (double wow).

So I basically wonder if there is an easy way to come close to the 35KB. I couldn't find a solution so far which I could easily implement in python and javascript (to decompress).

Right now, I send around 4-5MB of data every second, which is - you guessed right - a "little bit" too much.

Answer 1

Old question but I was searching for this and it was the first result on Google. The link to the answer of Leon has a solution not for Flask and also it is old. With Python 3 now we can do all in few lines with the standard libraries (and Flask):

from flask import make_response, json
import gzip

@app.route('/data.json')
def compress():
    very_long_content = [{'a': 1, 'b': 2}, {'c': 3, 'd': 4}]
    content = gzip.compress(json.dumps(very_long_content).encode('utf8'), 5)
    response = make_response(content)
    response.headers['Content-length'] = len(content)
    response.headers['Content-Encoding'] = 'gzip'
    return response

With gzip.compress we have directly a byte string compressed and it is required as input a byte string. Then, as the link from Leon, we make a custom response saying that the content is a gzip so the browser will decompress by itself.

For decoding in Javascript using a JQuery ajax request there isn't any particular difference from a standard request:

$.ajax({
    url: '/data.json',
    dataType: 'json',
    success: function(data) {
        console.log(data);
    }
})

Note that this snippet compress and then send the long content. You should consider the amount of time that it takes to compress the content (especially in this case that we have very long content), so be sure to set an appropriate level of compression that doesn't require more time to compress + send than send the long content as it is.

My use case was that I sent the big content from a slow connection, so I had all the benefits to compress the content before send it.

Answer 2

Web requests do support GZip and you could implement it in python.

Here is someone who asked that exact question. How to use Content-Encoding: gzip with Python SimpleHTTPServer

According to the flask-compress repo

The preferred solution is to have a server (like Nginx) automatically compress the static files for you.

But you can do it in flask: https://github.com/colour-science/flask-compress.

If you go the gzip route you will not need to remove line breaks and white space, but if you still want to then according to the flask documentation you can disable pretty print by setting JSONIFY_PRETTYPRINT_REGULAR to false.

Answer 3

If you're looking for a library to help serve compressed content via Gzip (or Brotli), try flask-compress. It's very simple; this might be all you need to do:

from flask import Flask
from flask_compress import Compress

app = Flask(__name__)
Compress(app)

Answer 4

Inspired by Ripper346's answer, but rather than manually writing it for all routes/views. It is just better to add a snippet(compared to external dependency on a library), which will compress the response before sending it out.

Add the following to your app file, or wherever suited(eg. gzip_compress.py):

import gzip
from io import BytesIO
from flask import request

class GzipCompress:
    def __init__(self, app, compress_level=9, minimum_size=100):
        self.app = app
        self.compress_level = compress_level
        self.minimum_size = minimum_size
        self.app.after_request(self.after_request)

    def after_request(self, response):
        accept_encoding = request.headers.get('Accept-Encoding', '')

        if response.status_code < 200 or \
           response.status_code >= 300 or \
           response.direct_passthrough or \
           len(response.get_data()) < self.minimum_size or \
           'gzip' not in accept_encoding.lower() or \
           'Content-Encoding' in response.headers:
            return response

        gzip_buffer = BytesIO()
        gzip_file = gzip.GzipFile(mode='wb', 
                                  compresslevel=self.compress_level, 
                                  fileobj=gzip_buffer)
        gzip_file.write(response.get_data())
        gzip_file.close()
        response.set_data(gzip_buffer.getvalue())
        response.headers['Content-Encoding'] = 'gzip'
        response.headers['Content-Length'] = len(response.get_data())

        return response

And then in your app:

from flask import Flask
from gzip_compress import GzipCompress

app = Flask(__name__)
GzipCompress(app)

You can set your gzip compression level and the minimum response size for compression to kick in. It includes other checks for gzip support and response codes.

Answer 5

Spinoff of suvigyavijay's answer, without using awkward classes, io.BytesIO(), or gzip.GzipFile (he did a lot of heavy lifting though, props).

Note. This applies to all requests, I think.

import flask, gzip

app = flask.Flask(__name__)

@app.after_request
def compress(response):
  accept_encoding = flask.request.headers.get('accept-encoding','').lower()
  if response.status_code<200 or response.status_code>=300 or response.direct_passthrough or 'gzip' not in accept_encoding or 'Content-Encoding' in response.headers:  return response
  content = gzip.compress(response.get_data(), compresslevel=9)  # 0: no compression, 1: fastest, 9: slowest. Default: 9
  response.set_data(content)
  response.headers['content-length']   = len(content)
  response.headers['content-encoding'] = 'gzip'
  return response