0

I am writing a Flask API endpoint that takes s3 URL as a parameter and sends the s3 object as response.

pseudoCode:

def stream_s3():
    while remaining_bytes>0:
       # ... logic to choose right byte range
       actual_bytes = aws.get_object(bucket, key, Range)
       yield Response(actual_bytes)

The content type is JSON, but some JSON can be more than 200 MB.

Question 1: How do I compress the bytes being sent? so that I can make it faster to receive the response.

Question 2: Some solutions suggest that I gzip the file and set the 'content-Encoding' as gzip. This basically needs the complete file to be downloaded in the server before gziping right?

petezurich
  • 9,280
  • 9
  • 43
  • 57
Viswesh M
  • 1
  • 1
  • 1) Let your frontend web server/reverse proxy such as nginx deal with that. 2) No. The `zlib` module has streaming interfaces. – AKX Oct 27 '21 at 16:43
  • Thanks, if I use the Zlib to compress the byte stream, whether the client will be able to decompress it? client denotes chrome browser, postman, simple curl request, python request – Viswesh M Oct 27 '21 at 16:52
  • Your server would need to look at the Accept-Encoding header. – AKX Oct 27 '21 at 17:33
  • Hi, tried zlib, the client was unable to decompress with the setting of 'Accept-Encoding' – Viswesh M Oct 27 '21 at 20:26
  • If the client advertises it can do deflate, you'll need to declare `content-encoding: deflate` in the response and and use a negative `wbits` for your `zlib.compressobj()` so it doesn't include zlib headers (as documented but which isn't quite obvious). – AKX Oct 27 '21 at 20:28
  • Hey, I tried with https://pypi.org/project/Flask-Compress/ library, it works. It compresses. The problem is that it iterates the whole response in the server to compress. So basically compressing a file instead of a stream of bytes. This cogs up the server memory. For the above comment, chrome is unable to deflate. Any thoughts? – Viswesh M Oct 28 '21 at 04:45
  • Show the code you tried with `zlib`, please. "Tried zlib" is not quite enough. – AKX Oct 28 '21 at 13:02
  • return zlib.compress(response.get_data(), -1) – Viswesh M Oct 31 '21 at 02:23
  • That is also just buffering the data in memory and compressing it... – AKX Oct 31 '21 at 08:25

0 Answers0