0

I'm using Flask in AWS Api Gateway/Lambda environment (Thanks to Zappa), but there is a limit in response size, so Flask's send_file is not enough in this context.

Is there a way I can stream/multipart(not sure if these are the correct terms) a file-like object as response in Flask? I can't send request bodies with more than 5mb(6mb?) in the AWS Serverless environment.

Current code (simple S3 proxy that deletes the object once downloaded):

@app.route('/polling/<key>')
def polling(key):
    obj = BytesIO()
    try:
        s3.download_fileobj('carusoapi', key, obj)
        s3.delete_object(Bucket='carusoapi', Key=key)
        return send_file(obj, as_attachment=True, attachment_filename=key)
    except Exception:
        return 'File not ready yet', 204

I've seen some examples here but don't understand how to apply them or if that's even what I'm looking for.

I also noticed that boto3 S3 module has options like callback for download_fileobj here and you can specify chunksize here, but again, I don't understand how to apply this to a Flask response.

I know of a way to solve this that involves sending a signed download link to the client to download the item, but then I would have to implement in the client to delete the file.

Mojimi
  • 2,561
  • 9
  • 52
  • 116
  • 1
    Have S3 serve the data *directly* to the client, don't stream it through Flask, with a presigned URL. See the duplicate, the [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html#generating-presigned-urls) and perhaps the [Flask-S3 project](https://flask-s3.readthedocs.io/en/latest/), which automates this for you via a `url_for()` wrapper. – Martijn Pieters Mar 13 '19 at 13:54
  • 1
    @MartijnPieters I mentioned I'm aware of this solution in my last paragraph – Mojimi Mar 13 '19 at 13:58
  • Why would you need to have the client delete the file when it's a S3 link? Just have a batch process clean up S3 buckets. – Martijn Pieters Mar 13 '19 at 14:00
  • Better yet, [set a lifetime cycle](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html), and have Amazon take care of cleaning up. – Martijn Pieters Mar 13 '19 at 14:01
  • @MartijnPieters Already set that up, the minimum is 24H, in my case the file will be in "limbo" for 1 day after the client received and downloaded it, so I just wanted to know if I could delete it just after the client downloads it, just an ideal scenario. – Mojimi Mar 13 '19 at 14:03

0 Answers0