AWS Lambda Python - Return BytesIO file?

Question

I'm setting up a function in AWS Lambda using python 3.7 and it won't let me return a bytes type

Please notice that this is not an issue with API Gateway, I'm invoking the lambda directly.

The error is : Runtime.MarshalError, ... is not JSON serializable

output = BytesIO()
#Code that puts an excel file into output...
return {
        'Content-Disposition': 'attachment; filename="export.xlsx"',
        'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
        'body' : output.getvalue()
    }

If I do :

'body' : str(output.getvalue())

It outputs a corrupted file because it adds b'' to the string

If I do :

'body' : base64.b64encode(output.getvalue()).decode()

It also outputs a corrupted file, probably because it changes the binary representation of the file.

Maybe I need to upload to S3? But it doesn't fit in my flow, this is a one time file creation and it would stay in "S3 Limbo" until TTL

`output.getvalue().decode('utf-8')` will resolve the 'not JSON serializable' issue. Would that help? — Martin Valgur, Jul 24 '19 at 12:55
@MartinValgur it doesn't, it's a binary bytes stream, it gives 'utf-8' codec can't decode byte 0xda in position 16: invalid continuation byte — Mojimi, Jul 24 '19 at 12:56
Possible duplicate of [How to return binary data from lambda function in AWS in Python?](https://stackoverflow.com/questions/44860486/how-to-return-binary-data-from-lambda-function-in-aws-in-python) — Martin Valgur, Jul 24 '19 at 13:04
@MartinValgur it's not, that question is related to api gateway, I already mentioned I'm not using it — Mojimi, Jul 24 '19 at 13:10

Erik · Answer 1 · 2020-08-10T04:11:53.610

2

It is not possible to return unencoded binary data from a direct invoked AWS Lambda function.

Per the docs:

If the handler returns objects that can't be serialized by json.dumps, the runtime returns an error.

The reason you can do this with API Gateway is because API Gateway is performing the conversion of the base64 JSON content your function returns into binary for you. (See documentation here)

I would need to know more about how you are invoking Lambda to be sure but I suspect you could implement this same base64 decode logic into your direct invoke client. Alternatively, if you wanted to keep the client as simple as possible, use S3 with a lifecycle hook to keep the bucket from filling up with temporary files.

edited Aug 10 '20 at 04:11

answered Jan 12 '20 at 23:30

Erik

138
7

1

Downvotes are hilarious. Sorry, but you can’t return a unencoded binary stream from a direct invoked lambda no mater how much you might dislike it. – Erik Aug 10 '20 at 04:08
Is this answer out of date at this point? Your link now points to a page titled 'Enabling binary support via API gateway' – maxwell Nov 12 '20 at 18:01
Oh after reading further, it appears that it is only possible if you passthrough to an S3 bucket rather than using a lambda as an integration type. – maxwell Nov 12 '20 at 18:37

AWS Lambda Python - Return BytesIO file?

1 Answers1