How to return a stream from AWS ApiGateway?

Question

Can someone tell me how to return a stream from an AWS Api Gateway? My API calls a lambda written in C#, which successfully returns a string, but I don't know how to modify it to return a stream. I need this because it will be used to download a zipped file that is 100MB.

This is a simplified version of my current lambda handler.

    public APIGatewayHttpApiV2ProxyResponse MainHandler(APIGatewayProxyRequest request, ILambdaContext context)
    {
        ...
        var response = new APIGatewayHttpApiV2ProxyResponse
        {
            StatusCode = (int)HttpStatusCode.OK,
            Body = "MainHandler was called",
            Headers = new Dictionary<string, string> { { "Content-Type", "text/plain" } }
        };

        return response;
    }

I don't know how to modify this to return a stream, instead of text.

I have found examples of how to consume a stream using HttpClient:

c#: How to Post async request and get stream with httpclient?

but no examples of how to create one from AWS.

score 2 · Answer 1 · edited Jan 11 '23 at 12:07

2

You would hit the API Gateway payload limit of 10MB regardless of your return type with 100MB.

Your best option would be to save the data you want to return to S3, generate a pre-signed URL and then return the URL via your Lambda to the client.

Amazon Lambda functions and the API Gateway are not for returning large files.

edited Jan 11 '23 at 12:07

Ermiya Eskandary

15,323
3
31
44

answered Jan 10 '23 at 16:56

rleffler

430
6
15

Surely not? If I am streaming, the lambda will not load up the full file contents? Using an S3 link would require the client to authenticate with AWS IAM, which it currently doesn't need to do and would be quite awkward. – Ant Waters Jan 11 '23 at 09:23
This answer is correct, as you are returning more than the 6MB synchronous Lambda invocation payload limit and the 10MB API Gateway limit. Use S3 [pre-signed URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html) instead to stream files. Store your file in S3, generate a link and then return that to the client. – Ermiya Eskandary Jan 11 '23 at 12:05

score 0 · Accepted Answer · answered Jan 11 '23 at 14:51

It's important to understand that both the API gateway and lambda functions have limitations and they're not designed to deal with long open HTTP connections. API gateway has a hard response size limit of 10MB, while the lambda function has a hard limit of 6MB(sync) and 256kb(async), please refer to the below docs for more info

https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

I quote two solutions from the following answer API Gateway stream large size content in response

Have your EC2 machine upload the result to S3 and have API Gateway return a pre-signed url to download the response from S3. This would stream the download, but would have to wait for the EC2 -> S3 upload to complete first.
Use Elastic Beanstalk, that way you would be in control of the server and able to keep your connections open for as long as you wanted, and send as much data as you want.

How to return a stream from AWS ApiGateway?

2 Answers2