How to process binary data from API Gateway in Lambda and vice-versa?

Question

I'm currently have the following setup: 1. S3 Bucket 2. API Gateway with GET/POST resource 3. Two Lambda Functions (one for getting data from s3, one for storing data to s3)

For example the GET Resource, i pass a key parameter, which is used in the lambda function to get the object from s3 bucket. then i want to return the retrieved object to api gateway and to client as a binary.

In the POST resource, i want to send a binary payload, eg.: pdf or zip file, save it to s3 and return generated key.

So on one hand i want my api gateway and lambda to be able return binary data in GET request, on the other hand i want it to accept binary payload in POST request.

In api gateway settings, i've set Binary-Media-Types to application/octet-stream. Also i have lambda proxy integration activated.

My question is: How can I process binary data in api gateway/lambda?

I tried messing around with Headers and Content-Types but more or less i didn't know what I was doing :(

Example store-luggage lambda for storing data from POST request in S3

import boto3
import json
import uuid
import logging
from botocore.exceptions import ClientError

def lambda_handler(event, context):

    #data = <binary data from POST request>?

    logger = logging.getLogger()
    logger.setLevel(logging.INFO)

    bucket = 'bucket1'

    uniqueid = str(uuid.uuid4())
    logger.info("Generated UUID: " + uniqueid)

    s3 = boto3.resource('s3')

    logger.info("Trying to save file '" + uniqueid + "' to bucket '" + bucket + "'")

    savedObj = None

    try:
        savedObj = s3.Bucket(bucket).put_object(Key=uniqueid, Body=<here should be my binary payload from POST request>)
    except ClientError as e:
        logger.error("Saving of object has failed: " + str(e.response['Error']['Message']))

    if savedObj is None:
        return {
            'statusCode': 500,
            'body': json.dumps({
                'message': 'Saving of object has failed!'
            })
        }

    return {
        'statusCode': 200,
        'body': json.dumps({
            'token': uniqueid
        })
    }

get-luggage lambda for retrieving object from s3 by given key and return binary payload in response

import boto3
import json
from pprint import pprint
from botocore.exceptions import ClientError
import logging

def lambda_handler(event, context):

    logger = logging.getLogger()
    logger.setLevel(logging.INFO)

    bucket = 'bucket1'
    token = event["queryStringParameters"]["token"]

    s3 = boto3.client('s3')

    objectFromS3 = None

    try:
        objectFromS3 = s3.get_object(Bucket=bucket, Key=token)
    except ClientError as ex:
        if ex.response['Error']['Code'] == 'NoSuchKey':
            logger.info('No object found - returning empty')
        else:
            raise ex

    if objectFromS3 is None:
        return {
            'statusCode': 404,
            'body': json.dumps({
                'message': 'Cannot retrieve object from storage!'
            })
        }

    pprint(objectFromS3)
    #ContentType application/octet-stream

    return {
        'statusCode': 200,
        'body' : <binary data from retrieved s3 object here?>,
        'headers': {
            'content-type': 'application/octet-stream'
        },
        'isBase64Encoded': True
    }

I expected to retrieve binary payload in GET response and expected api gateway to pass binary payload from POST request to lambda, but none of this was achieved with the stuff i tried.

I hope i descriped my problem well enough, if any clearification is needed let me know.

best wishes, an aws noob

score 2 · Accepted Answer · answered May 27 '19 at 10:15

First, you need to configure the MIME types that you expect to treat as binary data in API gateway settings.

Settings -> Binary Media Types.

Specify the exact MIME type to use as binary data application/pdf, application/zip etc. application/octet-stream is representing generic data which exact type is unknown.

In your Http request, there should be Accept header indicating which MIME type includes in the payload.

Accept: application/pdf
Accept: application/zip

Since the AWS Lambda process requests and responses as base64 encoded, you should decode POST request body and get original binary content before uploaded into S3 bucket.

After fetching data from S3 bucket lambda encodes binary data before sending it to API gateway. In response, you should set the value of the isBase64Encoded flag to True. Then API gateway performs Base64 decoding on encoded data before sending it to the client if Http request includes corresponding Accept header.

score 1 · Answer 2 · answered May 27 '19 at 23:40

get-luggage:

If your file is already in S3, you can redirect the user to the file link in S3 by adding the Location attribute on the API-Gateway headers, reducing Gateway data traffic.

example:

exports.handler = (event, context, callback) => {
  return callback(null, {
    statusCode: 301,
    headers: {
      Location: 'https://<...S3...>',
    }
  });
}

score 0 · Answer 3 · answered Apr 08 '20 at 17:41

Submitting "Accept: application/pdf" at Method Request > HTTP Request Headers is not possible as it returns the error, "Invalid patch path /requestParameters/method.request.header.Accept: application/pdf."

For myself, I have the following Lambda:

import boto3
import base64

def lambda_handler(event, context):
    s3 = boto3.client("s3")

    fileObj = s3.get_object(Bucket="mytestbucket", Key="sample.pdf")
    file_content = fileObj["Body"].read()

    return {
        "statusCode": 200,
        "headers": {"Content-Type":"application/pdf"},
        "body": base64.b64encode(file_content),
        "isBase64Encoded": True
        }

But the response body is base64 encoded rather than the sample.pdf file itself. Yes, I set */* as a binary media type at Settings in the API Gateway console. Any suggestion what I may be missing?

How to process binary data from API Gateway in Lambda and vice-versa?

3 Answers3