API Gateway OpenAPI definition POST json file as binary

Question

In AWS, I'm using API Gateway backed by lambda function to upload a JSON file and do some small checks in lambda and then dump it to S3. I'm getting stuck where the JSON file upload is done fine but when I look the file in the lambda logs I see that JSON file is converted into huge string. For example, I'm making the following request

curl --location 'https://[API_ID].execute-api.us-east-1.amazonaws.com/v1/mappings/data.json' \
--header 'Content-Type: application/octet-stream' \
--data '@./data.json'

Now, when I see the lambda function logs I see the following:

Following is my OpenAPI specs in API Gateway:

        paths:
          /mappings/{filename}:
            post:
              requestBody:
                content:
                  application/octet-stream:
                    schema:
                        format: binary
                        type: string
              parameters:
              - in: path
                name: filename
                required: true
                schema:
                  type: string
              x-amazon-apigateway-integration:
                httpMethod: POST
                uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${MyLambda}/invocations
                requestParameters:
                  integration.request.path.filename: method.request.path.filename
                requestTemplates:
                  application/json: $input.body
                passthroughBehavior: "when_no_match"
                type: "aws_proxy"

I'm not sure what I'm doing wrong here.

UPDATED:

My Python lambda function

def lambda_handler(event, context):
    print(event)

    s3_client = boto3.client('s3')
    s3_client.put_object(Body=event["body"], Bucket=BUCKET, Key=event['pathParameters']['filename'])

This is what file looks like in S3

tom redfern · Answer 1 · 2023-03-17T12:39:24.310

0

First of all, you won't see JSON in the log because what you are passing into the gateway is a text file, which is a binary object. In fact you are actually telling the gateway to expect a binary object with this request header:

Content-Type: application/octet-stream

If you want JSON to be passed instead then you need to change the Content-Type header value to application/json, and then pass the actual JSON content from the file in, not the file itself.

edited Mar 17 '23 at 12:39

answered Mar 17 '23 at 12:15

tom redfern

30,562
14
91
126

actually the file is copied correctly into S3 bucket but with that long text which I showed in the image. Also adding the image of S3 bucket and lambda in the original post – muazfaiz Mar 17 '23 at 12:23
@muazfaiz so what is the problem you are trying to solve?? – tom redfern Mar 17 '23 at 12:33
the JSON file is not the same which I'm uploading. API gateway or lambda converts it into a long string – muazfaiz Mar 17 '23 at 12:40
thanks. Yes your recommendation can be done quite easily but I was wondering what is causing the issue with uploading the JSON file? – muazfaiz Mar 17 '23 at 13:06
@muazfaiz I don't know what s3_client.put_object(...) does, but it seems not to be working as you expect. It is taking the body of the request, which is a binary file, converting it to a string, and then wrapping it in JSON braces and then writing to S3. – tom redfern Mar 17 '23 at 13:16

score 0 · Accepted Answer · answered Mar 26 '23 at 09:25

The body was base64 encoded so that is why it seems like the body is not JSON but a long string. I just converted it back and it worked fine using the following code

import base64
 
base64_bytes = event["body"].encode('ascii')
message_bytes = base64.b64decode(base64_bytes)
message = json.loads(message_bytes.decode('ascii'))

print(message) # You'll get correct json object as you sent in the body

API Gateway OpenAPI definition POST json file as binary

2 Answers2