5

I am trying to create a lambda function which accepts an image as multipart/form-data, do some processing on the image and upload it to s3 and also returns reponse to the client. But i am stuck at the very first part of uploading the image to aws lambda using API gateway. I tried to do this in NodeJS as shown below:

exports.handler = async (event, context, callback) => {
  var buf = Buffer.from(event.body.replace(/^data:image\/\w+;base64,/, ""),"base64");

  var data = {
    Bucket: "bucket-name", 
    Key: "abc.jpg", 
    Body: buf,
    ContentType: 'image/jpg',
    ACL: 'public-read'
  };
  data = await s3.upload(data).promise();
  return {
         statusCode: 200,
         body: JSON.stringify(buf),
     };

I am getting the following response in Postman by making POST request to the api:

{
    "ETag": "\"b0e5b18d38904f109e0aef0b29e132be\"",
    "Location": "https://bucket-name.s3.us-east-2.amazonaws.com/abc.jpg",
    "key": "abc.jpg",
    "Key": "abc.jpg",
    "Bucket": "bucket-name"
}

But when i try to view the uploaded image in my browser using public url in the returned in the above response, i am getting empty image.

Can somebody point me the mistake here or suggest some better approach. Thank you.

flamelite
  • 2,654
  • 3
  • 22
  • 42
  • For debug purpose, can you log event in console and see if you are getting payload properly from API Gateway. – Imran Dec 29 '18 at 16:36
  • Yes, i am getting the required payload. – flamelite Dec 29 '18 at 16:41
  • 1
    Most likely s3 upload was interrupted since you already returned response back to API Gateway and lambda execution was done. Check this question and answers. https://stackoverflow.com/q/31633912/5030709. Try moving `context.done()` into callback function of s3 upload and see if upload is success. – Imran Dec 30 '18 at 03:08
  • 1
    This is not a recommended pattern to upload files to S3 since API Gateway has a payload limit and it also increases cost. A better approach would be to use S3 Signed URLs to upload directly from Browser client. – Ashan Dec 31 '18 at 04:54
  • @Ashan Thanks for the suggesting this aproach. But After uploading the file i would have to make another HTTP call to my lambda function, don't you think this will increase the processing time? I want to make my application work in real time. – flamelite Dec 31 '18 at 05:17
  • You could possible use S3 Lambda trigger to asynchronous do any further actions. This approach will happen near realtime and would be the fastest. However, it also depends on your usecase. If above approach is not possible and performance is a consideration, you can also implement an optimistic strategy to do these in parallel (upload directly to S3 and API call) and have a fallback strategy if upload fails for some reason which is a rare case. – Ashan Dec 31 '18 at 05:27
  • Yes i need to make a single api which handles th file upload, process the image and return the output. Can you suggest me better approach? – flamelite Dec 31 '18 at 05:28
  • Use AWS CloudFront and connect both S3 and API Gateway. This will allow you tocuse the same URL domain to access S3 and API Gateway. Directly upload to S3 via CloudFront (Use EdgeLambda to validate any tokens and update any other data). – Ashan Dec 31 '18 at 05:34
  • What is the size of the file you are uploading and the size of the file actually stored in S3? – K Mo Jan 02 '19 at 09:09
  • file size varies from 100 kb to 300 kb – flamelite Jan 02 '19 at 09:28
  • @flamelite so the same file varies in size on S3 when it is uploaded at different times? – K Mo Jan 02 '19 at 09:30
  • That i did not verify if file size remains exactly same. – flamelite Jan 02 '19 at 09:37
  • @flamelite File size difference can help point to certain issues, like base64 conversion or even no file being uploaded. Also, can you confirm the last 5 lines of code are correct, as that will not produce the response you have. – K Mo Jan 02 '19 at 10:00
  • @flamelite this [question](https://stackoverflow.com/q/53998940/5030709) has working javascript code to upload to S3 if you still want to try lambda approach. – Imran Jan 02 '19 at 22:17
  • @Imran thanks for posting the link here. When using that code i am getting an error which is occuring because of asynchronous calling of s3.upload() method. Do i have to make that synchronous as i have posted in my question? – flamelite Jan 04 '19 at 05:49
  • @flamelite you cannot. SDK `s3.upload()` is an async method and whatever you want to perform after object is uploaded has to be put in inside callback. In the example, the author is returning an API response saying file is uploaded successfully. If you make is sync then you will get back to same error as your question. – Imran Jan 04 '19 at 22:01
  • @imran could you please provide me a link some resource which explains file upload to aws lambda ? I am confused with alot of points. – flamelite Jan 06 '19 at 13:37
  • @flamelite AWS JavaScript SDK methods are [async](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/calling-services-asynchronously.html) only. – Imran Jan 06 '19 at 13:44
  • @imran i am looking for a complete tutorial on how to upload a file to aws lambda using post request from POSTMAN which describes how to parse the request data, reads the file data and uploads the file to s3. – flamelite Jan 06 '19 at 13:48
  • @flamelite Tbh, I don't know anyone created tutorial with that. The question I have shared is doing that exactly same thing. I have tested it personally and it works fine. Just take that lambda function and expose it via Api Gateway. In your JSON request to API, make the payload to something like `{"user_avatar":"<>"}`. It should work as I stated to author of question and he was able to work it as well. – Imran Jan 06 '19 at 13:56
  • If i am using aws lambda proxy in the api gateway the where and how should i do this "In your JSON request to API, make the payload to something like {"user_avatar":"<>"}", or how do i add "<>" in the postman? – flamelite Jan 06 '19 at 13:59
  • @flamelite give me few hours or a day. I will create a github repo reference with [serverless](https://serverless.com/framework/docs/providers/aws/examples/hello-world/node/) utility to deploy lambda/api to upload an image. Again, it doesn't matter if you are using lambda proxy integration or not. The author is also using lambda proxy integration only to upload the image. – Imran Jan 06 '19 at 14:03
  • @Imran Thanks in advance that would be a big help. – flamelite Jan 06 '19 at 14:04
  • @imran also if can share your contact details i can share the error screenshot with you. – flamelite Jan 06 '19 at 14:09
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/186260/discussion-between-imran-and-flamelite). – Imran Jan 06 '19 at 14:39

1 Answers1

6

As suggested by Ashan, you can go with best practice of uploading images via browser.

If the image size is not that large, here is working example of uploading image to S3 via ApiGateway/Lambda Proxy Integration.

Lambda Function Code - 

exports.handler = (event, context, callback) => {
  let encodedImage = JSON.parse(event.body).user_avatar;
  let decodedImage = Buffer.from(encodedImage, 'base64');
  var filePath = "avatars/" + event.queryStringParameters.username + ".png"

  var params = {
    "Body": decodedImage,
    "Bucket": "bucketName",
    "Key": filePath,
    "ContentType " : "mime/png"
  };
  s3.upload(params, function (err, data) {
    if (err) {
      callback(err, null);
    } else {
      let response = {
        "statusCode": 200,
        "body": JSON.stringify(data),
        "isBase64Encoded": false
      };
      callback(null, response);
    }
  });
};

Serverless.yml

service: aws-api-lambda-s3-image-upload

frameworkVersion: ">=1.1.0 <2.0.0"

provider:
  name: aws
  runtime: nodejs8.10
  iamRoleStatements:
    -  Effect: "Allow"
       Action:
         - "s3:ListBucket"
       Resource: "arn:aws:s3:::bucketName"
    -  Effect: "Allow"
       Action:
         - "s3:PutObject"
       Resource: "arn:aws:s3:::bucketName/*"

functions:
  index:
    handler: handler.handler
    events:
      - http: POST handler

Json Payload Should be - 

{
  "user_avatar" : "<<base64 encoded image>>"
}
Imran
  • 5,542
  • 3
  • 23
  • 46