0

I have an API written in nodeJS (/api/uploadS3) which is a PUT request and accepts a video file and a URL (AWS s3 URL in this case). Once called its task is to upload the file on the s3 URL.

Now, users are uploading files to this node API in different formats (thanks to the different browsers recording videos in different formats) and I want to convert all these videos to mp4 and then store them in s3.

I wanted to know what is the best approach to do this?

I have 2 solutions till now

1. Convert on node server using ffmpeg -

The issue with this is that ffmpeg can only execute a single operation at a time. And since I have only one server I will have to implement a queue for multiple requests which can lead to longer waiting times for users who are at the end of the queue. Apart from that, I am worried that during any ongoing video conversion if my node's traffic handling capability will be affected.

Can someone help me understand what will be the effect of other requests coming to my server while video conversion is going on? How will it impact the RAM, CPU usage and speed of processing other requests?

2. Using AWS lambda function -

To avoid load on my node server I was thinking of using an AWS lambda server where my node API will upload the file to S3 in the format provided by the user. Once, done s3 will trigger a lambda function which can then take that s3 file and convert it into .mp4 using ffmpeg or AWS MediaConvert and once done it uploads the mp4 file to a new s3 path. Now I don't want the output path to be any s3 path but the path that was received by the node API in the first place.

Moreover, I want the user to wait while all this happens as I have to enable other UI features based on the success or error of this upload.

The query here is that, is it possible to do this using just a single API like /api/uploadS3 which --> uploads to s3 --> triggers lambda --> converts file --> uploads the mp4 version --> returns success or error.

Currently, if I upload to s3 the request ends then and there. So is there a way to defer the API response until and unless all the operations have been completed?

Also, how will the lambda function access the path of the output s3 bucket which was passed to the node API?

Any other better approach will be welcomed.

PS - the s3 path received by the node API is different for each user.

Gurmeet Singh
  • 774
  • 8
  • 21

1 Answers1

1

Thanks for your post. The output S3 bucket generates File Events when a new file arrives (i.e., is delivered from AWS MediaConvert).

This file event can trigger a second Lambda Function which can move the file elsewhere using any of the supported transfer protocols, re-try if necessary; log a status to AWS CloudWatch and/or AWS SNS; and then send a final API response based on success/completion of them move.

AWS has a Step Functions feature which can maintain state across successive lambda functions, for automating simple workflows. This should work for what you want to accomplish. see https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-creating-lambda-state-machine.html

Note, any one lambda function has a 15 minute maximum runtime, so any one transcoding or file copy operation must complete within 15min. The alternative is to run on EC2.

I hope this helps you out!

aws-robclem
  • 324
  • 2
  • 5
  • is there a way which handles the following - Nodejs has an API - /api/uploadS3 which uploads a file to S3 which further triggers a lambda function and I get the response of the lambda function as a response of the node API (/api/uploadS3). @aws-rtc – Gurmeet Singh Jul 01 '22 at 04:42
  • 1
    Have a look at the SNS documentation. Lambda Fns can send messages to an SNS queue as well as respond to messages posted to an SNS queue. Another alternative is to maintain a state in a DynamoDB table or for simple states, S3 object metadata such as Tags. – aws-robclem Jul 05 '22 at 16:01