1

What is the best way to with reliability get our client to send audio files to our S3 bucket that will process the audio files (ML processes that will do speech-to-text-insights)?

The files could be in .wav / mp3 other such audio formats. Also, some files may be larger in size.

Love to get best ideas? (e.g. API Gateway / Lambda / S3 ?) Would love to hear from anyone who may have done this before.

Some questions and answers to give context:

How do users interface with your system? We are looking for API based approach vs. a browser based approach. We can get browser based approach to work but not sure if that is the right technical/architectural / scalable approach

Do you require a bulk upload method? Yes. We would need bulk upload functionality and some individual files may be larger as well

Will it be controlled by a human, or do you want it to upload automatically somehow? Certainly want it automatically

ultimately, we are building a SaaS solution that will take the audio files and meta data and perform analytics on it and deliver results of our analysis through an API back to the App. So the approach we are looking for is something that will work within this context

bazooka720
  • 21
  • 4
  • Is this one client, or many clients? If it is just one client, you could provide them with IAM User credentials and then have them transfer files with the [AWS Command-Line Interface (CLI)](http://aws.amazon.com/cli/), or they might want to write their own tools that use an AWS SDK. You can then use an Amazon S3 Event to trigger the processing. What have you tried so far? – John Rotenstein Dec 07 '20 at 04:18
  • Its many clients as our SaaS allows clients to add this information. We are just building an MVP at this this. We were unsure how to handle the data ingestion ...piece. All others have been figured out – bazooka720 Dec 07 '20 at 13:23
  • How do users interface with your system? Do you want to allow them to upload via web browser, or do you require a bulk upload method? Will it be controlled by a human, or do you want it to upload automatically somehow? Please edit your question to provide more details of your situation, and what you have considered/rejected so far. – John Rotenstein Dec 07 '20 at 20:59
  • I have added the comments – bazooka720 Dec 08 '20 at 01:49
  • I'd probably look at having clients authenticate either to your application or to Amazon Cognito, which can then vend them a set of temporary credentials that have permission to upload to a given path of an Amazon S3 bucket. The clients can then use these credentials to upload files to Amazon S3 via AWS CLI or via a custom app that runs locally on their system. The authentication step could be combined with this script so that it can operate automatically on a schedule without human involvement. Amazon S3 can then trigger an AWS Lambda function to "process" the uploaded files. – John Rotenstein Dec 08 '20 at 01:56

1 Answers1

1

I have a similar scenario.

If you intend to use Api Gateway/Lambda/s3 then you should know that there is a limit on the payload size that Gateway & Lambda can accept. Specifically, Api Gateway accepts payloads till 10 MB & Lambda till 6MB.

There is a workaround for this issue though. You can upload your files directly on an s3 bucket and attach a lambda trigger on object creation.

I'll leave some articles that may point you to the right direction :

  1. Uploading a file using presigned URLs : https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html
  2. Lambda trigger on s3 object creation: https://medium.com/analytics-vidhya/trigger-aws-lambda-function-to-store-audio-from-api-in-s3-bucket-b2bc191f23ec
  3. A holistic view of the same issue: https://sookocheff.com/post/api/uploading-large-payloads-through-api-gateway/
  4. Related GitHub issue : https://github.com/serverless/examples/issues/106

So from my pov, regarding uploading files, the best way would be to return a pre-signed URL, then have the client upload the file directly to S3. Otherwise, you'll have to implement uploading the file in chunks.

AbstactVersion
  • 46
  • 4