5

I have multiple related files being uploaded to S3 bucket as group, which I want to process using aws Lambda. For examples externaly, inventory.txt, orders.txt, order_details.txt are received in one folder in s3 bucket. These are part of one batch. Someone else will send the same files in another folder in the same bucket. I want to process these files(cleanse,combine, etc) at the same time (so 3 files at the same time) as a batch.

I have dabbled with Lambda, on S3 create object event level but it gets triggered for each file being uploaded. I want the lambda to trigger for the 3 files (and for the additional 3 files in another directory if applicable).

Amit R
  • 101
  • 1
  • 7
  • Possible duplicate of [AWS - want to upload multiple files to S3 and only when all are uploaded trigger a lambda function](https://stackoverflow.com/questions/34376697/aws-want-to-upload-multiple-files-to-s3-and-only-when-all-are-uploaded-trigger) – BryanJ Jul 26 '19 at 03:14

1 Answers1

0

After the upload process completes, make it create a trigger file(dummy file). For example, your process uploads orders.txt, inventory.txt in s3://my-bucket/today_date/ Make it create a s3://my-bucket/today_date/today_date.complete after copy is complete.

Add S3 event trigger to your lambda function to execute when it a .complete file is uploaded to S3 and the process the rest of the files using lambda and delete the .complete file. And the process repeats for next day.

srinivasb
  • 55
  • 1
  • 7
  • 1
    Thank you. The problem is it is not my process which is uploading the files but external partners uploading the files, like sending the files in sftp, just in this case it's uploaded in s3. – Amit R Jul 26 '19 at 13:02
  • You can perhaps create a lambda that is triggered one one of the files, and which busy-waits for all the other files to be created. If they are coming in a batch, then that should be reasonable efficient. Then it can write some kind of manifest to a file. The writing of that file triggers the real work in another lambda. – mabraham Jun 17 '20 at 12:32