2

My use-case is to design a system to keep reading events (content) from Source S3 Buckets and merge with the file present in the Target S3 Bucket continuously.

For instance,

Bucket-1(A) | Bucket-2(B) | Bucket-3(Target)

A: Any Update ---> Trigger Lambda (L) ---> Read file from Target and merge the newly uploaded content and write back to Target

B: Any Update ---> Trigger Lambda (L) ---> Read file from Target and merge the newly uploaded content and write back to Target

These events can happen at the same time and the lambda should keep on updating the same file in Target.

How can we guarantee consistency of data in Target Bucket?

I can think of the following approaches:

  1. Setting lambda concurrency limit to 1: One write operation on Target at a time
  2. Implement locking mechanism on Target Bucket.
  3. Schedule a CRON job to trigger lambda at a particular time and read all the source buckets and merge the newly uploaded content with the existing one and then write to the target s3 bucket.

Any recommendations?

Prashant
  • 25
  • 6

0 Answers0