My use-case is to design a system to keep reading events (content
) from Source S3 Buckets and merge with the file present in the Target S3 Bucket continuously.
For instance,
Bucket-1(A
) |
Bucket-2(B
) |
Bucket-3(Target
)
A
: Any Update ---> Trigger Lambda (L
) ---> Read file from Target
and merge the newly uploaded content
and write back to Target
B
: Any Update ---> Trigger Lambda (L
) ---> Read file from Target
and merge the newly uploaded content
and write back to Target
These events can happen at the same time and the lambda should keep on updating the same file in Target
.
How can we guarantee consistency of data in Target
Bucket?
I can think of the following approaches:
- Setting lambda concurrency limit to
1
: One write operation onTarget
at a time - Implement locking mechanism on
Target
Bucket. - Schedule a
CRON job to trigger lambda at a particular time
and read all the source buckets and merge the newly uploaded content with the existing one and then write to the target s3 bucket.
Any recommendations?