App1: sends the communications (eg: email)
App2: gets webhook parse the JSON and update DB records
When I trigger for example 10k communications from App1, I will receive webhooks based on the events on App2 so far so good...
Now as I receive these webhooks I want to club these requests into one and update the records in one go rather than updating one by one which leads to downtime of my App2 and along with that also increases the loads on my RDS DB instance.
while receiving the webhooks I route those requests through celery and made a workflow to execute the requests and controlled the outflow accordingly it somewhat solves the problem but still, I am updating the records one by one so that's not what anyone wants to do right. So, need some suggestions on celery workflow on how to make these incoming celery tasks into one batch (10requests/per batch) and process these 10 tasks and make a master query and initialize a new task which will take this master query and update the data of 10 records in one go.
Workflow :
App1 ---> sends communications
App2 --> gets webhooks --> push to celery-service --> another service to update DB
Solution: I will put a middleware (any storing data structure like:: Redis or MongoDB) in between celery-service and App2 which stores the requests data in batch (ie. list of 10 requests data) and there is a periodic celery task that polls from the middleware and process the 10 requests data into one and further updates the DB records. The problem with a solution is now my middleware becomes the bottleneck here
App2 --> gets webhooks --> pushes to ->(middleware)-> celery-service polls 10 request batch from Middleware --> make bulk update to Db
Please suggest something to batch these incoming webhooks and update records in one go. Feel free to ask if something is confusing.
