0

I have a use case which for which i want to make my system scalable. Lets say some producers are pushing different events in a Message queue. Event 1,Event2, Event3 etc. In backend consumer I want to create configurations for these events, for example aggregate and send notifications(assuming i have a api) for event 1 every hour(cron expression), send notification for event 2 every 5 mins etc. Can someone help me as to how to achieve this design?

Example lets say for event 1 my previous run according to cron expression was 9 am and next run is going to be 10 am, I want to aggregate all events from message queue between 9 am and 10 am and send batched notification. Queue might also have event2 in it.

Using event and messages interchangeably.

Inderjit Chopra
  • 31
  • 1
  • 7

1 Answers1

0

  1. create cron jobs for each event: event 1: every 1 hour, event 2: every 5 mins, ...

  2. read the messages from the queue: increment the counter for the current read event(ex: event 1)

  3. When a cron job is called on the scheduled time: read the current value of the counter for this event(ex: event 2) and send notification using the API

you may use redis INCR for aggregating and reading the events count

balaji balu
  • 7
  • 2
  • I am thinking of storing events in a db and running cron for each event by config and then aggregating from db using sql query. – Inderjit Chopra Mar 25 '23 at 18:10
  • use redis-like db for storing events count. storing events count in regular db will not be performant and scale your app. "redis incr" aggregates the events count. When you read the value from redis, it is already aggregated. No need to aggregate during query – balaji balu Mar 25 '23 at 23:05
  • aggregation logic can be flexible, example- every event has a user id, aggregate based on user id(some other parameter) So i am thinking of using sql query as a rule for every event key. – Inderjit Chopra Mar 26 '23 at 06:17