Design architecture of a notification system

Question

I need to design a notification system that will send notifications to users when events happen.

I have a microsystem environment, actually, when an event happen, a ms sends a message in a queue (q.send-event), that will be consumed by a notifications microservice. The listener of q.send-event queue, actually, makes a REST call to users microservice, to gather the list of all users that will receive the notification. After the user's list is gathered, then it store the notifications in a database.

I feel that this approach is not so scalable, can you suggest me a more efficient approach? Thanks

Answer 1

A very scalable solution (see arch diagram) to what you are describing is similar but if I were to build this it would be using Kafka or managed Kafka with a data connection to your database (e.g. Debezium connector for MySQL). This connector would perform your change data capture (CDC) and stream those events using Kafka queues.

The reason why the use of Kafka is important here vs. AWS MQ or some other queue solution like RabbitMQ is that Apache Kafka guarantees delivery of the events in the correct order even when you have multiple consumers. Both Rabbit and managed MQ can make similar guarantees but this breaks down when you introduce many consumers of the event stream.

As for the messaging component of your solution, it would be more pragmatic and scalable to publish messages to a webhook service or similar that is a consumer of your kafka queues and is recording events to a separate database so that the event stream can be subscribed to vs. relying on a service that has to know who its consumers are.

So your gut is correct that this solution is not incredibly scalable albeit not a "bad" solution by any means. The two recommendations I made I think would help you move forward in scalability.

I hope this helped. Cheers.

Answer 2

The listener of q.send-event queue, actually, makes a REST call to users microservice, to gather the list of all users that will receive the notification.

In microservices architecture synchronous calls between services in general case are considered to be not the best option. In this case I think the standard approach with data duplication is a way to go:

notifications maintains a readonly copy of user data (only the relevant one) which is asynchronously updated via message broker (i.e. every updated of data in user service produces corresponding message which is consumed by subscribers one of which is notifications which saves only the needed data to it's own storage). This removes the necessity of calling users during processing of q.send-event.

As an extension of this you can migrate user notification settings to the notifications service completely (better done with moving the notification settings to separate part of the UI or managing this with BFF/API gateway approach). I would argue that from business logic/domain point of view notification service can be a natural owner of user notification settings (unless you want to introduce another service):

Note that actual "best" implementation can be heavily depended on the actual business requirements, data amounts and SLAs.

Answer 3

You can use something like socket.io to broadcast notifications to all/some users in real-time. You'll have to set it up on both the server and the client side, if you want to start quickly there are managed solutions like Pusher that you can use.

Answer 4

There are a lot of approaches when you want to build a scalable notification system solution. This was my approach when I built a system like this. The main components of my system are:

1. User Service: This is where you manage your user information and provides APIs for fetching user details.

2. Delivery Service: Depending on the different delivery channels you want to use e.g SMS or email to deliver the messages. You will create a component that will be responsible for delivering notifications to your users

3. Event Service: Create a listener that receives events from other services and triggers the appropriate notifications.

4. Database and Message Queue: I used Kafka (A message queue) with a data connection to my database (MySQL - This stores information about notifications and users). Kafka decouples the Event Service from the Notification Service and handles a large number of events.

5. API Gateway: Create a component that makes a REST call to users microservice to gather the list of all users that will receive the notification. This serves as the entry point for external clients to access the notification system's APIs.

6. Monitoring and Logging: Lastly, you can use AWS services to monitor your system and troubleshoot issues, tools such as Elasticsearch, Logstash, and Kibana (ELK) can be used to collect and analyze log data.

If you are still confused and need a pictorial diagram, I can check my architectural diagram and upload it.

Answer 5

From what I understood you only need list of user Id's to send notifications. What you can use here is publisher subscriber method. The User micro service will give user information to all its subscriber.
There are multiple methods to achieve that, I'll mention few methods here:

1. User service can send user events to messaging queue for ex - Kafka and notification service can subscribe to this queue and get user info.
2. User service can expose apis to register different services as a subscriber and on user events (you can select for which user events) it will notify its subscriber that these user Id's have been updated or added.
   

By Having this mechanism notification service will have user Id's information stored and can use it to send notification to users asynchronously.
As the exact use case is not clear here you can use these methods and modify them according to your needs and if this solution doesn't work for you please add more information about your use case.