1

I´m currently developing a system which has the requirement to work asynchronous, so i decided it was time to give event driven architecture a go. Most of the functionality is CRUD

The different parts is:

  • An UI which communicates with an API Gateway using a websocket connection
  • The gateway receives the payload and creates a message with payload that goes into a Kafka cluster
  • A service consumes the messages, validates and stores the entity in a database. An event is pushed to Kafka for consumption by the gateway and back to the UI

For create, update and delete its pretty straight forward.

However when it comes to retrieving data where i normally would use a HTTP GET call, stuff gets a little tricky. How can i properly retrieve this data? Do i create a request event, or what options do i have? The data in the database also needs to be "searchable" and "pageable" through the service - usually i would use a querystring for that. I find my "request data events" gettings pretty huge, and containing a lot of logic on what data is requested. Bacically the flow is the same right now as for create, update ect. where an event is pushed to Kafka containing information about what data that should be queried.

What is the correct way of handling data reads in an event driven way?

How do i property make my data searchable? (like get all resources which has a foreign key to resource x, or contains name = "xyz")

frisk0k
  • 157
  • 1
  • 1
  • 11
  • You say that a requirement is for the system to be `asynchronous` - what is the business reason behind that? For reading, you are most likely better of querying the DB from API Gateway (maybe through some thin layer to abstract persistence details). If the business requirement is performance/scalability, you could address that with other tools (caching, read-only DB mirror, etc.), which are simpler and much more efficient than doing this via events. – JME May 21 '20 at 21:34

1 Answers1

0

There are multiple options depending on your business requirement:

Option A:

  • You may query and distribute data at "write time". Meaning before putting data into Kafka cluster you can keep map of connected clients and distribute data to these clients depending on their interest.
  • Also using this map of connected clients based on their interest you can subscribe these clients to relevant topics in Kafka.
  • You can also consider Server Sent Events and separate ingress websocket connection from updates to your clients if that's needed

Option B:

  • You can periodically check your database for the mentioned query and stream results to the client based on their interest.
  • You can consider distributed scheduling of these queries if you need to scale this to multiple replicas.

Option C:

  • You can also consider tech like GCP Firestore and write the query results either on write time or read time and let the Firestore handle for distribution.

Also you can combine these options and create a hybrid approach.

ismet özöztürk
  • 106
  • 1
  • 5