I have a streaming serving using kafka, where I receive data from multiple users and I want to process the data where each users data must be processed in sync manner where as different User's data data can be processed on async manner? Is there any standard pattern available for such scenarios or situations ?
Asked
Active
Viewed 266 times
0
-
do you mean any particular user's data you want to processed in sequential manner? i.e. if there are 2 users, all updates on U1 must be processed sequentially & all updates on U2 must be processed sequentially, but the processing of U1 & U2 can be done parallel. – satyaprakash Jul 16 '22 at 11:36
-
@satyaprakash yes you are correct. – KISHAN KUMAR PATEL Jul 16 '22 at 11:43
1 Answers
1
You can achieve so, by using userId as the key while publishing the message to kafka.
Keys are used to ensure the messages published to kafka with a particular key are ordered by pushing them into a single partition.
And as each consumer is assigned one partition (in best case, i.e. there can't be any such case where one partition is shared among consumers), thus consumer would be consuming the data from partition in sequence it is pushed.
satyaprakash
- 151
- 1
- 5
-
This assumes there's multiple partitions in the topic. If not, all users are still processed sequentially – OneCricketeer Jul 16 '22 at 14:27