0

I have a use case where I need to process a set of events. I need them to process in parallel holistically but serially for each user. Can this be done in PubSub (maybe GCP Tasks?)?

For Example:

6 events come in at one time (User_A_Event_1, User_A_Event_2, User_B_Event_1, User_B_Event_2, User_C_Event_1, User_D_Event_1).

I want to group them by UserID, process each user in parallel and then and process each event serially (subsequent event processing won't begin until successful completion of the prior event). Something like:

  • User A Serial Processing: Process User_A_Event_1 --> Process User_A_Event_2
  • User B Serial Processing: Process User_B_Event_1 --> Process User_B_Event_2
  • User C Serial Processing: Process User_C_Event_1
  • User D Serial Processing: Process User_D_Event_1

If it matters, I have not idea what users will have events and at what times. We might go months without seeing any events for a user and then start getting lots of them.

I am trying to figure out a way to accomplish this in GCP PubSub but I am open to other solutions as well. My preference would be to do this through a push instead of a pull as I could go long periods of time with nothing in the queue.

Appreciate your help.

Craig

mornindew
  • 1,993
  • 6
  • 32
  • 54
  • 3 questions: 1. How do you know that an event is before another one? Do you have a timestamp? An incremental ID? Anything else? 2. Do you know the number of user in advance? If so, what is the process for the provisioning of new users? 3. Do you need real time or you can process the events per batch (every hour for example)? – guillaume blaquiere Nov 21 '20 at 19:42
  • I do have a timestamp but I am actually ok with FIFO. I will also know the UserID when the event happens. I have no idea when a user will have events but all users in my system could end up having events on a daily basis. I do need it in real time unfortunately, processing in a batch will not meet my needs. – mornindew Nov 22 '20 at 01:49
  • Ok, in fact, if you have 2 event that happen very close in the time, you want something like a "lock" to not process them in parallel. Do you know the high frequency (or the lowest interval) that you can have between 2 events of the same user? Do you know the max processing time for an event? – guillaume blaquiere Nov 22 '20 at 13:19

1 Answers1

2

Cloud Pub/Sub's ordered delivery could help here. You would use the user as the ordering key. This would mean that Cloud Pub/Sub would deliver the messages to your subscribers in the order in which they were received by the service from your publishers. Ordered delivery would have the properties you desire where you don't know the set of users in advance and where events for a particular user can be rare or bursty.

On the subscribe side, the guarantees made depend on the type of subscriber. For the client libraries (which use streaming pull), the callback you provide will be executed to completion for messages with the same key one at a time. For subscribers using pull, each pull request will contain messages for a key in the order in which they were received and a key's messages will only be outstanding in one pull response at a time. For push subscribers, each message for an ordering key will be sent individually to your endpoint and the next message will not be sent until the previous message for the same key is acknowledged.

Note that Cloud Pub/Sub's ordered delivery still has at-least-once delivery semantics, meaning that an acknowledged message could be redelivered, which would also result in the redelivery of subsequent messages for the same key.

See the Medium post about ordering for more details.

Kamal Aboul-Hosn
  • 15,111
  • 1
  • 34
  • 46