1

My question is:

If I receive multiple event sequences in event loop threads, how could I process every sequence blocking and ordered but different sequences be processed by the whole worker thread pool.

Use case:

I have gRPC client in a vert.x with 4 event loop and 20 worker threads. I start remote calls in 10 threads.

  • Every call receives 2 events StreamObserver.onNext and StreamObserver.onCompleted
  • On every such event I'd like to start blocking task in such way that for a gRPC call onNext and onClose are ordered, while for different gRPC calls events are executed in different worker threads.

The goal is to distribute call execution to as much worker threads as possible but events for single call to be ordered.

  • If I use executeBlocking with ordered true - they will be executed in a worker thread per event thread.
  • If false - there will be no order in processing onNext and onClose.
Avgustin Marinov
  • 11
  • 2

1 Answers1

0

I think it's a good usecase for Kafka.

You can keep working with a worker pool with every thread creating and listening kafka with a consumer. Instead of a worker pool, you can also instanciate multiple verticles in the same java process or even multiple jvm instances (or even containers, why not :p ). I prefer this second approach to handle the dynamic scalability in an event-driven and distributed architecture (I wrote more details about that in this answer).

Either way, all the consumers needs to be instanciate with the same group.id to be a part of the same group of consumers. That way, kafka will be able to assign the partitions of the topic to each group's member (a single partition cannot be read by two members at the same time).

And finally to be able to create "event sequences", you only need to produce the events in the kafka topic with a distribution key that will define your sequence (two messages produced with the same distribution key will be stored in the same partition and will be read by the same consumer of the group).

Idriss Neumann
  • 3,760
  • 2
  • 23
  • 32
  • I have a simple use case - processing gRPC calls (a light-weight communication between client and server) outside event loops and in semi-ordered way. Putting a kafka between them seems like an overkill to me. And the question is more general - how vert.x worker pool could be effectively used (without any externals)/ when nor full ordering nor lack of ordering is proper. – Avgustin Marinov Jun 01 '20 at 10:44
  • @AvgustinMarinov In that case, I think that you'll have to reinvent the wheel with something like broadcasting the events and check by your listeners that they can process the event because it's an event from a new sequence or from a sequence that had been started by the same listener (and keep the sequence's belonging known by all your listeners using a database or a distributed cache for example). Maybe you'll consider redis a little bit less overkill in this case? – Idriss Neumann Jun 01 '20 at 11:28
  • I'd rather think about possibility to "somehow" get set of different Vert/x contexts (bound to every worker thread), assigning such a context (i.e. its worker thread) to each sequence and executing each sequence ordered in its context. Thus I'll effectively get sequential processing per sequence (it is processed in one worker thread ordered) and fully usage of worker thread pool - by using all contexts that use all worker threads. The current issue is that I don't know (or Vert.x doesn't allow) how to get such set of contexts - n contexts bound to n worker threads – Avgustin Marinov Jun 05 '20 at 07:25