Disruptor - Are the Consumers Multithreaded?

Question

I have following questions regarding the disruptor:

The consumers (event processors) are not implementing any of the Callable or Runnable interfaces they implement EventHandler, Then how can they run in parallel, so for example I have a disruptor implementation where there is a diamond pattern like this

     c1
P1 - c2 - c4 - c5
     c3

Where c1 to c3 can work in parallel after p1, and C4 and C5 work after them.

So conventionally I'd have something like this (with P1 and C1-C5 being runnables/callables)

p1.start();
p1.join();

c1.start();
c2.start();
c3.start();
c1.join();
c2.join();
c3.join();

c4.start();
c4.join();
c5.start();
c5.join();

But in case of the Disruptor none of my event handlers implement Runnable or Callable, so how'd the disruptor framework end up running them in parallel?

Take following sceanrio:

My consumer C2 requires to make a webservice call for some annotation to the Event, In SEDA I can startup 10 threads for such 10 C2 requests [for pulling the message out of queue + make Webservice Call and update the next SEDA Queue] and that will make sure that I don't sequentially wait for a web service response for each of the 10 requests where as in this case my eventprocessor C2 (if) being the single instance would wait sequentially for 10 C2 requests.

score 5 · Answer 1 · edited May 23 '17 at 12:06

5

Your EventHandler is composed into an instance of a BatchEventProcessor, which is a Runnable.

The BatchEventProcessor implements the run() loop.
It is responsible for fetching updated event sequence numbers from the ring buffer.
When events are available for processing, it passes them to your EventHandler for processing.

When using the DSL, the Disruptor takes care of creating these threads via an Executor instance.

You provide an Executor in the Disruptor constructor.
You provide a list of your EventHandlers using and/then/etc on the DSL.
Then, you call Disruptor start() method. As part of the start() method, each EventProcessor (Runnable) is submitted to the Executor.
Executor will in turn launch your EventProcessor/EventHandler on a thread.

With respect to your specific scenario (ie: long running event handlers), you might refer to this question:

Solution to slow consumer(eventProcessor) issue in LMAX Disruptor pattern

edited May 23 '17 at 12:06

Community

1
1

answered Jun 10 '13 at 12:21

jasonk

969
5
9

Many thanks for such prompt answer. While creating the dependency graph (handleWith, and/then etc) I am already providing the instances of my consumers. So while I can have three batcheventprocessors (one per c1,c2 and c3) running in parallel, each of them has only one instance of their respective event handler (which i provided at the start). In other words, Suppose C1 does a webservice Call to WebSrv1, One can never have two WebServ1 calls in parallel across two event buckets in the ring buffer (a scenario in which slot 2 and 3 both executing C1 at a time). wouldn't that be true? – NiksP Jun 10 '13 at 13:48
I interpret to say that c2 and c3 handle different stages in your pipeline (ie: not calling WebSrv1). If so, yes. You might want to look at using Netty / async http; feed the http responses back into the ring buffer, and turn the ringbuffer into a state machine. Alternatively, see the entry linked above. – jasonk Jun 10 '13 at 22:31
Hi Jason, We have observed that if we use like 5-10 disruptors together in an application (sort of like a chain of disruptors with every disruptor having one consumer on it performing a specified task and then handing over the message to the next disruptor/ ringbuffer), What happens is the CPU utilization reaches 90% and above and system becomes unresponsive until we bring down the application, We feel it's because of so many active disruptor threads. This happens even when the disruptors are not really processing anything. Can you comment on this? – NiksP Mar 24 '14 at 17:25
1

Check your WaitStrategy. sleep and busy spin will consume CPU as they idle. If you have more threads than cores, Disruptor is probably not right for your use case. Consider collapsing the event handlers from multiple disruptors into a single disruptor? – jasonk Apr 17 '14 at 11:31

score 0 · Answer 2 · answered Jun 06 '14 at 13:53

0

Disruptor default strategy is multi-threaded, so if every of your processors are working in a different Handler (Consumer), so it should be fine, and your processors are multi-threaded.

answered Jun 06 '14 at 13:53

mohamus

83
1
2
10

Disruptor - Are the Consumers Multithreaded?

2 Answers2