7

I was curious regarding the most common (or recommended) implementations of disruptor about the journaling step. And the most common questions of mine are:

  • how it is actually implemented (by example)?
  • Is it wise to use JPA?
  • What DB is commonly used (by the community that has already implement projects with disruptor)?
  • Is it wise to be used at the intermediate handlers (of EventProcessors) so the State of each message should be saved, rather than before and after the business logic process?

By the way (I am sorry, I know this is not related with the journalling step), what is the right way to delete a message from the RingBuffer during an eventHandler process (assuming that the message is dead/expired and should be removed by the whole procedure). I was wondering something similar as the Dead Letter Channel pattern.

Cheers!

Evan P
  • 1,767
  • 1
  • 20
  • 37

1 Answers1

5

The Disruptor is usually used for low latency, high throughput processing. E.g. millions of messages with a typical latency in the hundreds of micro-seconds. As very few databases can handle this sort of rate of updates with reasonably bounded delays, journaling is often done to a raw file with replication to a second (or third) system.

For reporting purposes, a system reads this file or listens to messages and updates a database as quickly as it can but this is taken out of the critical path.

An entry is dead in the ring buffer when every event processor has processed it.

The slot a message uses is not available until every event processor has processed it and all the message before it. Deleting a message would be too expensive, both in terms of performance and impact on the design

Every event processors sees every message. As this happens concurrently, there is little cost in doing this, but it quite normal for event processors to ignore messages as a result. (possibly most messages)

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • Yes the DB Reporting that I was referring it was assumed to be processed out of the Business critical path. And regarding the dead message, What if a message is expired during the process, it should be deleted (or forward it to a dead message repository). – Evan P Sep 06 '12 at 08:09
  • An expired message needs to be processed like any other message, its only what you do with it that differs e.g. you send an expired response instead of an accepted response. – Peter Lawrey Sep 06 '12 at 08:15
  • An Expiration response is usually depended on other factors as well (on asynchronous architecture). But just to be clear, can't a message be removed from the RingBuffer before its completion? – Evan P Sep 06 '12 at 08:28
  • 1
    The slot a message uses is not available until every event processor has processed it and all the message before it. Deleting a message would be too expensive, both in terms of performance and impact on the design. – Peter Lawrey Sep 06 '12 at 08:30
  • I see, that explains a lot regarding the message manipulation during the business logic process. thx! – Evan P Sep 06 '12 at 08:36
  • Every event processors sees every message. As this happens concurrently, there is little cost in doing this, but it quite normal for event processors to ignore messages as a result. (possibly most messages) – Peter Lawrey Sep 06 '12 at 08:48
  • Indeed, I didn't include the cost of performance factor in that scale. Regarding the journaling step shouldn't be implemented by using a DB as a persistence layer? assuming that the Service (which use Disruptor) is resuming its process, this wouldn't effect overall performance to fetch Messages from a DB repository, and then start the RingBuffer to spin. – Evan P Sep 06 '12 at 08:55
  • The message would be fetched from the raw file rather than a DB which I imagine would be too slow. – Peter Lawrey Sep 06 '12 at 08:57
  • BTW I implemented a different solution to processing messages which made persistent and replication the heart of the solution https://github.com/peter-lawrey/Java-Chronicle It can handle over ten million messages per second with very low latency and has almost no warmup time regardless of the number of messages you are behind. Your event processors can be in the same JVM or different processes with sub-microsecond latency. This allows you to restart an individual process with minimal impact on the overall system. – Peter Lawrey Sep 06 '12 at 08:59
  • If nothing else you can use it for the journaling with the Disruptor. – Peter Lawrey Sep 06 '12 at 09:19
  • I was just thinking of that, Where examples of your library can be found? – Evan P Sep 06 '12 at 09:22
  • In the unit tests. It pretty simple and supports the DataInput and DataOutput interfaces. You can also write text logs. Its designed to be lock less, mostly off heap and GC free. – Peter Lawrey Sep 06 '12 at 09:25
  • it really seems Ideal, i'll keep up testing it but seems promising. thx! – Evan P Sep 06 '12 at 09:40
  • It performs well even if the consumer is more messages behind the producer than will fit in memory e.g. hundreds of millions. ;) – Peter Lawrey Sep 06 '12 at 09:44
  • You should edit your answer to include your library and the answer regarding message deletion cost in performance. – Evan P Sep 06 '12 at 09:51
  • Since I wrote the library, I don't like self promotion. ;) – Peter Lawrey Sep 06 '12 at 10:07