0

I'm trying to emulate the stream drawing from Sources.mapJournal through IMap which receives data from IoT device. The processing of this stream is too slow and I'm getting the big accumulated outcome after 30-60 seconds.

When I started to update the IMap frequently with small data (12 KB per value), the exception is:

com.hazelcast.ringbuffer.StaleSequenceException: sequence:123 is too small and data store is disabled.

I increased the default capacity of IMap journal 10 times. It became stable after that, but very slow. A similar issue is when I'm updating the IMap with big values (about 1.2 MB per 5 seconds). Additionally I have several connected IoT devices and each of them has its own Jet job with the same pipeline:

StreamStage<TagPosition> sourceSteam = 
    p.drawFrom(Sources.<TagPosition, String, TagPosition>mapJournal(
        Constants.IMAP_TAGS_POSITIONS_BUFFER,
        Util.mapPutEvents().and(entry -> ((String) entry.getKey()).startsWith(instanceNumber)),
        Util.mapEventNewValue(),
        JournalInitialPosition.START_FROM_OLDEST));

// Drain to SmartMap
sourceSteam.drainTo(SmartMapSinks.newTagPositionSink(instanceNumber));

Thanks in advance!

UPD:

  • The journal size is EventJournalConfig.DEFAULT_CAPACITY * 10 = 100 000 (1 partition)
  • Jet version is 0.7.2
  • Serialazable classes implements com.hazelcast.nio.serialization.IdentifiedDataSerializable
Sergey Granev
  • 107
  • 8
  • Could you give more numbers: the actual frequency of IMap updates, size of the item, cluster size etc. 1.2MB in five seconds isn't "big". What is `SmartMapSinks`? Isn't it the bottleneck? – Oliv Jan 17 '19 at 12:03
  • What's your journal size? Journal size is per partition, so you need to calculate the journal size per partition as `journalSize / partitionCount`. Also what version of Jet are you using? Another thing to check is what kind of serialization you are using. Plain Java `Serializable` can be quite slow. – Can Gencer Jan 17 '19 at 12:11
  • @CanGencer, answered in the description. – Sergey Granev Jan 17 '19 at 12:54
  • @Oliv, I've tried 500 ms per transaction and plain java foreach loop. Size of checked item was from 12 KB to 1.2MB (different tests). The last bigger size was used when I put List instead of single element. Cluster size (available xmx) is about 1.7 GB. `SmartMapSink` is just a custom sink for batching of data and send it futher through websocket. I've called the service which `SmartMapSink` wraps directly and there were no delays. – Sergey Granev Jan 17 '19 at 13:01
  • Maybe the issue is that the one IMap journal are used by several jet jobs? – Sergey Granev Jan 17 '19 at 13:07
  • @SergeyGranev how many partitions you have? You say 100,000 (1 partition)? But it will actually be 100,000 / 271 entries per partition. Something else you can try is to increase the localParallelism of the source, by default it's set to 2. – Can Gencer Jan 17 '19 at 13:56
  • @CanGencer, thank you for the answers. Yes I've already read that default number of partitions is 271 and didn't get your question at the begining. So I just increased the capacity of journal without any other changes of default values `config.getMapEventJournalConfig(Constants.IMAP_TAGS_POSITIONS_BUFFER).setCapacity(EventJournalConfig.DEFAULT_CAPACITY * 10).setEnabled(true);`. If you say that it is only ~369 entries in the journal - it could be a reason. I also will try to increase localParallelism, but I need to keep the order of received messages. – Sergey Granev Jan 17 '19 at 14:51
  • Guys... I increased the localParallelism and capacity of mapJournal, but it is still slowing. The processing works fast at the start of application and slows down after some time. Need I clean or reset mapJournal after each event? – Sergey Granev Jan 22 '19 at 20:34

1 Answers1

0

The issue was in the using of single IMap (and map journal) by multiple jobs. Map journal was producing events like a batch with delay, but not as stream.

Solved. Thanks!

Sergey Granev
  • 107
  • 8