1

I'm trying to get more up to speed with Hazelcast Jet as a processing pipeline and began digging into the Sources documentation a bit more

https://jet-start.sh/docs/api/sources-sinks#sources

I was very surprised to see that Hazelcast Topics and ReliableTopics aren't natively supported sources, however Kafka and JmsQueue are supported out of the box. I do see that a hazelcast reliabletopic is a supported sink... which makes me question whether or not I am misunderstanding a topic as a valid source for a pipeline.

Am I misunderstanding something?

Erick J
  • 53
  • 6

1 Answers1

2

Hazelcast ITopic is a non-distributed in-memory facility, and doesn't support exactly-once processing. These are probably some of the reasons Hazelcast hasn't so far offered first-class support for it. Kafka is an example of a persistent, distributed and fault-tolerant facility, which makes it a good match for an architecture where you require the exactly-once consistency level.

On the other hand, you can use the SourceBuilder and SinkBuilder to implement your own connectors for it. Here's an example from our code samples repository.

Oliv
  • 10,221
  • 3
  • 55
  • 76
Marko Topolnik
  • 195,646
  • 29
  • 319
  • 436
  • ahh, I had missed that ITopic "doesn't support exactly-once processing" from the hazelcast docs... I think I had assumed too much about the 'globalOrderEnabled' flag and attributed 'exactly-once' processing to that too. Makes sense. Thanks! – Erick J Dec 08 '20 at 14:56