2

This question is about the Pipeline API in Hazelcast Jet 0.5.1

The pipeline I am trying to create has two infinite sources: one is a ticker (a custom source which sends one event every minute), the other is a Kafka Topic.

It looks like that:

Pipeline pipeline = Pipeline.create();
ComputeStage<Object> tickerSource = pipeline.drawFrom(Sources.fromProcessor("ticker", TickerSource.getSupplier()));
ComputeStage<Object> kafkaSource  = pipeline.drawFrom(KafkaSources.kafka(sourceProperties, KAFKA_TOPIC));

When either of those sources emit an event, I want that event to go through the same steps and drain to the same sinks. I want a "UNION", if we translate my problem to SQL terms. Something that would look like this:

target pipeline

All of the examples and documentation I've found about having two nodes go into one would be the equivalent of a SQL "JOIN" operation, not a "UNION".

The only way I've found to bypass my issue is to do something like this, but I feel like this is something the framework should already have despite the fact that I can't seem to find it.

Arrays.asList(tickerSource, kafkaSource).forEach(source ->
{
    ComputeStage<Object> result = source.map(MyCustomProcessor::process);
    result.drainTo(Sinks.fromProcessor("first-sink", MyFirstSink.getSupplier());
    result.drainTo(Sinks.fromProcessor("second-sink", MySecondSink.getSupplier());
});

The result looks like this:

resulting pipeline

Can Gencer
  • 8,822
  • 5
  • 33
  • 52
Valentin Prévost
  • 51
  • 2
  • Yes you are right, you can currently only do a "merge" through using the DAG API directly. Since your pipeline looks quite simple I would consider using that instead perhaps for now? Or your workaround will work fine as well. However we will be adding this functionality (among other things) to Pipeline API in the near future. – Can Gencer Mar 21 '18 at 21:03
  • I will also add that union and merge is slightly different - union typically returns distinct items. – Can Gencer Mar 21 '18 at 21:12
  • 1
    This is now merged into master so available in 0.7-SNAPSHOT: https://github.com/hazelcast/hazelcast-jet/pull/885 – Can Gencer May 31 '18 at 06:44

0 Answers0