Hazelcast-Jet drainTo syntax issue

Question

I am trying to aggregate using Jet, source and sink are a Kafka topic, requirement is to take GPB (google proto buf) messages from source and publish a GPB messages. Problem is I am able to publish Double but not a GPB message and it gives me compile error.

This works fine:

    Pipeline p = Pipeline.create();
    p.drawFrom(KafkaSources.<String, Balance> kafka(<properties>, <topic>)) 
    .map(s->s.getValue() ).groupingKey(x->x.account)
    .rollingAggregator(AggregateOperations.summingDouble(Balance::amount))
    .drainTo(KafkaSinks.kafka(<prop>,<sinktopic>));

Even though the above code works fine, it publishes double to sink topic while my requirement is to publish a GPB having double attribute to sink topic. When I try to do that by putting a map before drainTo, it gives me syntax error. Below is what I tried:

    .rollingAggregator(AggregateOperation.summingDouble(Balance::amount))
    .map(k->Amount.newBuilder().setAmount(k.getValue()).build())
    .drainTo(KafkaSinks.kafka(<prop>,<sinktopic>));

Amount is a GPB message having a double attribute. This gives me syntax error which i don't understand. Could you please help me getting this through.

Can you please share some docs or links as well where there are different aggregations for different scenarios? I went through Hazelcast samples, demos, not all, but few, but didn't find my use case there. Thanks a lot.

Answer 1

I guess the syntax error was this:

Incompatible types. Required Sink<? super Amount> but 'kafka' was inferred to Sink<Entry<K, V>>: no instance(s) of type variable(s) K, V exist so that String conforms to Entry<K, V>

(Next time, please share the exception, your code has dependencies to non-shared classes and I cannot compile it.)

It means that the Kafka sink expects java.util.Map.Entry on input, but you gave it Amount. You need to map it this way:

.map(entry-> Util.entry(entry.getKey(), Amount.newBuilder().setAmount(entry.getValue()).build()))