0

I am using alpakka connector to consume packets from Kafka and insert them into Mongo db. I was trying to commit the offset after getting response from Mongo db but couldn't find anything regarding the same. How can I make sure that offset will be committed only after the packet successfully inserted to the Mongodb?

I am using Consumer.CommittableSource as a source and MongoSink as a sink and running the stream using RunnableGraph. Please refer the code for more clarification.

Source:

    public Source<ConsumerMessage.CommittableMessage<String, String>, Consumer.Control> source() {
        return Consumer.committableSource(consumerSettings, subscription);
    }

Flow:

   public Flow<ConsumerMessage.CommittableMessage, String, NotUsed> transformation() {
        return Flow.of(ConsumerMessage.CommittableMessage.class).map(i -> i.record().value().toString());
    }

Sink:

    public Sink<String, CompletionStage<Done>> sink() {
        return MongoSink.insertOne(mongoCollection);
    }

Graph:

        RunnableGraph graph = RunnableGraph.fromGraph(GraphDSL.create(sink(), (builder, s) -> {
            builder.from(builder.add(source()).out()).via(builder.add(transformation())).to(s);
            return ClosedShape.getInstance();
        }));
        graph.run(ActorMaterializer.create(actorSystem));

EDIT:

Using PassThroughFlow, insertion to Mongo is working and it is not giving any exception or error but still not able to commit the packet. transformationCommit() function has never been called.

Updated Flow :

    public Flow<String, String, NotUsed> transformationMongo() {
        LOGGER.info("Insert into Mongo");
        return  MongoFlow.insertOne(connection.getDbConnection());
    }

public Flow<ConsumerMessage.CommittableMessage, ConsumerMessage.CommittableOffset, NotUsed> transformationCommit() {
        return Flow.of(ConsumerMessage.CommittableMessage.class).map(i -> i.committableOffset());
    }

Sink:

    public Sink<ConsumerMessage.CommittableOffset, CompletionStage<Done>> sinkCommit() {
        CommitterSettings committerSettings = CommitterSettings.create(actorSystem);
        return Committer.sink(committerSettings);
    }

PassThroughFlow:

public class PassThroughFlow {
    public Graph<FlowShape<ConsumerMessage.CommittableMessage, ConsumerMessage.CommittableMessage>, NotUsed> apply(Flow<ConsumerMessage.CommittableMessage, String, NotUsed> flow) {
        return Flow.fromGraph(GraphDSL.create(builder -> {
            UniformFanOutShape broadcast = builder.add(Broadcast.create(2));
            FanInShape2 zip = builder.add(ZipWith.create((left, right) -> Keep.right()));
            builder.from(broadcast.out(0)).via(builder.add(flow)).toInlet(zip.in0());
            builder.from(broadcast.out(1)).toInlet(zip.in1());
            return FlowShape.apply(broadcast.in(), zip.out());
        }));
    }
}

Graph:


RunnableGraph graph = RunnableGraph.fromGraph(GraphDSL.create(sinkCommit(), (builder, s) -> {
            builder.from(builder.add(source()).out()).via(builder.add(passThroughFlow.apply(transformation().via(transformationMongo())))).via(builder.add(transformationCommit())).to(sales);
            return ClosedShape.getInstance();
        }));
        graph.run(ActorMaterializer.create(actorSystem));

//Insertion to mongo is working but still the packet not committed. transformationCommit() function has never been called.
mayank bansal
  • 61
  • 9

1 Answers1

0

Instead of MongoSink, use MongoFlow which lets you to continue the stream afterwards, where you will be able to commit offsets.

dvim
  • 2,223
  • 1
  • 17
  • 17