4

I'm starting with Akka Streams and so far everything is going well. However, I have met with a use case that I don't know how to approach. The scenario is a stream with an ActorPublisher as a source that is consuming messages from Kafka and a subscriber as a sink that updates a Cassandra table.

Kafka ~> some mapping operations ~> Cassandra

The point is that I'd like to explicitly confirm to Kafka every time a message has been successfully processed and inserted into Cassandra so that I could re-read the message in case a disaster happens and the service fails, i.e. some kind of at least once delivery behaviour. How could I approach this in terms of Akka Streams?. Is a supported scenario?.

It's true that I always can configure the Kafka consumer with auto-commit behaviour but I'd rather take control of how I'm reading the messages.

Update

Regarding this topic, we're currently evaluating Reactive Kafka where they've included manual commit in kafka as of 0.8 version (kudos for these guys). This feature would allow us implement the alod behaviour we need.

Willi Mentzel
  • 27,862
  • 20
  • 113
  • 121
Juan José Vázquez Delgado
  • 121
  • 1
  • 6

1 Answers1

2

I think you could use a custom stream processing using a PushPullStage as explained in akka streams documentation

Carlos Fernandes
  • 203
  • 1
  • 2
  • 7
  • This sounds as a proper way to get a fine-grained control over the streaming process. I'll have it a look. Thanks!. – Juan José Vázquez Delgado May 29 '15 at 07:18
  • @JuanJoséVázquezDelgado. I'm looking into a similar use-case. Storm has those messaging guarantees, but Akka streams look interesting if it can provide at-least-once messaging guarantees with Kafka. Did you get a chance to look into this? Any interesting conclusions or findings you an share with us? – joholo Jun 11 '15 at 18:07
  • I'll try to put into practice [this approach](https://groups.google.com/d/msg/akka-user/ujluAC4RCPo/AMbkqZidxAoJ) as it was mentioned in the Akka list. I'll update this thread as soon as I come up with a solution. Thanks for your interest. – Juan José Vázquez Delgado Jun 15 '15 at 12:58