0

I need to build the following graph:

val graph = getFromTopic1 ~> doSomeWork ~> writeToTopic2 ~> commitOffsetForTopic1

but trying to implement it in Reactive Kafka has me down a rabbit hole. And that seems wrong because this strikes me as a relatively common use case: I want to move data between Kafka topics while guaranteeing At Least Once Delivery Semantics.

Now it's no problem at all to write in parallel

val fanOut = new Broadcast(2)
val graph = getFromTopic1 ~> doSomeWork ~> fanOut ~> writeToTopic2
                                           fanOut ~> commitOffsetForTopic1

This code works because writeToTopic2 can be implemented with ReactiveKafka#publish(..), which returns a Sink. But then I lose ALOS guarantees and thus data when my app crashes.

So what I really need is to write a Flow that writes to a Kafka topic. I have tried using Flow.fromSinkAndSource(..) with a custom GraphStage but run up against type issues for the data flowing through; for example, what gets committed in commitOffsetForTopic1 should not be included in writeToTopic2, meaning that I have to keep a wrapper object containing both pieces of data all the way through. But this conflicts with the requirements that writeToTopic2 accept a ProducerMessage[K,V]. My latest attempt to resolve this ran up against private and final classes in the reactive kafka library (extending/wrapping/replacing the underlying SubscriptionActor).

I don't really want to maintain a fork to make this happen. What am I missing? Why is this so hard? Am I somehow trying to build a pathological graph node or is this use case an oversight ... or is there something completely obvious I have somehow missed in the docs and source code I've been digging through?

Current version is 0.10.1. I can add more detailed information about any of my many attempts upon request.

Keith Nordstrom
  • 354
  • 2
  • 9
  • I'm not entirely sure, but [this](http://doc.akka.io/docs/akka-stream-kafka/current/producer.html#producer-as-a-flow) seems to be what you're looking for. There's even an example which seems to be doing what you want. – Vladimir Matveev Jun 30 '17 at 06:16
  • Thanks, but this is Akka Streams Kafka 0.16. We're using Reactive Kafka 0.10.1. I know it's possible to do this in general - my problem is that I have an existing library in use in a variety of services and it should be possible to do this without a major rewrite. – Keith Nordstrom Jun 30 '17 at 13:26
  • Ah, I see, sorry. For some reason I thought that 0.10 is the latest version of Reactive Kafka. Indeed, 0.10 doesn't seem to contain such a thing. – Vladimir Matveev Jun 30 '17 at 13:54
  • In order to use this I'd need to upgrade our Kafka to 10+. Pretty big scope creep for that, though, considering how many services would need to be upgraded. – Keith Nordstrom Jun 30 '17 at 14:11

0 Answers0