0

I'm running an akka stream reading from Kafka and I want to commit messages back to Kafka when the file serialization succeeds. But I don't know how to notify upstream stages about a failure downstream.

Right now, I've created a GraphStage with FanOutShape2[ConsumerMessage.CommittableMessage[Array[Byte], Array[Byte]], ConsumerRecord[Array[Byte], Array[Byte]], ConsumerMessage.CommittableOffsetBatch].

I would like push to the offset commits outlet when the downstream sink finish consuming all the hourly grouped ConsumerRecords, but I want to hold those commits when the File couldn't be properly finished.

So given this simplified scenarion, say that I have the following stream

  Source(List("one", "two"))
    .map(ByteString(_))
    .runWith(FileIO.toPath(Paths.get("/file-in-root-will-fail.txt")))

that will end in a IOResult(0,Failure(java.nio.file.AccessDeniedException: /file-in-root-will-fail.txt)).

How could I notify an upstream stage that this happened?

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
tonicebrian
  • 4,715
  • 5
  • 41
  • 65
  • What would the upstream be able to do about it? – Viktor Klang Mar 06 '18 at 21:36
  • By not commiting the Kafka offsets and leave them unconsumed when we fix the problem downstream, records and offsets are still there to flow regularly through the downstream when the stream restarts – tonicebrian Mar 07 '18 at 11:17
  • Add the commit after serialization, then do the write, or add a stage which reads the file and commits what got successfully written. – Viktor Klang Mar 09 '18 at 12:33

0 Answers0