0

I would like to have an akka-stream operator with two inlets. On one inlet it receives metadata about messages. On the second inlet the messages themselves.

The problem is that while metadata is received for one message at a time, messages are grouped in batches.

I would like an operator that accumulates all the metadata on the first port, such that when the message batch comes in the second port, the operator emits two batches (possibly zipped).

The obvious solution would be to just use Zip and group the metadata upstream. But the problem is that the size of the batches is unknown.

gurghet
  • 7,591
  • 4
  • 36
  • 63
  • When you say messages are grouped in batches, do you mean that you're getting some iterable-ish of messages, or just that the messages are very bursty? – Levi Ramsey Dec 04 '19 at 12:22
  • 1
    While there's no built-in operator for exactly this, `zipLatest` comes very close: the only difference is that it will emit the latest message if new meta comes along. If you can augment the message with some sort of correlation ID and dedupe downstream, then all you need is the two `Source`s, a `zipLatest`, a `statefulMapConcat` to accumulate the meta, a `statefulMapConcat` to dedupe, a `Source.actorRef` along with a `merge` to allow you to reset the accumulator, and a `foreach` after dedupe to reset the accumulator. – Levi Ramsey Dec 04 '19 at 12:35

1 Answers1

0

Thanks @LevyRamsey, I done did it.

Combine the elements of 2 streams into a stream of tuples. The elements coming from the left port will accumulate until an element on the right port is available.

A ZipAccumulateLeft has a left and a right input port and one out port

'''Emits when''' all of the inputs have an element available

'''Backpressures when''' downstream backpressures

'''Completes when''' any upstream completes

'''Cancels when''' downstream cancels

https://gist.github.com/gurghet/31cd99a8441ba5fdd380fa7d95bdb628

gurghet
  • 7,591
  • 4
  • 36
  • 63