Akka stream actor-conflation-ratelimit-actor drops first few messages (sometimes)

Question

A simple conflating combo (below) sometimes prints a debug message at staartup saying it's dropping messages because of zero demand. I would expect conflation stage to provide infinite demand, so the above should never be the case. What am I missing?

val sourceRef = Source.actorRef[KeyedHighFreqEvent](0, OverflowStrategy.fail)
.conflateWithSeed(...into hash map...)
.throttle(8, per = 1.second, maxBurst=24, ThrottleMode.shaping)
.mapConcat(...back to individual KeyedHighFreqEvent...)
.groupedWithin(1024, 1.millisecond)
.to(Sink.actorRef(networkPublisher, Nil))
.run()

system.eventStream.subscribe(sourceRef, classOf[KeyedHighFreqEvent])

score 1 · Accepted Answer · answered Apr 03 '19 at 07:15

The documentation of Source.actorRef is quite clear about this:

The buffer can be disabled by using bufferSize of 0 and then received messages are dropped if there is no demand from downstream. When bufferSize is 0 the overflowStrategy does not matter. An async boundary is added after this Source; as such, it is never safe to assume the downstream will always generate demand.

The problem is the async boundary between the source and the conflation stage. The conflation stage indeed provides infinite demand, but the async boundary kind of makes it slow to propagate to the source.

You can either use a buffer in your source (increase bufferSize), or use another source e.g., Source.queue if appropriate as it doesn't introduce an async boundary

Thanks, makes perfect sense, I overlooked the "An async boundary is added" part. — bobah, Apr 03 '19 at 12:04

Akka stream actor-conflation-ratelimit-actor drops first few messages (sometimes)

1 Answers1