Akka streams: attach new publishers/subscribers to Flow

Question

I'm building an Akka application and want to expose the FSM state transitions of some actors to external consumers. (The end goal is to be able to push the state transition messages to a websocket so that they can be viewed in realtime.)

Based on the documentation, Combining dynamic stages to build a simple Publish-Subscribe service, it looks like I need to expose the Flow that represents the pub-sub channel so that it can be used by consumers and producers.

The part I'm having trouble with is attaching new Source(s) to the Flow such that each new actor that is spawned will publish its state transitions to the Source. The other issue is adding new Sinks to the Flow (in the end, this will be websockets but for testing purposes, it could be any new Sink).

First I connect a MergeHub and a BroadcastHub to form a "channel" and then create the Flow from the materialized sink and source:

    val orderFlow: Flow[String, String, NotUsed] = {
        val (sink, source) = MergeHub.source[String](16)
            .toMat(BroadcastHub.sink(256))(Keep.both).run()
        Flow.fromSinkAndSource(sink, source)
    }

The question is then how can I dynamically add new producers and consumers to this Flow? Any ideas?

Do you have any code? Have you tried isolating the pub-sub concern and write a simplified version along with a test for it? It's gonna be very difficult for people to help you when the question is too broad or unclear. — Nader Ghanbari, Feb 22 '18 at 23:21
Added a code snippet that perhaps illustrates the problem a little better. — Eric, Feb 23 '18 at 09:37

andrey.feoktistov · Answer 1 · 2019-05-29T08:54:08.717

I use the following solution in one of my projects to handle the requests from the websocket by multiple request processors which could produce a stream of responses or provide an endless subscription.

// requests coming from websocket, it could be any source, it's doesn't matter
val requests: Source[Request, NotUsed] = ... 

// the request processing here can provide endles stream of responses
val requestProcessing: Flow[Request, Response, NotUsed] = ...

val (outSink, outSource) =
  MergeHub
    .source[Result](perProducerBufferSize = 4)
    .toMat(BroadcastHub.sink(bufferSize = 32))(Keep.both)
    .run()

Source.tick(Duration.Zero, KeepAliveInterval, ConnectionKeepAlive)
  .to(outSink)
  .run()

requests.fold {
  case State(state, AuthRequest(r)) if checkAuth(r) => 
    Source.single(AuthenticationAck).to(outSink)
    state.copy(isAuthenticated = true)
 case State(state, AuthRequest(r)) => 
    Source.single(AuthenticationFailedError).to(outSink)
    state

 case State(state, request) if s.isAuthenticated =>
    // here the most of busines
    Source.single(request).via(requestProcessing).to(outSink)
    state

 case State(state, _) => 
    Source.single(NonAuthorizedError).to(outSink)
    state
}.toMat(Sink.ignore)(Keep.right)

outSource.runForeach { response =>
  // here we get the stream of responses mixed from all requests
}

outSource.runForeach { response =>
  // of course, we could have as many subscribers as we need
}

I hope it helps :)

score 0 · Accepted Answer · answered Feb 23 '18 at 11:15

For this particular problem I would not use akka-stream. The type of multicast pub-sub that you describe is much more suited for raw Actor messaging and the EventStream.

I am a huge fan of akka-stream for certain situations, but in this case I think you are trying to fit a square peg through a round hole.

Akka streams: attach new publishers/subscribers to Flow

2 Answers2