Should I emit from a coroutine when collecting from a different flow?

Question

I have a use case where I need to trigger on a specific event collected from a flow and restart it when it closes. I also need to emit all of the events to a different flow. My current implementation looks like this:

scope.launch {
    val flowToReturn = MutableSharedFlow<Event>()
    while (true) {
        client
            .connect()                                   // returns Flow<Event>
            .catch { ... }                               // ignore errors
            .onEach { launch { flowToReturn.emit(it) } } // problem here
            .filterIsInstance<Event.Some>()
            .collect { someEvent ->
                doStuff(someEvent)
            }
    }
}.start()

The idea is to always reconnect when the client disconnects (collect then returns and a new iteration begins) while having the outer flow lifecycle separate from the inner (connection) one. It being a shared flow with potentially multiple subscribers is a secondary concern.

As the emit documentation states it is not thread-safe. Should I call it from a new coroutine then? My concern is that the emit will suspend if there are no subscribers to the outer flow and I need to run the downstream pipeline regardless.

score 1 · Accepted Answer · answered Oct 07 '21 at 14:12

1

The MutableSharedFlow.emit() documentation say that it is thread-safe. Maybe you were accidentally looking at FlowCollector.emit(), which is not thread-safe. MutableSharedFlow is a subtype of FlowCollector but promotes emit() to being thread-safe since it's not intended to be used as a Flow builder receiver like a plain FlowCollector. There's no reason to launch a coroutine just to emit to your shared flow.

There's no reason to call start() on a coroutine Job that was created with launch because launch both creates the Job and starts it.

You will need to declare flowToReturn before your launch call to be able to have it in scope to return from this outer function.

answered Oct 07 '21 at 14:12

Tenfour04

83,111
11
94
154

Yes, indeed I was looking at the wrong `emit`. "There's no reason to launch a coroutine just to emit to your shared flow." what about suspending on emit? I need to collect and `doStuff` regardless of the flow subscribers ability to consume. – Jakub Licznerski Oct 07 '21 at 14:23
2

Ok, I suppose that's one reason to launch a coroutine to emit. But then you're trading your buffer size for an unlimited number of coroutine Jobs, which are heavier. Alternatives would be to set the BufferOverflow strategy of the SharedFlow to drop oldest, or to use `tryEmit()`, which would drop an item from being emitted if the buffer is full. – Tenfour04 Oct 07 '21 at 14:36
yeah, good point but it seems like it's not possible to use a size unconstrained buffer. – Jakub Licznerski Oct 07 '21 at 14:41
@JakubLicznerski why would you want the buffer to be *actually* unconstrained? Wouldn't `MutableSharedFlow(extraBufferCapacity = Int.MAX_VALUE)` be sufficient (and probably equivalent for all intents and purposes) to ensure `doStuff` is called even with slow collectors on the shared flow? – Joffrey Oct 07 '21 at 14:48
the events are rather small data classes carrying a text message or objects. I don't suspect to receive much traffic within this connection – Jakub Licznerski Oct 07 '21 at 14:48
1

Also, I guess the buffer shouldn't even be that big because it should technically only be there to handle spikes. If your shared flow collectors are actually constantly slower than the rate of events, you will eventually run out of memory (and you'll run out earlier when using `launch`, as pointed out by @Tenfour04). – Joffrey Oct 07 '21 at 14:51
I was thinking of a buffer as a replay buffer in this case. I want all subscribers to receive all events accumulated since the flow start. I know that they will appear eventually so that the buffer won't grow indefinitely (they should be faster than events rate). Wouldn't `Int.MAX_VALUE` replay size be very costly as we would initialize a very big array internally? – Jakub Licznerski Oct 07 '21 at 14:57
A quick look at the source looks to me like it does. Maybe return `flowToReturn.buffer(capacity = Channel.UNLIMITED)` from your function? Downside is redundant buffers for each subscriber. – Tenfour04 Oct 07 '21 at 15:58
yeah, that looks like my case. Thanks! – Jakub Licznerski Oct 07 '21 at 16:06
*Wouldn't Int.MAX_VALUE replay size be very costly as we would initialize a very big array internally* - good point, sorry for suggesting that. – Joffrey Oct 07 '21 at 16:39

Should I emit from a coroutine when collecting from a different flow?

1 Answers1