Implement backoff strategy in flow

Question

I'm trying to implement a backoff strategy just using kotlin flow.

I need to fetch data from timeA to timeB

result = dataBetween(timeA - timeB)

if the result is empty then I want to increase the end time window using exponential backoff

result = dataBetween(timeA - timeB + exponentialBackOffInDays)

I was following this article which is explaining how to approach this in rxjava2.

But got stuck at a point where flow does not have takeUntil operator yet.

You can see my implementation below.

fun main() {
    runBlocking {
        (0..8).asFlow()
            .flatMapConcat { input ->
                // To simulate a data source which fetches data based on a time-window start-date to end-date 
                // available with in that time frame.
                flow {
                    println("Input: $input")
                    if (input < 5) {
                        emit(emptyList<String>())
                    } else { // After emitting this once the flow should complete
                        emit(listOf("Available"))
                    }
                }.retryWhenThrow(DummyException(), predicate = {
                    it.isNotEmpty()
                })
            }.collect {
                //println(it)
            }
    }
}

class DummyException : Exception("Collected size is empty")

private inline fun <T> Flow<T>.retryWhenThrow(
    throwable: Throwable,
    crossinline predicate: suspend (T) -> Boolean
): Flow<T> {
    return flow {
        collect { value ->
            if (!predicate(value)) {
                throw throwable // informing the upstream to keep emitting since the condition is met
            }
            println("Value: $value")
            emit(value)
        }
    }.catch { e ->
        if (e::class != throwable::class) throw e
    }
}

It's working fine except even after the flow has a successful value the flow continue to collect till 8 from the upstream flow but ideally, it should have stopped when it reaches 5 itself.

Any help on how I should approach this would be helpful.

score 0 · Answer 1 · answered Jul 21 '20 at 11:28

Maybe this does not match your exact setup but instead of calling collect, you might as well just use first{...} or firstOrNull{...} This will automatically stop the upstream flows after an element has been found.
For example:

flowOf(0,0,3,10)
    .flatMapConcat {
        println("creating list with $it elements")
        flow {
            val listWithElementCount = MutableList(it){ "" }  // just a list of n empty strings
            emit(listWithElementCount)
        }
    }.first { it.isNotEmpty() }

On a side note, your problem sounds like a regular suspend function would be a better fit. Something like

suspend fun getFirstNonEmptyList(initialFrom: Long, initialTo: Long): List<Any> {
    var from = initialFrom
    var to = initialTo
    while (coroutineContext.isActive) {
        val elements = getElementsInRange(from, to) // your  "dataBetween"
        if (elements.isNotEmpty()) return elements
        val (newFrom, newTo) = nextBackoff(from, to)
        from = newFrom
        to = newTo
    }
    throw CancellationException()
}

I hope I'm not too late for the party – Adrian K Jul 21 '20 at 11:29 — Adrian K, Jul 21 '20 at 11:29

Implement backoff strategy in flow

1 Answers1