When does AkkaHttp backpressure kick in?

Question

.. when the http response entity is not consumed, or the client tcp buffer becomes full, or when the rate of client taking from its tcp buffer is lower then the rate of server pushing data to it?

I am looking for a way for to achieve the following:

Let's assume that there is a backpressure-able source of data on the server, such as an Apache Kafka topic.

If I consume this source from a remote location it may be possible that the rate at which that remote location can consume is lower - this is solved if Kafka client or consumer is used.

However let's assume that the client is a browser and that exposing direct Kafka protocol / connectivity is not a possibility.

Further, let's assume that there is a possibility of getting all the value even if jumping over some messages.

For instance in case of compacted topics, getting only the latest values for each key is enough for a client, no need to go through intermediate values.

This would be equivalent to Flowable.onBackpressureLatest() or AkkaStreams.aggregateOnBackpressure or onBackpressureAggregate.

Would it be a way to expose the topic over HTTP REST (e.g. Server Side Events / chunked transfer-encoding) or over web-sockets, that would achieve this effect of skipping over intermediate values for each key?

Please advise, thanks

Answer 1

Akka http supports back pressure based on TCP protocol very well and you can read about using it in combination with streaming here

Kafka consumption and exposure via http with back pressure can be easily achieved in combination of akka-http, akka-stream and alpakka-kafka.

Kafka consumers need to do polling and alpakka covers back pressure with reduction of polling requests.

I don't see the necessity of skipping over the messages when back pressure is fully supported. Kafka will keep track of the offset consumed by a consumer group (the one you pick for your service or http connection) and this will guarantee eventual consumption of all messages. Of course, if you produce messages way faster in a topic, the consumer will never catch up. Let me know if this is your case.

As a final note, you may check out Confluent REST Proxy API, which allows you to read Kafka messages in a restful manner.