1

I am creating a service that sends lots of data to kafka-rest-proxy. I am only sending data (producing) to kafka. What I'm finding is that kafka-rest-proxy is easily overwhelmed and runs out of java heap space. I've allocated additional resources, and even horizontally scaled out the number of hosts running kafka-rest-proxy, yet I still encounter dropped connections and memory issues.

I'm not familiar with the internals of kafka-rest-proxy, but my hunch is that it's buffering the records and sending them to Kafka asynchronously. If that is the case then what mechanism does it have to control back pressure? Is there a way to configure it such that it writes records to Kafka synchronously?

jlunavtgrad
  • 997
  • 1
  • 11
  • 21

1 Answers1

0

Kafka REST Proxy exposes all of the functionality of the Java producers, consumer's command-line tools. Rest Proxy doesn't need any back pressure concept.

To be more specific, Kafka is capable of delivering messages over the network at an alarmingly fast rate.

You need to scale the brokers as per the rate you are producing and consuming the data.

Rohit Yadav
  • 2,252
  • 16
  • 18
  • Throughput tests on my kafka cluster show that it cannot ingest data as fast as my producers are capable of sending it. I don't mind if there is a bit of a bottleneck when things get bursty, but when kafka-proxy fails it causes bigger problems. A larger Kafka cluster comes with a lot of expense and that is not in the cards – jlunavtgrad Jan 13 '21 at 06:17
  • Other rest proxies are designed as ingress instead of for managing Kafka. They typically perform better under load. Zilla is one that has been optimized for this. https://docs.aklivity.io/zilla/latest/get-started/quickstart/rest.html – A.J. Jun 16 '23 at 15:21