0

As part of my existing application (which runs in the public network), I want to send some messages to AWS MSK Kafka cluster. As my Kafka cluster is within a VPC, I couldn't connect to it directly. That I understood. In my research, I came across a pattern - by running confluent's kafka-rest on a ec2-instance within same VPC having a public subnet. But my application needs to make HTTP calls to send the messages to my kafka cluster, which adds extra latency.

I'm trying to figure out a way by having some persistent connection and send messages to MSK cluster.

Any Thoughts Please!!

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Venkat Papana
  • 4,757
  • 13
  • 52
  • 74
  • If you are not able to (securely) open the Kafka TCP port on the VPC, then you're going to need some sort of proxy, whether that is the REST Proxy or something else. – OneCricketeer Jul 12 '21 at 23:42
  • Even, I tried by running kafka-proxy on a ec2 within same VPC, but when I send a HTTP call to it, failing with `ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': java.lang.OutOfMemoryError: Java heap space`. This is on xlarge instance. – Venkat Papana Jul 13 '21 at 17:57
  • The instance size doesn't matter if you don't increase the heap space of the JVM – OneCricketeer Jul 13 '21 at 18:02
  • 1
    After updating heap size from 256M to 1024M, now I'm getting a different error response. `INFO [AdminClient clientId=adminclient-1] Metadata update failed org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1626200389678, tries=1, nextAllowedTryMs=1626200389813) timed out at 1626200389713 after 1 attempt(s)` – Venkat Papana Jul 13 '21 at 18:22
  • Seems like a network issue. You can increase the retries or the timeouts – OneCricketeer Jul 13 '21 at 18:24
  • Updated below properties from 1000 to 10000, but no luck. `consumer.request.timeout.ms = 10000 simpleconsumer.pool.timeout.ms=10000` – Venkat Papana Jul 13 '21 at 18:34
  • Those are consumer properties. Your error is from the adminclient, which has separate configs – OneCricketeer Jul 13 '21 at 20:20

0 Answers0