5

I'm developing a streaming API with Apache Kafka version (2.1.0). I have a Kafka cluster and an external server. The external Server will be producing data to be consumed on the Kafka cluster.

Let's denote the external Server as E and the cluster as C . E doesn't have Kafka installed. I run on it a JAR file to produce messages. Here is the snippet for Producer properties: 

properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "bootstrapIp:9092");
properties.put(ProducerConfig.CLIENT_ID_CONFIG, "producer");

I set bootstrapIp to the Kafka broker IP.

In the cluster side, I start the consumer console using this command:

kafka-console-consumer --bootstrap-server bootstrapIp:9092 --topic T1 --from-beginning

I set bootstrapIp to the cluster bootstrap server IP.

When running the producer and the consumer on the cluster, it works very fine, but when i run the producer in the external server (E) and the consumer in the cluster (C) the data not being consumed.

In localhost every thing is working fine also when i run the producer and the consumer in the cluster (C) everything is working fine, when running the producer externally i can't consume the data in the cluster.

The ping from cluster(C) to external server (E) is working, but i can't see where the problem is exactly.

I am not able to figure out how to consume messages from an external server.

EDIT

From the external server (E) i telnet the (bootstrapIp): telnet bootstrapIp 9092 and it works, i don't understand the problem

Joe jim
  • 59
  • 3
  • 1
    Did you try pinging from 'E' to 'C' ? I think it might be some DNS issue. Also I will suggest you to configure the Logger in your producer code and run with DEBUG level log. – mrnakumar Mar 06 '20 at 14:07
  • Are you sure that the "external" producer can connect and send messages to Kafka? Can you run `$ kafka-topics.sh --bootstrap-server bootstrapIp:9092 --describe --topic T1`, and share producer logs as well. – mazaneicha Mar 06 '20 at 14:07
  • @mrnakumar yes the ping is working from E to C and from C to E – Joe jim Mar 06 '20 at 15:08
  • 1
    @mazaneicha this is the problem, the "external" producer cannot connect to kafka, but when i run the producer and the consumer inside the cluster they can connect – Joe jim Mar 06 '20 at 15:10
  • 1
    In your question, you're making a distinction between broker IP and "boostrap server" IP. Are they really different? – mazaneicha Mar 06 '20 at 16:02
  • By the way, running Kafka clients on the actual brokers is discouraged – OneCricketeer Mar 06 '20 at 18:17
  • @mazaneicha no they aren't, they are the same i edited the question – Joe jim Mar 08 '20 at 08:39
  • Did you find a solution? I'm trying to run a producer from a remote server and I have the same problem. I'm a noob in kafka. My producer tries to connect to localhost even if I give the ip address of my server. – Danielle Paquette-Harvey Mar 05 '21 at 19:59

1 Answers1

1

Tbis works for me: From server.properties Uncomment

listeners=PLAINTEXT://:9092

And

advertised.listeners=PLAINTEXT://<HOST IP>:9092

Replace with actual IP. My case:

advertised.listeners=PLAINTEXT://192.168.75.132:9092
jpm1590
  • 11
  • 1