0

The below are my Kafka broker configuration

broker.id=1
port=9092
host.name=127.0.0.1
advertised.listeners=PLAINTEXT://127.0.0.1:9092
listeners=PLAINTEXT://127.0.0.1:9092

Console producer and consumer are working perfectly but when I try to connect through java, it is throwing broker not available error. But Kafka broker is running and able to produce and consume messages through console.

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

KafkaProducer<String, String> producer = new KafkaProducer<String,String>(props);

producer.send(new ProducerRecord<String, String>("Sample","Hey","From java program"));
producer.close();

[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker 127.0.0.1:9092 (id: -1 rack: null) disconnected

I'm using STS IDE to write java program and using JDK 1.8, kafka 2.8.1 windows OS and using ubuntu 20.04.4 LTS to execute console producer and consumer.

Thanks in advance !

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Soundarya Karthikeyan
  • 1
  • 1
  • Where is your Kafka broker running? – Robin Moffatt Mar 18 '22 at 10:47
  • my kafka is running on this port in my local 127.0.0.1:9092 – Soundarya Karthikeyan Mar 18 '22 at 10:56
  • 1
    right, but which machine? You've mentioned windows and ubuntu – Robin Moffatt Mar 18 '22 at 13:52
  • Started kafka server using Ubuntu and tested console producer and consumer. Written sample kafka producer and consumer in STS (in windows 10) and trying to connect to the same kafka which was started using ubuntu LTS. Do i need to start the same kafka in windows again using .bat files and then try connecting through java kafka producer? – Soundarya Karthikeyan Mar 20 '22 at 14:55
  • That wouldn't be recommended, no. You'll have other issues running Kafka in windows directly. Instead, if you really don't want to deal with networking configurations, you need to run your code in WSL2 as well. In other words, localhost is not the same between Windows and the hypervisor running Kafka. – OneCricketeer Mar 20 '22 at 16:05
  • Thank you so much OneCricketeer. I'm able to make it work if I run the code in WSL2. – Soundarya Karthikeyan Mar 21 '22 at 06:13

2 Answers2

2

Too late for this response, but hope this helps someone.

  • Identify the ip address of the WSL2 using ifconfig or ip addr or hostname -I

  • From the windows command prompt (as admin) run below command

> netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X

172.X.X.X is the Ip of the WSL2

With that change I was able to access the Kafka cluster.

Reference link

Tim
  • 1,321
  • 1
  • 22
  • 47
0

I have answered it here: https://stackoverflow.com/a/76325645/4594452

Here's the answer from that link:

Here's what worked for me for Apache Kafka 3.4.0:

  1. Change Apache Kafka's server.properties
  2. Ask WSL2 to prefer IPV4 over IPV6
  3. Java code with producer to use WSL2 IP

Step 1

In WSL2 config/server.properties

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://[::1]:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT

Step 2

You also have to tell WSL2 to prefer IPV4 instead of IPV6[1], like so:

  1. Edit the /etc/gai.conf

$ sudo vi /etc/gai.conf

  1. At the end of the lines uncomment the last 3 lines, and you should have something like this:
#
# scopev4  <mask>  <value>
#    Add another rule to the RFC 6724 scope table for IPv4 addresses.
#    By default the scope IDs described in section 3.2 in RFC 6724 are
#    used.  Changing these defaults should hardly ever be necessary.
#    The defaults are equivalent to:
#
scopev4 ::ffff:169.254.0.0/112  2
scopev4 ::ffff:127.0.0.0/104    2
scopev4 ::ffff:0.0.0.0/96       14

Step 3

In Windows 10 Java Code:

  1. Get the WSL2's IP first with this command: hostname -I | cut -d' ' -f1
        Properties props = new Properties();
        // WSL2 IP = 172.xx.xxx.xx:9092
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "172.xx.xxx.xx:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            ProducerRecord<String, String> message = new ProducerRecord<>("test-topic", "Hello, World!");
            producer.send(message);
        } catch (Exception e) {
            e.printStackTrace();
        }
Miko Chu
  • 1,087
  • 14
  • 22