running Kafka on WSL and make producer on windows

Question

I'm running Kafka on WSL. I'm trying to make simple producer like this (I'm using intellj)

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class ProducerDemo {

    public static void main(String[] args) {

        String bootstrapServers = "127.0.0.1:9092";

        //create Producer properties
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        //create the producer
        KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);

        //create a producer record
        ProducerRecord<String,String> record =
                new ProducerRecord<String, String>("first_topic","hallo world");

        //send data
        producer.send(record);

        //flush + close
        producer.flush();
        producer.close();
    }

}

but there's a problem, when I try to run the code it shows this error

[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error connecting to node AD17-2.localdomain:9092 (id: 0 rack: null)
java.net.UnknownHostException: No such host is known (AD17-2.localdomain)
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:932)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1505)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:851)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1495)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1354)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1288)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:293)
    at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:350)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:323)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
    at java.base/java.lang.Thread.run(Thread.java:832)

Process finished with exit code -1

i even checked the port on my Windows

TCP    0.0.0.0:9092           0.0.0.0:0              LISTENING

my question is, is it impossible to make producer on windows and while running kafka on WSL ?

Answer 1

Your Kafka broker is using the local hostname (AD17-2.localdomain) in its advertised listener. When you client connects to it initially on 127.0.0.1:9092 the broker returns this address for it to connect to for producing messages. Your producer fails to resolve this address, and fails.

To fix it, in the broker's server.properties set

advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

More info: https://rmoff.net/2018/08/02/kafka-listeners-explained/

Answer 2

Here's what worked for me for Apache Kafka 3.4.0:

1. Change Apache Kafka's server.properties
2. Ask WSL2 to prefer IPV4 over IPV6
3. Java code with producer to use WSL2 IP

Step 1

In WSL2 config/server.properties

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://[::1]:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT

Step 2

You also have to tell WSL2 to prefer IPV4 instead of IPV6[1], like so:

1. Edit the /etc/gai.conf

$ sudo vi /etc/gai.conf

2. At the end of the lines uncomment the last 3 lines, and you should have something like this:

#
# scopev4  <mask>  <value>
#    Add another rule to the RFC 6724 scope table for IPv4 addresses.
#    By default the scope IDs described in section 3.2 in RFC 6724 are
#    used.  Changing these defaults should hardly ever be necessary.
#    The defaults are equivalent to:
#
scopev4 ::ffff:169.254.0.0/112  2
scopev4 ::ffff:127.0.0.0/104    2
scopev4 ::ffff:0.0.0.0/96       14

Step 3

In Windows 10 Java Code:

1. Get the WSL2's IP first with this command: hostname -I | cut -d' ' -f1

        Properties props = new Properties();
        // WSL2 IP = 172.xx.xxx.xx:9092
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "172.xx.xxx.xx:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            ProducerRecord<String, String> message = new ProducerRecord<>("test-topic", "Hello, World!");
            producer.send(message);
        } catch (Exception e) {
            e.printStackTrace();
        }

Answer 3

It works for me with this solution: Whit the command: "ip addr | grep eth0" on Ubuntu terminal I can get external interface IP, I set this IP on advertised.listener and with the command:

netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=XXX.XX.XX.XX

on Windows cmd I can forward ports. With the command "netstat -ab" I can see on Windows cmd the IP and Port TCP 0.0.0.0:9092. It works correctly! Running Kafka Confluent Platform on WSL 2 (Ubuntu Distribution) and Spring application on Windows (Broker may not be available)