Unable to produce to Kafka topic that is running on WSL 2 from Windows

Question

I am running the latest Kafka on Ubuntu WSL2 successfully. I can start zookeeper, kafka server, create topics, console produce and console consume just fine from within the Ubuntu that I have running on the WSL. However, when I go into my Intellij on Windows and create a simple Java Producer it does not seem to be able to connect to the broker

Versions & Hostname

    Java version: 1.8
    Kafka Version: 2.6
    hostname (from Ubuntu): KDAAPPDEV04
    hostname (from Powershell): KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
    netstat from CMD:
        TCP    [::1]:9092             [::]:0                 LISTENING

server.properties I found this settings on another SO answer but these did not work for me.

advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

then tried (and restarted zookeeper and kafka)

advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

Producer

I run this producer with three different values: hostname, localhost and 127.0.0.1 but it never connects to the broker

    public class ProducerDemo{

    private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);

    public static void main(String[] args) throws UnknownHostException{

        System.out.println(InetAddress.getLocalHost().getHostName());
        System.out.println(InetAddress.getLocalHost().getCanonicalHostName());

        String bootstrapServers = "127.0.0.1:9092";
//        String bootstrapServers = "localhost:9092";
//        String bootstrapServers = "KDAAPPDEV04:9092";

        //create Producer properties
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        //create the producer
        KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);

        //create a producer record
        ProducerRecord<String,String> record = new ProducerRecord<String, String>("first-topic","hola mundo");

        //send data
        producer.send(record);

        //flush + close
        producer.flush();
        producer.close();
    }
}

Error

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) could not be established. Broker may not be available.

score 26 · Answer 1 · answered Feb 26 '21 at 02:04

26

Had this same issue. The root cause seems to be that WSL2 is broken with regards to IPv6 and localhost (See: https://github.com/microsoft/WSL/issues/4851)

The only fix I found that doesn't involve changing configs every time you reboot (per the "172.*" suggestion above) is to use the IPv6 loopback address ::1 in both the Kafka server config running in Linux and the Java client in Windows.

In server.properties I have this:

listeners=PLAINTEXT://[::1]:9092

And likewise in my Java client bootstrap server config I use

"[::1]:9092"

answered Feb 26 '21 at 02:04

Ed P

361
3
3

2

This was a great solution. up voted. Thanks! – jdawiz May 06 '21 at 20:37
4

Thank you. A light note, inside the WSL2, we'll need to change the bootstrap server declaration in the CLI as well. E.g: `kafka-console-consumer.sh --bootstrap-server [::1]:9092 --topic my_topic` – ThangLeQuoc Jun 10 '21 at 01:29
I am running the confluent platform in wsl - kafka runs after changing this, but the connect service does not run even after changing the bootstrap server props in connect related services – Hiresh Feb 04 '22 at 14:17
It should be noted however that the example applications (e. g. https://kafka.apache.org/31/documentation/streams/quickstart) don't work out of the box, as their code still uses localhost:9092 as host connection – Walnussbär Apr 26 '22 at 11:56

score 12 · Answer 2 · answered Jan 03 '21 at 18:50

I had the exact problem you are having and I resolved it as follows:

I ran the following command in my WSL2 Ubuntu shell: ip addr | grep "eth0" I made note of the ip address against the inet property, for example, 172.27.10.68
In my Kafka server.properties I replaced the listeners property value as follows: listeners=PLAINTEXT://172.27.10.68:9092 I commented out the advertised.listeners property. But you can alternatively assign the ip in question to this property, and have the listeners property set to 0.0.0.0. But I assume you are using the Kafka installation for testing/learning purposes, so I would keep it simple.
I made no change to the Zookeeper's default ip:port
I am using the Schema Registry, so I modified the Kafka bootstrap property as follows: kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092 I made no change to the default schema registry listener listeners=http://0.0.0.0:8081
I used the same ip (as listed above) in my IntelliJ Kafka Producer. It then happily connected to my Kafka broker in WSL2.

More information on WSL2 networking can be found at https://learn.microsoft.com/en-us/windows/wsl/compare-versions .

The only problem with this setup is that every time you shutdown or restart your Windows machine, or close your Ubuntu terminal, the ip address for eth0 changes. And this results in redoing steps 2, 4 and 5. I am sure there is a better way, but everything I tried failed, except for this.

It works. But I had an issue with my connection even after this changes. Then I looked on ubuntu kafka server.log and there was an error 'ERROR Processor got uncaught exception. (kafka.network.Processor)', and I've found that the issue also can be caused by kafka-clients version missmatch in your maven dependecies. I'am also recommend check this — Andrew, Mar 06 '23 at 17:56

score 3 · Answer 3 · answered Mar 19 '22 at 12:59

3

Stop Kafka and Zookeeper, then

Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Start Kafka, and you're good to go!

answered Mar 19 '22 at 12:59

Stephane Maarek

5,202
9
46
87

This workaround seems best, since it's a one-time fix applied to the server and doesn't require changing client config. Also keeps the fix confined to WSL itself, which is where the bug supposedly is. – Kyle McClellan Mar 22 '23 at 21:44

score 2 · Answer 4 · answered May 28 '21 at 11:47

WSL2 runs on hypervisor and you need port proxy to connect Kafka Broker running on WSL2.

Step 1 . Check you WSL2 IP using following command and copy inet value

$ ifconfig
inet 172.X.X.X

Step 2. Open cmd with Admin permsissions

   netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X

You should be able to connect now

Note : WSL2 IP changes everytime you restart machine

score 2 · Answer 5 · answered Jan 21 '22 at 17:34

I am able to find a work around . Thanks to Goose's comments

I ran the following command in my WSL2 Ubuntu shell: ip addr
Then ip address against the inet property global eth0 . for example, inet 172.20.XXX.XXX/20 .... scope global eth0
I replaced all localhost with this IP address in the docker-compose.yml
I replaced the localhost with this IP address in springboot yml or properties file.
My Kafka producer and consumer able to connect to the Kafka running in Ubunti - WSL 2 from Windows

denise · Answer 6 · 2022-09-26T15:51:28.643

1

I got this problem when running a kafka producer in IntelliJ and a consumer in ubuntu terminal while on WSL2.

First, stop Kafka and Zookeeper. Then run these commands on WSL2, one by one:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

After that, in the kakfa folder, go to config/server.properties and edit the file to add the line:

listeners=PLAINTEXT://localhost:9092

When these commands have succeeded relaunch zookeeper and kafka.

https://www.conduktor.io/kafka/kafka-fundamentals

edited Sep 26 '22 at 15:51

answered Sep 21 '22 at 20:45

denise

21
2

1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Sep 26 '22 at 13:44

score 0 · Answer 7 · answered Nov 19 '20 at 23:30

This is not the optimal solution, but you will be able to connect if you run your producer in Ubuntu/WSL. This means if you are using a Windows IDE, writing the code, switching to Ubuntu and using a command line compiler and running the producer. See this post Error connecting to kafka server via IDE in WSL2

score 0 · Answer 8 · answered Jan 23 '23 at 10:12

0

Edit the file etc/sysctl.conf and add following lines in it.

net.ipv6.conf.all.disable_ipv6=1

net.ipv6.conf.default.disable_ipv6=1
Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.
Update the sysctl config by using the following command. (Everytime you restart your machine this command needs to be run to update the configuration)

sudo sysctl -p

answered Jan 23 '23 at 10:12

Vijay Chaudhari

41
3

I was strugguling and tried lot of option. Finally change server.properties as below. listeners=PLAINTEXT://:9092, advertised.listeners=PLAINTEXT://172.25.205.68:9092 please make note 172.25.205.68 need to be replace with your wsl2 address. and make chanes in java code like advertised.listeners=PLAINTEXT://172.25.205.68:9092 . That will work. – omankame Jun 10 '23 at 08:34

score 0 · Answer 9 · edited Aug 12 '23 at 00:40

0

Step 1 Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Step 2:

In \config\server.properties under Kafka folder Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.

Step 3: In WLS2 Ubuntu,

sudo sysctl -p

After making above changes it worked for me in my local

edited Aug 12 '23 at 00:40

toyota Supra

3,181
4
15
19

answered Aug 11 '23 at 22:30

Simon Sundarraj

1
1

Unable to produce to Kafka topic that is running on WSL 2 from Windows

9 Answers9

Linked