Connect to Kafka inside VirtualBox

Question

I'm having troubles connecting from my Host (Windows) to Guest (Linux) where I installed Kafka.

I have set up a VM (with VirtualBox) where I installed Confluent tools. In this VM, I run the command:

confluent start schema-registry

It starts zookeeper, kafka, and schema-registry.

Under this VM, I can run

kafka-console-producer --broker-list localhost:9092 --topic test

and

kafka-console-consumer --bootstrap-server localhost:9092 --topic test

and everything is working fine, I can produce and receive messages.

My goal is however to be able to produce and consume messages from my Host, so I setup this port forwarding rule :

From my Windows, I was expecting this command to work:

bin\windows\kafka-console-producer.bat --broker-list 127.0.0.1:9092 --topic test

But all I get is this :

ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 1529 ms has passed since batch creation plus linger time

I've tried many different stuff but still can't find a solution.... Any ideas?

score 5 · Accepted Answer · edited Nov 29 '17 at 14:39

5

It seems to be a host name gap between Guest and Host. What host name does the Linux Guest have?

When producer/consumer accesses the Kafka broker, the Kafka broker returns its host name for data producer or consumer at default settings. So producers/consumers need to resolve broker's host name to IPAddress.

For broker returning an arbitrary host name, use the advertised.listeners settings.

Details are in the "advertised.listeners" configuration docs.

edited Nov 29 '17 at 14:39

Jeroen

60,696
40
206
339

answered Sep 06 '17 at 22:33

kimutansk

131
1
3

Thanks a lot for your help, I'm really feeling lost here. The hostname of the Linux Guest is kafka-VirtualBox. In the advertised.listeners, I now have this: advertised.listeners=PLAINTEXT://kafka-VirtualBox:9092 And in the Windows Host, I added this line to the hosts file : 127.0.0.1 kafka-VirtualBox I still have the same error though.... – Florian Sep 07 '17 at 14:21
Hmm... Then, it needs to resolve step by step. First, confirm "Are kafka commands enable to access kafka-VirtualBox?" by kafka-topics.bat. – kimutansk Sep 09 '17 at 22:24
I posted on the way. Then, it needs to resolve step by step. First, confirm "Are kafka commands enable to access kafka-VirtualBox?" by kafka-broker-api-versions.bat. If commands can't access to kafka-VirtualBox, this trouble is virtual box network settings problem. If commands can access to kafka-VirtualBox, this trouble becomes kafka broker or client settings problem. – kimutansk Sep 09 '17 at 22:36
1

Thanks a lot for your help. The solution was to add this : 127.0.0.1 kafka-VirtualBox to my windows hosts file. You were probably right saying that my host was not able to resolve my guest hostname. I re-installed a new guest from scratch to be sure, and by using the default confluent config, everything is working fine if you redirect zookeeper, kafka, and schema-registry ports, and if you add this line to your host's hosts file. Thanks again! – Florian Sep 10 '17 at 09:57
wanted to add a link to my post as well to compliment original answer https://boristyukin.com/connecting-to-kafka-on-virtualbox-from-windows/ – mishkin Oct 22 '18 at 18:48

score 0 · Answer 2 · answered Nov 30 '17 at 10:57

I was trying to set up something similar. Although it's probably better to set up proper virtual networks for Kafka and Zookeeper, I did push on and want to share my personal approach.

I've got things to work with Kafka and Zookeeper running on the VirtualBox Linux guest itself, as well as a docker-based solution running on that Linux guest. Here's the docker-compose.yml I used inside the guest:

version: '3'

services:

  zookeeper:
    image: "wurstmeister/zookeeper"
    ports:
      - "2181:2181"

  kafka:
    image: "wurstmeister/kafka"
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "localhost"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

The crux here is the KAFKA_ADVERTISED_HOST_NAME, which does a dirty little trick that plays well with the default NAT-based networking in VirtualBox (assuming you've patched 2181 and 9092 through as OP did). With this trick any consumer or producer on the VirtualBox Host OS will learn from Kafka that it is located on "localhost", which resolves to the Host, but because the port on the host is patched through to the guest things end up in the right place.

The proper way to do the same in 1.x is to use the Advertised Listeners properties, but the principle remains the same.

For reference, here's my VirtualBox settings:

With those settings, after docker-compose up on the guest OS, I could list/create topics and consume/produce messages from the host OS.

See my answer at https://stackoverflow.com/questions/50204667/how-do-you-configure-vagrant-so-host-can-communicate-with-kafka-docker-running-o/50277507#50277507 — Bertrand_Szoghy, May 10 '18 at 16:33

Connect to Kafka inside VirtualBox

2 Answers2