docker-compose kafka - local machine client cannot produce message to kafka

Question

I've read a lot of similar subjects but they aren't able to answer my problem here.

Trying to run some short integration tests, I'm using docker-compose 3, a single-node kafka. On client side I'm using Go shopify/sarama to consume / produce

zookeeper:
  image: confluentinc/cp-zookeeper:5.2.2
  hostname: zookeeper
  container_name: zookeeper
  ports:
    - "2181:2181"
  environment:
    ZOOKEEPER_CLIENT_PORT: 2181
    ZOOKEEPER_TICK_TIME: 2000
kafka:
  image: confluentinc/cp-enterprise-kafka:5.2.2
  hostname: kafka
  container_name: kafka
  depends_on:
    - zookeeper
  ports:
    - "29092:29092"
  expose:
    - 9092
  environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
    KAFKA_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

I have another container from the docker-compose that will listen to

- "BROKERS_URL=kafka:9092"

the consumer is working just fine:

Sarama consumer up and running. {"brokers": ["kafka:9092"], "topics": ["validated"], "group": "event-service"}

But on the producer part, running directly from my machine:

kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

producer, err := sarama.NewSyncProducer([]string{"http://localhost:29092"}, nil)
...
msg := &sarama.ProducerMessage{
    Topic: "validated",
    Key:   sarama.StringEncoder(""),
    Value: sarama.ByteEncoder(payload),
}

partition, offset, err := producer.SendMessage(msg)
...

Nothing weird / extravagante here, but it's not working and I'm confused.

also: nc -vz localhost 29092

Connection to localhost port 29092 [tcp/*] succeeded!

have you tried to use PLAINTEXT_HOST://kafka:29092 instead of PLAINTEXT_HOST://localhost:29092 — m303945, Mar 12 '20 at 12:50
What debugging did you do to verify `Is your cluster reachable?` — OneCricketeer, Mar 12 '20 at 14:04
I have a shopify/saram syncProducer trying to send a message to kafka (see code above) and I tried: nc -vz localhost 29092 which returns >Connection to localhost port 29092 [tcp/*] succeeded! — Jay Cee, Mar 12 '20 at 14:08

score 0 · Accepted Answer · answered Mar 12 '20 at 13:43

0

Instead of

    KAFKA_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

you need

    KAFKA_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://0.0.0.0:29092

Testing connectivity from my host machine using kafkacat shows that this works:

➜ kafkacat -b localhost:29092 -L
Metadata for all topics (from broker 1: localhost:29092/1):
 1 brokers:
  broker 1 at localhost:29092 (controller)
 0 topics:

This difference is that the listener is binding to all available interfaces (0.0.0.0). With your original configuration it binds to the loopback interface (lo) for localhost, and so only accepts traffic on this and not externally.

answered Mar 12 '20 at 13:43

Robin Moffatt

30,382
3
65
92

so, this leaded the producer to this message: > kafka: invalid configuration (Producer.Return.Successes must be true to be used in a SyncProducer) Which I did, but then same error `is your cluster reachable?` – Jay Cee Mar 12 '20 at 14:06
OK, that's a different question. In the spirit of keeping StackOverflow questions separate and useful, can you raise this as a new one? I've answered your original one and the mis-config will be useful for others who encounter it. – Robin Moffatt Mar 12 '20 at 14:14
so even if I still get `client has run out of available brokers to talk to (Is your cluster reachable?)` you think this is a sarama problem towards docker? – Jay Cee Mar 12 '20 at 14:18
Yes. My answer shows that your original config was broken and that it now works. The issue is thus with your Go code, and so should be a new question. You could refer to this one, obviously, for background. – Robin Moffatt Mar 12 '20 at 14:38

docker-compose kafka - local machine client cannot produce message to kafka

1 Answers1