0

I have got a successful connection between a Kafka producer and consumer on a Google Cloud Platform cluster established by:

$ cd /usr/lib/kafka
$ bin/kafka-console-producer.sh config/server.properties --broker-list \
PLAINTEXT://[project-name]-w-0.c.[cluster-id].internal:9092  --topic test

and executing in a new shell

$ cd /usr/lib/kafka
$ bin/kafka-console-consumer.sh --bootstrap-server \
PLAINTEXT://[project-name]-w-0.c.[cluster-id].internal:9092 --topic test \
 --from-beginning

Now, I want to send messages to the Kafka producer server using the following python script:

from kafka import *

topic = 'test'
producer = KafkaProducer(bootstrap_servers='PLAINTEXT://[project-name]-w-0.c.[cluster-id].internal:9092', 
api_version=(0,10))

producer.send(topic, b"Test test test")

However, this results in a KafkaTimeoutError:

"Failed to update metadata after %.1f secs." % (max_wait,))
kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs.

Looking around online told me to consider:

  • uncommenting listeners=... and advertised.listeners=... in the /usr/lib/kafka/config/server.properties file.

However, listeners=PLAINTEXT://:9092 does not work and this post suggests to set PLAINTEXT://<external-ip>:9092.

So, I started wondering about accessing a Kafka server through an external (static) IP address of the GCP cluster. Then, we have set up a firewall rule to access the port (?) and allow https access to the cluster. But I am unsure whether this is an overkill of the problem.

I definitely need some guidance to connect successfully to the Kafka server from the python script.

iJup
  • 281
  • 2
  • 13

2 Answers2

1

You need to set advertised.listeners to the address that your client connects to.

More info: https://rmoff.net/2018/08/02/kafka-listeners-explained/

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
0

Thanks Robin! The link you posted was very helpful to find the below working configurations.

Despite the fact that SimpleProducer seems to be a deprecated approach, the following settings finally worked for me:

Python script:

from kafka import *
topic = 'test'
kafka = KafkaClient('[project-name]-w-0.c.[cluster-id].internal:9092')
producer = SimpleProducer(kafka)

message = "Test"
producer.send_messages(topic, message.encode('utf-8'))

and uncomment in the /usr/lib/kafka/config/server.properties file:

listeners=PLAINTEXT://[project-name]-w-0.c.[cluster-id].internal:9092
advertised.listeners=PLAINTEXT://[project-name]-w-0.c.[cluster-id].internal:9092
iJup
  • 281
  • 2
  • 13