0

I did a test with code below to send data to topic. The kafka is 

kafka_2.12-1.1.0

The code are 

import kafka
print(kafka.version.__version__)
from kafka import KafkaProducer 
producer = KafkaProducer(
    bootstrap_servers=['172.25.44.238:9092'],
    sasl_mechanism="PLAIN",
    api_version=(0, 10),
    retries=2
)
f = producer.send("test", "some")
f.get()

If I change the server config like this:

listeners=PLAINTEXT://172.25.44.238:9092

Then my code can send data to my topic

If I change the server config like this which is default:

listeners=PLAINTEXT://:9092

Then my code will hit error:

kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Batch for TopicPartition(topic='test', partition=0) containing 1 record(s) expired: 30 seconds have passed since batch creation plus linger time

The difference is that the sencond will use hostname by default. And yes my machine running the producer code can not reslove the kafka hostname. But I did not use the hostname in producer code either. So it should not cause the error. So why the hostname matter?

Kramer Li
  • 2,284
  • 5
  • 27
  • 55
  • Where do you get "the second will use hostname by default"? I was pretty sure it would use `localhost`. – daniu Jan 13 '20 at 08:13

1 Answers1

0

I think you're misunderstanding the concept of "bootstrapping".

The address you provide only establishes initial connection. The address the clients actually use is defined by the advertised.listeners.

The listeners should always be ://0.0.0.0, in my opinion, then you use OS level firewall settings to restrict access. Yes, the default is the hostname, and this means only that host can communicate with the broker

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Not really - it's perfectly valid to configure several listeners, [for example in a Docker setup](https://rmoff.net/2018/08/02/kafka-listeners-explained/). – daniu Jan 13 '20 at 12:57
  • 1
    @daniu `listeners` should always be `0.0.0.0`... There is never a reason to use the Docker service name as the [bind address](https://en.wikipedia.org/wiki/0.0.0.0#As_a_host_address). **`advertised.listeners`** should always point to a resolvable address by any potential client . – OneCricketeer Jan 13 '20 at 20:53
  • Hm, then why does theconfiguration item exist in the first place? I was assuming it was supposed to be used to limit access, eg for security reasons. Wouldn't I need to configure two non-`0.0.0.0` interfaces if I get inbound traffic from one subnet and route traffic into another? – daniu Jan 15 '20 at 10:45
  • @daniu IMO, you can use OS-level firewalls to limit interfaces. Sure, you can use `listeners=192.168.0.0` to bind to that subnet, but that's not very common – OneCricketeer Jan 15 '20 at 14:01