0

I have been struggling to understand how the following two lines of configuration are interpreted.

KAFKA_CFG_LISTENERS: EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL_SAME_HOST://localhost:29092

These lines come from a docker-compose.yml file which configures Kafka environment variables for a Bitnami Kafka container.

I understand that for both environment variables, KAFKA_CFG_LISTENERS and KAFKA_CFG_ADVERTISED_LISTENERS that the names which follow are arbitrary.

For example, here INTERNAL and EXTERNAL_SAME_HOST are specified, but these could be literally any string so I don't understand how they have a meaningful interpretation to the Kafka process.

What is the difference between these two configuration variables (KAFKA_CFG_LISTENERS, KAFKA_CFG_ADVERTISED_LISTENERS) and how are they interpreted?

FreelanceConsultant
  • 13,167
  • 27
  • 115
  • 225

1 Answers1

0

They're parsed at runtime of the container (with bash, usually) and templated into server.properties. The bitnami containers accept both KAFKA_CFG_ and KAFKA_ prefix in some cases, so best to focus on the actual broker config, rather than the container image setup.

how they have a meaningful interpretation to the Kafka process

You also need KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP

You can read about advertised.listeners (the address clients get in a response after bootstrapping to the cluster) and listeners (the server bind address, where bootstrap requests are accepted), and listener.security.protocol.map at https://kafka.apache.org/documentation/#brokerconfigs

And in further details at

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • This doesn't answer my question. I am asking how do they work? What does the Kafka executable do with them when it reads their values? What is the point of them? – FreelanceConsultant Apr 23 '23 at 07:51
  • I'm also aware of the security protocol environment variable but I don't think it's relevant to what I am asking here because it doesn't really matter which authentication type is associated with each one. Afaik... – FreelanceConsultant Apr 23 '23 at 07:52
  • Another way to think about it is this. Let's say the string used is "EXTERNAL". This is obviously some kind of key and it groups different parts of the config together. There is a security protocol defined as well as "advertised listeners" and "normal listeners" ... Basically what are those things? What does putting a particular hostname and port combination for "advertised listeners" and the other type of "listeners" do? – FreelanceConsultant Apr 23 '23 at 08:03
  • As the documentation says, advertised ones get stored in Zookeeper **for clients to use**. Example - you have a 3 node cluster. You only set `bootstrap-server=192.168.1.1:9092` but **all 3** get returned in a network response. You're connecting on `listeners` to one server but returned every broker `advertised.listeners` for the whole cluster... That's explained fairly well in the Confluent blog above... Regarding the name, PLAINTEXT is still a "security protocol" kafka will not accept an unmapped listener named EXTERNAL. It must be mapped to one of the option listed in the documentation – OneCricketeer Apr 24 '23 at 14:31
  • Sorry to say that I don't understand a word of what you are trying to say here. What you have written is so vague and unclear that I have no idea how to interpret your comment. I can *guess* at what you are trying to say, but I can imagine several possible interpretations of the above comment within just a few seconds of reading it. That doesn't help me to understand what you are trying to say. – FreelanceConsultant Apr 24 '23 at 14:40
  • It's networking config that is unique to kafka, and unintentionally complicated, IMO... But which part is vague? I don't know how else to explain it. Try creating your own cluster with multiple nodes. Use commands `nc -vz` and `netstat` commands to check the host+ports set in `listeners`. Then compare that to the output of `advertised.listeners`, returned by `kcat -L`. – OneCricketeer Apr 24 '23 at 14:48