1

I have two VPCs in AWS:

VPC-A has an ec2 instance in it.
VPC-B has an ec2 instance in it running kafka and zookeeper via docker-compose

The VPCs are connected via AWS Privatelink (endpoint --> endpoint service --> nlb (in VPC-B) --> kafka) I have given the privatelink endpoint a DNS name: broker.confluent-playground

I can telnet fine to both port 9092 and 2181 from VPC A to Kafka and Zookeeper in VPC-B. No problems

[ec2-user@ip-10-1-0-90 etc]$ telnet broker.confluent-playground 9092
Trying 10.1.1.200...
Connected to broker.confluent-playground.

My problem is that while the network connectivity is there between the VPCs, I seem to be having problems with the kafka listener configuration. When I set the advertised.listeners to broker.confluent-playground:9092, my producer cannot seem to connect, and when running kafkacat -b broker.confluent-playground -L I only list 7 of the 40 topics. (they seem like internal system topics eg."_confluent_balancer_partition_samples").

Here is an excerpt from the docker-compose file:

broker:
    image: confluentinc/cp-server:6.1.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker:29092,EXTERNAL://broker.confluent-playground:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Now if I change the advertised listener to the private IP of the ec2 holding kafka ie:
FROM

EXTERNAL://broker.confluent-playground:9092

to

EXTERNAL://192.168.35.65:9092

then kafkacat in VPC-A can see all the 40 topics correctly! I still cannot produce (because i assume i have been given an advertised private ip address in a different network).

I seem to have these listener configurations messed up somewhere and I am confused how by changing the advertised listener I can get a subset of topics with one setting and then all the topics with another.

Another interesting thing
My kafkacat scanning using broker.confluent-playground advertised listener returns this

[ec2-user@ip-10-1-0-90 ~]$ sudo docker run --rm --network=host edenhill/kafkacat:1.5.0 kafkacat -b broker.confluent-playground:9092 -L

Metadata for all topics (from broker 1: broker.confluent-playground:9092/1):
 1 brokers:
  broker 1 at broker.confluent-playground:9092 (controller)
 7 topics:

And when I use the private IP address as (that I cannot reach from VPC-A) as the advertised listener I get the /bootstrap with all the topics: 

[ec2-user@ip-10-1-0-90 ~]$ sudo docker run --rm --network=host edenhill/kafkacat:1.5.0 kafkacat -b broker.confluent-playground:9092 -L
Metadata for all topics (from broker -1: broker.confluent-playground:9092/bootstrap):
 1 brokers:
  broker 1 at 192.168.54.226:9092 (controller)
 40 topics:
Alexander Witte
  • 195
  • 1
  • 11
  • Perhaps you have more than one EC2 instance behind the same NLB, but the Kafka brokers are unaware of each other as they are deployed inside Docker containers on each EC2 instance? If topics were created on only one of these instances, then connecting directly would show all of the topics, but connecting via NLB could send you to the other EC2 instance instead with only system level topics. Consider using tcpdump to capture Kafka protocol for a more detailed diagnosis. – HumblePuzzler Sep 02 '21 at 17:16

0 Answers0