1

I have installed confluent-oss-5.0.0 on Azure VM and exposed all necessary ports to access using public IP Address.

I tried to change the etc/kafka/server.properties below things to achieve but no luck

Approach - 1

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<publicIP>:9092

--------------------------------------            
Approach - 2

advertised.listeners=PLAINTEXT://<publicIP>:9092

--------------------------------------            
Approach - 3

listeners=PLAINTEXT://<publicIP>:9092

I experienced below error

pj@pj-HP-EliteBook-840-G1:~/confluent-kafka/confluent-oss-5.0.0/bin$ kafka-console-producer --broker-list <publicIp>:9092 --topic pj_test123>dfsds
[2019-03-25 19:13:38,784] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


pj@pj-HP-EliteBook-840-G1:~/confluent-kafka/confluent-oss-5.0.0/bin$ kafka-console-producer --broker-list <publicIp>:9092 --topic pj_test123
>message1
>message2
>[2019-03-25 19:20:13,216] ERROR Error when sending message to topic pj_test123 with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for pj_test123-0: 1503 ms has passed since batch creation plus linger time
[2019-03-25 19:20:13,218] ERROR Error when sending message to topic pj_test123 with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)




pj@pj-HP-EliteBook-840-G1:~/confluent-kafka/confluent-oss-5.0.0/bin$ kafka-console-consumer --bootstrap-server <publicIp>:9092 --topic pj_test123 --from-beginning
[2019-03-25 19:29:27,742] WARN [Consumer clientId=consumer-1, groupId=console-consumer-42352] Error while fetching metadata with correlation id 2 : {pj_test123=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

pj@pj-HP-EliteBook-840-G1:~/confluent-kafka/confluent-oss-5.0.0/bin$ kafka-console-consumer --bootstrap-server <publicIp>:9092 --topic pj_test123 --from-beginning
[2019-03-25 19:27:06,589] WARN [Consumer clientId=consumer-1, groupId=console-consumer-33252] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

All other service like zookeeper, kafka-connect and restAPI are working fine using the <PublicIP>:<port>

kafka-topics --zookeeper 13.71.115.20:2181 --list  --- This is working

Ref: Not able to access messages from confluent kafka on EC2

https://kafka.apache.org/documentation/#brokerconfigs

Why I cannot connect to Kafka from outside?

Solutions

Thanks, @Robin Moffatt, It works for me. I do below changes along with allowing all Kafka related ports on Azure networking

kafka@kafka:~/confluent-oss-5.0.0$ sudo vi etc/kafka/server.properties 

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://<privateIp>:9092,EXTERNAL://<publicIp>:19092
inter.broker.listener.name=INTERNAL
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Priyabrata
  • 629
  • 1
  • 8
  • 23

2 Answers2

1

You need to configure both internal and external listeners for your broker. This article details how: https://rmoff.net/2018/08/02/kafka-listeners-explained/.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • 1
    It works for me ``` kafka@kafka:~/confluent-oss-5.0.0$ sudo vi etc/kafka/server.properties listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092 listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://:19092 inter.broker.listener.name=INTERNAL ``` – Priyabrata Mar 26 '19 at 06:44
  • First I kept INTERNAL port as 19092 and changed few other places(etc/kafka/connect-distributed.properties, connect-standalone.properties, kafka-rest.properties, ksql-server.properties ...) with port 19092. But It did not work and new other issues come to the picture (like kafka-connect, shcema-registry not started). Also debezium mysql connector did not work. Any clue on this? – Priyabrata Mar 26 '19 at 06:52
0

You will also have to give public access to port 9092 (your broker). TO do that,

  1. Go to your Virtual machine in Azure portal
  2. Select Networking under settings in the left menu
  3. Add inbound port rule
  4. Add port 9092 to be accessbile from anywhere
Adx
  • 119
  • 2
  • 9