54

I am running kafka on ec2 instance. So amazon ec2 instance has two ips one is internal ip and second one is for external use.

I created producer from local machine, but it redirect to internal ip and give me connection unsuccessful error. Can anybody help me to configure kafka on ec2 instance, so that I can run producer from local machine. I am tried many combinations but didn't work.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
sms_1190
  • 1,267
  • 2
  • 12
  • 24

8 Answers8

49

In the Kafka FAQ (updated for new properties) you can read:

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in bootstrap.servers in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip and port to be registered in ZK by setting the advertised.listeners property in server.properties.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Lundahl
  • 6,434
  • 1
  • 35
  • 37
  • 6
    Just wanted to note that the mentioned hostname property in server.properties is called: `host.name=` – Marc Juchli Sep 23 '15 at 07:52
  • 2
    I am in the rare case: `In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.`. How do I do this? – Sodved Nov 30 '16 at 14:11
  • It's "host.name" not "hostname". – Alex Worden Mar 13 '18 at 17:00
  • 4
    `host.name` is [deprecated](https://kafka.apache.org/documentation.html#brokerconfigs) - as are `advertised.host.name` and `advertised.port` – Mr_and_Mrs_D Sep 14 '18 at 13:06
  • wanted to add a link to my blog post to compliment this answer https://boristyukin.com/connecting-to-kafka-on-virtualbox-from-windows/ – mishkin Oct 22 '18 at 18:49
  • @Mr_and_Mrs_D You are wrong. Because it worked out for me. – c0degeas Feb 07 '19 at 12:39
  • 1
    @c0de Deprecated means it won't work forever, but it currently works – OneCricketeer Feb 03 '20 at 15:10
40

I solved this problem, by setting advertised.host.name in server.properties and metadata.broker.list in producer.properties to public IP address and host.name to 0.0.0.0.

MirroredFate
  • 12,396
  • 14
  • 68
  • 100
sms_1190
  • 1,267
  • 2
  • 12
  • 24
  • @ sms_1190 ..Hi.. I am facing the same issues. I have kafka set up running in ec2 linux. I have written java code in eclipse installed in windows machine.I am able to create events in eclipse but i don't see them in kafka producer which is running in ec2. How to connect your eclipse or kafka code in maven in windows to connect to ec2 linux. I have modified advertised.host.name and advertised.port in kafka server config and also metadata.broker.list in producer as pub ip:0000. – user3858193 Jul 19 '15 at 03:16
  • @user3858193: Check your security groups in ec2. you should open 9092 port to 0.0.0.0/0. – sms_1190 Jul 20 '15 at 13:44
  • 1) can you please paste (those two lines) here the actual changes which you made in server.properties and producer.properties. 2) which kafka version are you using? Thanks in advance! – Dynamic Remo Jan 18 '17 at 18:23
  • @sms_1190 and did you have any properties file on your windows machine or every config file is only at the EC2? Thanks again. – Dynamic Remo Jan 18 '17 at 18:26
  • This absolutely solved one week of breaking the head. High five! Thank you! – Arun Apr 07 '20 at 03:03
25

The easiest way how to reach your Kafka server (version kafka_2.11-1.0.0) on EC2 from consumer in external network is to change the properties file 

kafka_2.11-1.0.0/config/server.properties

And modify the following line

listeners=PLAINTEXT://ec2-XXX-XXX-XXX-XXX.eu-central-1.compute.amazonaws.com:9092

Using your public address

Verified on 2.11-2.0.0

Shubham Rajvanshi
  • 43
  • 1
  • 9
Peter Koncz
  • 381
  • 4
  • 8
  • worked as a charm :-) ... thanks P. Koncz. You can also, via command line, override it `--override listeners=PLAINTEXT://ec2-XXX-XXX-XXX-XXX.eu-central-1.compute.amazonaws.com:9092` – Mahieddine M. Ichir Apr 30 '18 at 14:36
  • Hi, I have installed confluent-oss-5.0.0 on Azure VM and I am trying to access through Public IP to consume messages. Port is opened. NO LUCK, it's not working after changing all you have suggested. Getting below error :~/confluent-kafka/confluent-oss-5.0.0/bin$ kafka-console-producer --broker-list 13.71.115.20:9092 --topic pj_test123 >sdf [2019-03-25 18:23:11,288] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {pj_test123=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) – Priyabrata Mar 25 '19 at 13:09
16

I just did this in AWS. First get the Kafka server to listen on the correct interface/IP using host.name. For your case this would be the internal IP, not localhost, since your intent is for outside Kafka clients to connect. Any local clients will need to use that same address, not localhost.

Then set advertised.host.name to a host name, not an IP address. The trick is to get that host name to always resolve to the correct IP for both internal and external machines. I use /etc/hosts inside and DNS outside. See my full answer about Kafka and name resolution here.

Community
  • 1
  • 1
Cedar Myers
  • 418
  • 5
  • 5
6

If you want to access from LAN, change following 2 files-

  1. In config/server.properties:

    advertised.listeners=PLAINTEXT://server.ip.in.lan:9092

  2. In config/producer.properties:

    bootstrap.servers=server.ip.in.lan:9092

In my case, the server.ip.in.lan value was 192.168.15.150

Giacomo1968
  • 25,759
  • 11
  • 71
  • 103
Sharif Shahriar
  • 114
  • 1
  • 5
3

Below are the steps to connect Kafka from outside of EC2 instance.

  1. Open Kafka server properties file on EC2.

    /kafka_2.11-2.0.0/config/server.properties

  2. Set the value of advertised.listeners to

    advertised.listeners=PLAINTEXT://ec2-xx-xxx-xxx-xx.compute-1.amazonaws.com:9092

    This should be your Public DNS (IPv4) of EC2 instance.

  3. Stop Kafka server.

  4. Start Kafka server to see above configuration changes in action.

  5. Now you can connect to your Kafka of EC2 instance from outside or from your localhost.

    Tried and tested on kafka_2.11-2.0.0

Abhijeet Ashok Muneshwar
  • 2,480
  • 2
  • 31
  • 32
2

SSH to your EC2 instance or wheverver you're hosting Kafka.

sudo nano /etc/hosts

Add:

127.0.0.1 <your-host-name> localhost

In my case it's:

127.0.0.1 ec2-12-34-56-78.ap-southeast-1.compute.amazonaws.com

Save and exit.

mate.gvo
  • 1,093
  • 14
  • 20
  • Editing hosts file is a hack and should be avoided. `advertised.listeners` is the real solution to the problem, and is not specific to Unix environments – OneCricketeer Jun 25 '22 at 14:36
-1

For EC2 you should edit the /etc/hosts file to add:

XXX.XXX.XXX.XXX ip-YYY-YYY-YYY-YYY

where XXX... is your external IP and the ip-YYY-YYY-YYY-YYY is the string returned by the hostname command. You can use 127.0.0.1 instead of your external IP to communicate inside the server.

host.name is deprecated - as are advertised.host.name and advertised.port

Mr_and_Mrs_D
  • 32,208
  • 39
  • 178
  • 361