error while starting kafka broker

Question

I was able to successfully set up zookeeper and one kafka broker yesterday. Everything worked as expected. I shut down kafka (ctrl + c) and then zookeeper.

Today I started zookeeper and when I started kafka (bin/kafka-server-start.sh config/server0.properties), I get the following error. I tried various remedies suggested (removing completely my kafka installation and doing it again from scratch). Still I get same error.

[2016-09-28 16:15:55,895] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
        at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)
        at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)
        at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
        at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:51)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:244)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
        at kafka.Kafka$.main(Kafka.scala:67)
        at kafka.Kafka.main(Kafka.scala)
[2016-09-28 16:15:55,896] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

All set up in mac

SRJ · Answer 1 · 2021-02-11T06:12:34.107

24

I have faced the same issue while setting up the kafka and zookeeper multinode cluster.

Root Cause :

As zookeeper ensemble takes some time to setup so when we start kafka brokers, all zookeeper nodes might have not joined zookeeper ensemble yet.

So It may be the case that one kafka node connected to a zookeeper node which is not yet in the ensemble.

Another reason can be kafka broker is killed instead of stopped. When you kill your kafka process or it crashes and it does not close its zookeeper connection correctly.

Solution :

To avoid this situation make sure your zookeeper ensemble is up and running before starting kafka brokers.

Add proper waits between zookeeper and kafka broker services start and stop.

edited Feb 11 '21 at 06:12

answered Dec 19 '17 at 09:56

SRJ

2,092
3
17
36

5

But how do you fix the issue once it occurs? – Shane Gannon Jan 11 '18 at 16:06
You can delete the node in zookeeper that's listed in the log. I'm surprised that the node isn't an ephemeral node cleaned on disconnect. – JasonG Oct 09 '18 at 15:52
1

Go to zookeeper.properties and find the zookeeper data dir. Delete the contents of data dir in zookeeper. Restart zookeeper and kafka. It will work – SHM Dec 06 '19 at 04:42
1

I am in windows environment and there is a point, I think, where you say: Anothre reason...broker killed; there is a bat for stopping kafka and for stopping zookeeper and I did nou use them - so I fall into that problem. Now I know that each time I stop the proceses - not killing them (or closing the cmd where they are running) - I am doing the right thing! Thank you for the tip @Shubham ! (since I am in tests, I start and stop it offen!) – Teo Jan 20 '20 at 15:02

score 9 · Answer 2 · answered Mar 17 '21 at 06:01

9

@ShubhWIP's anwwer is correct, but is missing a vital information.

Add proper waits between zookeeper and kafka broker services start and stop.

Introduce a 20 second delay between zookeeper and kafka. The zookeeper has a session expiry time of 18000ms. It needs this time to declare the old session dead. If the Kafka broker is brought up before this happens, the broker shuts down with "Error while creating ephemeral at /broker/ids/id, node already exists".

answered Mar 17 '21 at 06:01

patel deven

630
7
9

2

on docker-compose can do something like `entrypoint: sh -c 'sleep 30 && /etc/confluent/docker/run'` – Pants Jun 23 '21 at 20:21
This worked for me. – EJoshuaS - Stand with Ukraine Jun 06 '23 at 19:18

score 8 · Answer 3 · edited Jan 07 '20 at 13:11

8

I found similar issue in my AWS server. The problem was zookeeper was running. So I killed the process first and tried again and it worked.

$ ps aux | grep zookeeper
$kill -9 <zookeeper.properties process>

edited Jan 07 '20 at 13:11

geisterfurz007

5,292
5
33
54

answered Jun 08 '17 at 03:14

Avinash

2,093
4
28
41

For me this happened on my local machine. I had a set of docker commands running on start up. One of which, I did not realise, was zookeeper. Once I killed the docker process I was able to started my own zookeeper and then the kafka server worked. – Shane Gannon Jan 11 '18 at 16:19

score 6 · Answer 4 · edited Jan 07 '20 at 14:24

6

Under /tmp directory following directories have been removed:

kafka-log directory
Zookeeper-log directory

Then, I've restarted kafka service.

It works !!

edited Jan 07 '20 at 14:24

Dominique

16,450
15
56
112

answered Jan 07 '20 at 13:01

Praveen Muniraj

103
1
4

2

this could have been a comment – Milan Desai Jan 07 '20 at 13:39
2

Bad answer. You destroy the precious data. Do you format your harddisk because you couldn't open a file? – Semo Jun 08 '21 at 07:27
I took a backup of log files, deleted the `*-log` folder from `/tmp`. It worked. – NaveenKumar Namachivayam Apr 27 '22 at 14:57

score 5 · Answer 5 · answered Apr 08 '17 at 09:22

5

I came across the same issue and there are 2 ways to go about it

[Recommended] You need to clean up the broker ids in the zookeeper path /brokers/ids/[]. Use the zk-cli tool delete command to clean up the paths. Start your brokers and verify it registers with the coordinator.
The other way of resolving this is to change your broker-id from kafka server config and restarting the broker. However, this would corrupt your partitions and data is not recommended

answered Apr 08 '17 at 09:22

meliora245

53
1
3

Worked for me, and possibly a better solution than the somewhat brutal deletion of zk and kafka temp directories. Many thanks, I would just add, for those not familiar, you can also use the `./bin/zookeeper-shell.sh` in standalone kafka to do the job (eg `./kafka/kafka_2.13-2.8.0/bin/zookeeper-shell.sh 0.0.0.0:2181`). Check the brokerid with `ls /brokers/ids` and then delete it with e.g. `deleteall /brokers/ids/0`. – JL_SO Aug 04 '21 at 10:16

score 3 · Answer 6 · edited Oct 19 '20 at 02:08

3

The below approach worked for me.

Change the broker ID in config/server-1.properties to a different value. By default it is 0. Change it to 1 and test things again.

edited Oct 19 '20 at 02:08

YanetP1988

1,346
3
18
43

answered May 19 '18 at 02:23

Ram Repaka

137
1
3

score 3 · Answer 7 · answered Nov 18 '20 at 08:27

3

I had the when running multiple containers with docker-compose up and ending the processes with ctrl-C. Fixed it with docker-compose down.

answered Nov 18 '20 at 08:27

schoon

2,858
3
46
78

this deletes the attached volume as well... not helpful! I wanted to restart the docker after adding jdbc-driver to the broker. – Mitanshu Nov 09 '22 at 08:48

score 2 · Answer 8 · answered Nov 21 '18 at 13:39

I had similar issue in with docker container containing zookeeper and kafka (it isn't best practise to put two services in one container, but it was for testing purpose).

this was the error

[2018-11-21 12:54:24,434] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2018-11-21 12:54:24,434] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2018-11-21 12:54:24,435] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2018-11-21 12:54:24,453] INFO Loading logs. (kafka.log.LogManager)
[2018-11-21 12:54:24,457] INFO Logs loading complete in 4 ms. (kafka.log.LogManager)
[2018-11-21 12:54:24,468] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2018-11-21 12:54:24,469] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2018-11-21 12:54:24,694] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2018-11-21 12:54:24,728] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2018-11-21 12:54:24,741] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-11-21 12:54:24,741] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-11-21 12:54:24,741] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-11-21 12:54:24,750] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2018-11-21 12:54:24,790] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2018-11-21 12:54:24,802] ERROR Error while creating ephemeral at /brokers/ids/0, node already exists and owner '72057672936325120' does not match current session '72057674644258816' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2018-11-21 12:54:24,803] INFO Result of znode creation at /brokers/ids/0 is: NODEEXISTS (kafka.zk.KafkaZkClient)
[2018-11-21 12:54:24,808] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
    at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1525)
    at kafka.zk.KafkaZkClient.registerBrokerInZk(KafkaZkClient.scala:84)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:257)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:75)
    at kafka.Kafka.main(Kafka.scala)

Quickfix for me was to start kafka-server a few seconds later;

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties && (bin/kafka-server-start.sh config/server.properties || echo "kafka start failed. restarting ... " && sleep 3 && bin/kafka-server-start.sh config/server.properties)

score 1 · Answer 9 · answered Sep 29 '16 at 00:52

Make sure the kafka server shut down correctly by checking the default prot 9092 is already in use or not. Kill the process on 9092, and restart kafka server.

If not works, might be the existing kafka server (with broker id 0) is already in use. Give it a try to change broker.id to 1 instead of 0 in "server0.properties" (btw, why it was called server0, did you change any config here? by default, it should be server.properties),

score 1 · Answer 10 · answered Jan 20 '20 at 15:31

I was in the same boat in windows environment! And I am just in tests, so I have to close the processes every day since they are installed on my laptop which is put down daily! @Shubham gave me the idea but he did not elaborate so, I feel it's right to put it as an answer so that others who come in the same boat know what to do: It is important that the processes are not killed - in my case I closed the windows cmd in which the two processes were running. This is WRONG and the logs will not allow after that to start the kafka broker properly. I know now to stop the processes by running the proper bat (in \bin\windows folder where the other commands - for starting the servers- are). I will not kill the processes any more by closing the cmd. To go out of the pit I did erase the the kafka logs - but that is not the solution! The solution is to avoid killing the processes!

score 0 · Answer 11 · answered Nov 08 '20 at 09:53

I encountered this in a production environment and could not simply change the Kafka broker ID. After executing a rolling restart of all Zookeeper instances, I was able to successfully restart the Kafka broker. If you are running against a single Zookeeper instance, then restarting the single Zookeeper instance should suffice.

score 0 · Answer 12 · answered Aug 18 '21 at 19:04

I also encountered the same problem. You can follow my steps to do it:

A cluster with three nodes as an example:

For node h01: set broker.id=0 in file kafka/config/server.properties, then you can see a line of code in this file like this: log.dirs=/tmp/kafka-logs, that's mine, yours may be others. Then enter this directory(log.dirs) and you will find a file meta.properties: open and set broker.id=0.

Next do similar work:

For node h02: set broker.id=1 in file kafka/config/server.properties, then you can see a line of code in this file like this: log.dirs=/tmp/kafka-logs, that's mine, yours may be others. Then enter this directory (log.dirs) and you will find a file meta.properties: open and set broker.id=1.

For node h03: set broker.id=2 in file kafka/config/server.properties, then you can see a line of code in this file like this: log.dirs=/tmp/kafka-logs, that's mine, yours may be others. Then enter this directory (log.dirs) and you will find a file meta.properties: open and set broker.id=2.

Last restart kafka.

score -7 · Answer 13 · edited Jun 21 '18 at 05:09

-7

You just have to execute the zookeeper-server-stop.[sh|bat] script.

edited Jun 21 '18 at 05:09

Pang

9,564
146
81
122

answered Jan 24 '17 at 19:13

Gerardo Pucheta Figueroa

9

error while starting kafka broker

13 Answers13

Linked