64

I am trying to create topics in Kafka by following the guide on Apache Kafka website through command line. While running the command:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

I get the error as follows:

Error while executing topic command replication factor: 1 larger than available
brokers: 0
kafka.admin.AdminOperationException: replication factor: 1 larger than available
 brokers: 0
        at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
        at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171)
        at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:93)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:55)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

[2015-10-13 14:25:38,607] INFO Processed session termination for sessionid: 0x15
06056566d0004 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-10-13 14:25:38,608] INFO Closed socket connection for client /127.0.0.1:33
391 which had sessionid 0x1506056566d0004 (org.apache.zookeeper.server.NIOServer
Cnxn)

How can I resolve the issue? Any good documentation to get started with this? Thanks!!

Nicole
  • 859
  • 2
  • 8
  • 16

20 Answers20

36

Seems like your server not started , check logs for issue

Kill process

sudo fuser -k 2181/tcp

run zookeeper 

bin/zookeeper-server-start.sh config/zookeeper.properties

Run Kafka

bin/kafka-server-start.sh config/server.properties  '
vaquar khan
  • 10,864
  • 5
  • 72
  • 96
18

If I were you I would try this code:

bin/kafka-topics.sh --create --zookeeper localhost:2181/kafka
--replication-factor 1 --partitions 1 --topic test

If you are using a cluster where zookeeper be distributed in 3 nodes you should substitue localhost:2181/kafka for direccion1:2181,direction2:2181,direction3:2181/kafka

General Grievance
  • 4,555
  • 31
  • 31
  • 45
Anxo P
  • 759
  • 7
  • 12
10

For those who are sure that the brokers are up and running, check your server.properties configuration.

In my case the znodes for kafka is /kafka as shown below.

zookeeper.connect=zk1.beta:2181,zk2.beta:2181,zk3.beta:2181/kafka

So, try the following command while creating topic:

bin/kafka-topics.sh --create --zookeeper zk1.beta:2181/kafka --replication-factor 2 --partitions 2 --topic test2
Reyansh Kharga
  • 356
  • 5
  • 18
8

It clearly says 0 broker is available . Start a broker by 

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

base on the Apache Kafka tutorial then try

vasa.v03
  • 147
  • 2
  • 12
5

I have had the same problem.I solve the problem.I think the reason for this is because there is no background start-up. so,we neet start-up background

bin/kafka-server-start.sh config/server.properties &
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test &
chunbo
  • 51
  • 1
  • 1
4

I had the same problem but finally I tried the following steps which worked for me:

  1. Create the chroot in the Zookeeper with the following commands:
zkCli.sh -server localhost:2181
create /kafka1 []
  1. For the kafka brokers config , modify zookeeper.connect and add chroot path : For eg.

In server.properties :

zookeeper.connect=localhost:2181/**kafka1**

  1. Start Zookeeper

  2. Start Kafka Server

  3. Create a topic as below:

bin/kafka-topic.sh --create -zookeeper localhost:2181/**kafka1** --replication-factor 1 --partitions 1  --topic data
  1. Confirm the created topic:
bin/kafka-topics.sh --list --zookeeper localhost:2181/**kafka1**

I followed below for reference: https://community.hortonworks.com/questions/102132/run-multiple-broker-versions-in-the-same-cluster.html

General Grievance
  • 4,555
  • 31
  • 31
  • 45
Amit Kumar
  • 49
  • 2
2

Your Zookeeper is unable to see any Kafka brokers.Try logging in to your Zookeepr console : bin/zkCli.sh -server localhost:2181 Run [zk: localhost:2181(CONNECTED) 2] ls /brokers/ids on the console. It must be not be showing up any brokers. You need to restart all your Kafka brokers, and check for the kafka process log. Sometimes, even when the process shows as running (ps -ef | grep kafka), it may not have really started completely, because of which Zookeeper is unable to see it. I had the same issue with Apache Kafka distribution 0.11.0.

Binita Bharati
  • 5,239
  • 1
  • 43
  • 24
1

The same scenario I faced after deleting the only topic I had created.

The way to solve it was put Kafka and Zookeeper down, updated Kafka settings (server.properties file) where I add the following setting

delete.topic.enable=true

saved file, turn Zookeeper and Kafka up and everything run as usual.

LarsTech
  • 80,625
  • 14
  • 153
  • 225
nano
  • 11
  • 1
1

make sure the kafka server has started properly. If you are using -dameon parameter to start kafka server as daemon. Try to remove it and see if there are any errors during the startup.

The issue I ran into turned out to be a file access issue, where the user runs kafka doesn't have access to the log directory I configured. After I granted the access, it just works!

Mingliang
  • 69
  • 1
  • 6
1

The issue is because of logs in data/kafkalogs & data/zookeeperlogs folder.

  1. Keep these folder as is, just delete the logs inside them.
  2. Kill the previous processes, start fresh
  3. Start Zookeeper
  4. Start Kafka- wait to see if it restarting or throwing any error.
  5. Create a topic ( if both zookeeper & kafka cli throws no error, this should work)
Shrashti
  • 19
  • 2
0

Try adding in your serverwith:

delete.topic.enable=true
slfan
  • 8,950
  • 115
  • 65
  • 78
Enrique Zetina
  • 825
  • 5
  • 16
0

Probably there is leftover, runaway zookeeper on your server from one of the previous deployments. List the pods, containers, processes, etc. and check their age/start time to spot it.

imriss
  • 1,815
  • 4
  • 31
  • 46
0

In my case the error was caused by the fact that I started creating topics before the zookeeper became available. Since I did everything in a script, introducing some latency before creating topics helped. If your zookeeper client port is 2181, you can check the availability of your brokers by: 

echo dump | nc zookeeper 2181 | grep brokers
Tomasz Bartkowiak
  • 12,154
  • 4
  • 57
  • 62
0

  • Check if all the brokers are registered to the Zookeeper:

    zookeeper-shell.sh localhost:2181 ls /brokers/ids

  • Most probably some ID's are missing in the brackets []

  • Login to the machine where this ID resides and restart the broker:

    kafka-server-stop.sh
nohup kafka-server-start.sh ../config/server.properties > /dev/null 2>&1 &
  • Check again if the Zookeeper can now see all your brokers properly.
  • Retry your create topic command.

Hope it helps.

MoeHoss
  • 58
  • 4
0

Delete the logs and consumer offsets folder which is getting created in the below directory.

C:\Kafka\kafka_2.12-2.5.0\data\zookeeper

PS: You may loose all the previous topic details.

Kap
  • 1
0

HDFS/namenode happens to be in safe-mode

hdfs dfsadmin -safemode get hdfs dfsadmin -safemode leave

1 - Start zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

You can verify it on other terminal

telnet localhost 2181 //Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
stat

2 - Start kafka broker/server

bin/kafka-server-start.sh config/server.properties

Now create Topics

kafka-topics.sh --create --topic t1 --zookeeper localhost:2181 --replication-factor 1 --partitions 1

kafka-topics.sh --list --zookeeper localhost:2181

Reshan
  • 13
  • 2
0

Ensure your config/server.properties file has valid (and uncommented) values for the listeners and advertised.listeners properties.

Paul Robu
  • 77
  • 1
  • 3
0

Looks like your kafka server not running well Just make sure this

zookeeper.connect={{your zk ip/localhost}}:2181

at server.properties is correct.

Adial
  • 1
  • 1
0

This works for me with below commands \

first

bin\windows\kafka-topics.bat --create --bootstrap-server localhost:9092 --topic connect-configs --replication-factor 3 --partitions 1 --config cleanup.policy=compact

after

bin\windows\kafka-topics.bat --create --bootstrap-server localhost:9092 --topic connect-configs --replication-factor 1 --partitions 1
General Grievance
  • 4,555
  • 31
  • 31
  • 45
Mihiraj Madhusanka
  • 50
  • 1
-1

I have had the same problem. You have to follow this part of the guide: http://kafka.apache.org/documentation.html#quickstart_multibroker

I didn't have any created broker. Create 2 new brokers and you will fix the problem. Follow this steps: 

cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties

Once you create that files you have to configure it with this some changes:

In "server-1.properties":

broker.id=1
port=9093 
log.dir=/tmp/kafka-logs-1
host.name=localhost

(Port number must be the port number you have in your server.properties + 1. I had 9092 so I have to write 9093)

In "server-2.properties":

broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
host.name=localhost

(Port number must be the next to server-1. In my case 9094)

nyxtez37
  • 61
  • 5