Stack Overflow
Stack Overflow

Questions tagged [apache-zookeeper]

Apache ZooKeeper is a distributed service that provides configuration information storage, naming, synchronization, coordination, and group services.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. It was inspired by Chubby, Google's distributed lock system.

All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which makes them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

ZooKeeper is used by a variety of other software projects to maintain configuration information including Apache Storm, Apache Flink, Solr Cloud, Apache Kafka and Apache Hadoop.

Links

Related Tags

3686 questions
57
votes
13 answers

Zookeeper error: Cannot open channel to X at election address

I have installed zookeeper in 3 different aws servers. The following is the configuration in all the…
Rahul
  • 44,892
  • 25
  • 73
  • 103
48
votes
3 answers

Offsets stored in Zookeeper or Kafka?

I'm a bit confused about where offsets are stored when using Kafka and Zookeeper. It seems like offsets in some cases are stored in Zookeeper, in other cases they are stored in Kafka. What determines whether the offset is stored in Kafka or in…
Nikola Schou
  • 2,386
  • 3
  • 23
  • 47
47
votes
11 answers

Zookeeper: java.io.IOException: No snapshot found, but there are log entries. Something is broken

I have been working with Kafka 2.4.0 (2.11) and yesterday I had to forcefully terminate the process for some unknown reason. Since then I haven't been unable to start Zookeeper due to the following error: [2020-01-11 11:12:43,783] ERROR Unexpected…
João Matos
  • 6,102
  • 5
  • 41
  • 76
46
votes
2 answers

What is the role of Zookeeper vs Eureka for microservices?

I am going to implement the orchestration of a set of microservices in my application. Two widely using tools I found Apache Zookeeper and Netflix Eureka. Can anyone please give a comparison based on fundamental differences, those two services…
Supun Wijerathne
  • 11,964
  • 10
  • 61
  • 87
46
votes
2 answers

where should I put docker-compose.yml

When I pull the images from docker hub. Sometimes, I would like to run the images in a multi-container way. So I choose to use docker-compose. For example, I would run the zookeeper in replicated mode. I will new a file named docker-compose.yml, and…
Dr.Bit
  • 719
  • 1
  • 6
  • 14
46
votes
6 answers

How to create a Topic in Kafka through Java

I want to create a topic in Kafka (kafka_2.8.0-0.8.1.1) through java. It is working fine if I create a topic in command prompt, and If I push message through java api. But I want to create a topic through java api. After a long search I found below…
Jaya Ananthram
  • 3,433
  • 1
  • 22
  • 37
45
votes
4 answers

removing a kafka consumer group in zookeeper

I'm using kafka_2.9.2-0.8.1.1 with zookeeper 3.4.6. Is there a utility that can automatically remove a consumer group from zookeeper? Or can I just remove everything under /consumers/[group_id] in zookeeper? If the latter, is there anything else I'm…
Foo L
  • 10,977
  • 8
  • 40
  • 52
44
votes
6 answers

Is it possible to start a zookeeper server instance in process, say for unit tests?

Calling org.apache.zookeeper.server.quorum.QuorumPeerMain.main() isn't working.
marathon
  • 7,881
  • 17
  • 74
  • 137
43
votes
3 answers

Why do Kafka consumers connect to zookeeper, and producers get metadata from brokers?

Why is it that consumers connect to zookeeper to retrieve the partition locations? And kafka producers have to connect to one of the brokers to retrieve metadata. My point is, what exactly is the use of zookeeper when every broker already has all…
Luckl507
  • 477
  • 1
  • 4
  • 6
43
votes
5 answers

Whats the difference between Paxos and W+R>=N in Cassandra?

Dynamo-like databases (e.g. Cassandra) can enforce consistency by means of quorum, i.e. a number of synchronously written replicas (W) and a number of replicas to read (R) should be chosen in such a way that W+R>N where N is a replication factor. On…
user1128016
  • 1,438
  • 3
  • 16
  • 17
41
votes
3 answers

Bootstrap server vs zookeeper in kafka?

Why the use of zookeeper in kafka-consumer is deprecated and why it's recommended to use the bootstrap server instead ? what are the advantages of the bootstrap-server?
Slim AZAIZ
  • 646
  • 1
  • 8
  • 21
39
votes
12 answers

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Below the steps I did to get this issue : Launch ZooKeeper Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties And at the second step the error happens : ERROR Fatal error during KafkaServer startup. Prepare to…
TourEiffel
  • 4,034
  • 2
  • 16
  • 45
39
votes
7 answers

How to get data from old offset point in Kafka?

I am using zookeeper to get data from kafka. And here I always get data from last offset point. Is there any way to specify the time of offset to get old data? There is one option autooffset.reset. It accepts smallest or largest. Can someone please…
Sourabh
  • 680
  • 3
  • 8
  • 18
39
votes
24 answers

Zookeeper connection error

We have a standalone zookeeper setup on a dev machine. It works fine for every other dev machine except this one testdev machine. We get this error over and over again when trying to connect to zookeeper through testdev: 2012-11-09 14:06:53,909 -…
user1322614
  • 587
  • 1
  • 4
  • 11
39
votes
3 answers

Why do we need ZooKeeper in the Hadoop stack?

I am new to Hadoop/ZooKeeper. I cannot understand the purpose of using ZooKeeper with Hadoop, is ZooKeeper writing data in Hadoop? If not, then why we do we use ZooKeeper with Hadoop?
user1099871
1
2
3
…
99 100