Zookeeper: java.io.IOException: No snapshot found, but there are log entries. Something is broken

Question

I have been working with Kafka 2.4.0 (2.11) and yesterday I had to forcefully terminate the process for some unknown reason. Since then I haven't been unable to start Zookeeper due to the following error:

[2020-01-11 11:12:43,783] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:222)
    at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
    at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
    at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
    at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

And as soon as I searched for this problem I found issue ZOOKEEPER-3513 reported, which may or may not explain the problem. However, what I'm finding strange is that if I delete the Kafka/Zookeeper directory and download it again from scratch, the problem persists. Does anyone know how I can solve this?

Thank you for your help

You shouldn't need to re-download anything. There's no state stored outside of those data directories — OneCricketeer, Jan 11 '20 at 13:08
Isn't that the dataDir? Besides, you should not store data on /tmp anyway — OneCricketeer, Jan 11 '20 at 14:31
have you tried the workaround with `snapshot.0` file described in https://issues.apache.org/jira/browse/ZOOKEEPER-3056? — mazaneicha, Jan 11 '20 at 14:34

score 80 · Answer 1 · answered Jan 21 '20 at 11:28

80

Check for the tmp/zookeeper folder on the drive where you have kafka folder (lets say D:/), and delete the folder tmp, which will create automatically for you once run the zookeeper again.

answered Jan 21 '20 at 11:28

Veswanth

1,061
9
22

17

this works for me, in Linux I used: rm -rf /tmp/zookeeper/* – Hernaldo Gonzalez Dec 22 '20 at 13:10
I also had to delete broker files for this to work. deleted broker files from /tmp folder. – Deepak Poojari Nov 03 '21 at 05:51

score 12 · Answer 2 · answered Jul 07 '21 at 09:37

12

On windows ->

Go to the tmp folder where the zookeeper details are stored and delete the existing log files

Directory path = d:\tmp\zookeeper\version-2

On Linux ->

Path = /tmp/zookeeper/version-2

And remove all the existing log files using rm -r log.1

The log files will be created automatically again and will resolve the issue.

answered Jul 07 '21 at 09:37

CodingBee

1,011
11
8

Linux location was the same for mac OS – Rodolfo Velasco Jul 11 '23 at 17:14

score 10 · Answer 3 · answered Jan 11 '20 at 12:22

Try changing your zookeeper data directory. Your zookeeper data directory is defined in zookeeper.properties (I think the default is /tmp/zookeeper).

Perhaps you're not deleting the correct zookeeper directory?

I had the same problem, and this solution worked.

NOTE: I'm experimenting with Kafka, and not using it in production. I have no idea what else the above does, apart from fix this error...

score 6 · Answer 4 · edited Mar 09 '20 at 09:11

6

I've faced the same issue with Zookeeper after updating from version 3.4.x to 3.5.6. As described here. I've:

added empty snapshot.0 file in data directory
added a property 'zookeeper.snapshot.trust.empty=true' to Zookeeper configuration file (default is zoo.cfg)

edited Mar 09 '20 at 09:11

m1k3y3

2,762
8
39
68

answered Mar 09 '20 at 07:01

Sergei Doroshenko

86
1

1

For me, it didn't work with empty `snapshot.0`, but with the one attached at the link that you posted. +1 though, that's the workaround. – Miljen Mikic Oct 28 '20 at 10:32
Download `sanpshot.0` from the given link and paste it to dataDir. Now, the question is where exactly the data directory exists? I am using a mac and for me, the dataDir is located at `/usr/local/var/run/zookeeper/data/version-2`. You can check your dataDir inside zookeeper config file (zoo.cfg). Just add `version-x` to this dataDir. Please read [this](https://issues.apache.org/jira/browse/ZOOKEEPER-3056?focusedCommentId=16519839&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16519839) comment carefully. – mdaftab88 Nov 04 '20 at 15:39

score 2 · Answer 5 · answered Sep 19 '22 at 18:13

2

Faced same issue in macOS. Solution: In kafka dir, path cd /tmp/zookeeper/version-2 deleted the log.1 file. It worked for me

answered Sep 19 '22 at 18:13

Shriya Sundriyal

41
3

score 1 · Answer 6 · answered Jun 14 '23 at 06:27

I also faced this error. The above solution of deleting the zookeeper folder from tmp worked for me. For those who don't know how to find tmp folder on mac : Open a Finder window. Click on "Go" in the menu bar at the top of the screen. Select "Go to Folder" from the dropdown menu. In the dialog
box that appears, enter /tmp. Click the "Go" button.

This will open the /tmp directory where you can find the temporary files and folders. Then delete the zookeeper folder.

score 0 · Answer 7 · answered Jan 21 '20 at 04:57

0

if you are on windows make sure you escape the location of the zookeeper temp directory.

dataDir=d:\tmp\zookeeper

answered Jan 21 '20 at 04:57

DReamey

9
2

If you are not sure about any information to answer a question, you should ask them through comments. – prem Jan 21 '20 at 05:41

score 0 · Answer 8 · answered Jul 02 '20 at 22:26

0

Created a new dir for logs and configured the same path in zoo.cfg. It worked:)

answered Jul 02 '20 at 22:26

skumar

16
4

score 0 · Answer 9 · edited Oct 06 '20 at 07:34

0

I use macOS and my solution was to delete everything in the dataDir, the default value should be /usr/local/var/lib/zookeeper.

edited Oct 06 '20 at 07:34

Ivan Aracki

4,861
11
59
73

answered Oct 06 '20 at 02:43

Jinhuan Lei

1
1

score 0 · Answer 10 · answered Feb 10 '21 at 15:47

For those who are using docker, I'll share my experience:

I've been running zookeeper confluentinc/cp-zookeeper:5.2.1 as it follows:

 docker run \
    --network kafka-net --name=zookeeper \
    -e ALLOW_ANONYMOUS_LOGIN=yes \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    -v /tmp/zookeeper-data:/var/lib/zookeeper/data \
    -v /tmp/zookeeper-txn-logs:/var/lib/zookeeper/log \
    -p 2181:2182 confluentinc/cp-zookeeper:5.2.1

As expected, I can see a few files placed in /tmp/zookeeper-txn-logs and /tmp/zookeeper-data on host. After cleaning up /tmp/zookeeper-data and running again, I've got the error No snapshot found, but there are log entries.

In my case, I just had to purge the data on /tmp/zookeeper-txn-logs. For a dev/production environment, I'd recommend following the docs https://access.redhat.com/documentation/en-us/red_hat_amq/6.3/html/fabric_guide/ensemble-purgetxnlog

score 0 · Answer 11 · answered May 08 '23 at 18:58

0

on mac it helped me to restart the computer but first delete the tmp/zookeper files

answered May 08 '23 at 18:58

AlejandroGM020

1
1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 09 '23 at 19:03

Zookeeper: java.io.IOException: No snapshot found, but there are log entries. Something is broken

11 Answers11