Jetbrains xodus no more bytes available error

Question

I'm using Exodus from Jetbrains as an embedded database in my application. Recently I've faced with some weird behavior, I'm afraid that the one of the stores is broken. At least I can't access it anymore.

jetbrains.exodus.ExodusException: DataIterator: no more bytes available, address = 19359, file = 00000000000.xd
        at jetbrains.exodus.log.DataIterator.next(DataIterator.java:60)
        at jetbrains.exodus.log.Log.readNotNull(Log.java:440)
        at jetbrains.exodus.tree.btree.BTreeBase.getLoggable(BTreeBase.java:119)
        at jetbrains.exodus.tree.btree.BTreeBase.compareLeafToKey(BTreeBase.java:170)
        at jetbrains.exodus.tree.btree.BasePageImmutable.binarySearch(BasePageImmutable.java:169)
        at jetbrains.exodus.tree.btree.BasePageImmutable.binarySearch(BasePageImmutable.java:121)
        at jetbrains.exodus.tree.btree.BottomPage.get(BottomPage.java:139)
        at jetbrains.exodus.tree.btree.BottomPage.get(BottomPage.java:57)
        at jetbrains.exodus.tree.btree.InternalPage.get(InternalPage.java:98)
        at jetbrains.exodus.tree.btree.InternalPage.get(InternalPage.java:72)
        at jetbrains.exodus.tree.btree.InternalPage.get(InternalPage.java:98)
        at jetbrains.exodus.tree.btree.InternalPage.get(InternalPage.java:72)
        at jetbrains.exodus.tree.btree.BTreeBase.get(BTreeBase.java:181)
        at jetbrains.exodus.env.StoreImpl.get(StoreImpl.java:79)

And what is more important, I'm afraid the other stores will also break in same way and won't be accessible. I'm using single Environment and about 10 different stores. The gzipped backup of whole database is around 2 gb. Do you know what is the root cause of the issue and how to recove store?

There is another exception on environment startup:

2019-11-19 19:15:49,897 13408 [Exodus shared background cleaner] ERROR j.exodus.gc.GarbageCollector - cleanFile(000000r1g00.xd)
jetbrains.exodus.ExodusException: DataIterator: no more bytes available, address = 6019, file = 00000000000.xd
        at jetbrains.exodus.log.DataIterator.next(DataIterator.java:60)
        at jetbrains.exodus.log.Log.readNotNull(Log.java:440)
        at jetbrains.exodus.tree.btree.BTreeBase.getLoggable(BTreeBase.java:119)
        at jetbrains.exodus.tree.btree.BTreeBase.compareLeafToKey(BTreeBase.java:170)
        at jetbrains.exodus.tree.btree.BasePageImmutable.binarySearch(BasePageImmutable.java:169)
        at jetbrains.exodus.tree.btree.BTreeTraverser.getNextSibling(BTreeTraverser.java:141)
        at jetbrains.exodus.tree.btree.LeafNode.reclaim(LeafNode.java:154)
        at jetbrains.exodus.tree.btree.BTreeMutable.reclaim(BTreeMutable.java:295)
        at jetbrains.exodus.env.StoreImpl.reclaim(StoreImpl.java:160)
        at jetbrains.exodus.gc.GarbageCollector.cleanSingleFile(GarbageCollector.java:377)
        at jetbrains.exodus.gc.GarbageCollector.doCleanFiles(GarbageCollector.java:288)
        at jetbrains.exodus.gc.GarbageCollector.cleanFiles(GarbageCollector.java:215)
        at jetbrains.exodus.gc.BackgroundCleaningJob.cleanFiles(BackgroundCleaningJob.java:139)
        at jetbrains.exodus.gc.BackgroundCleaningJob.doCleanLog(BackgroundCleaningJob.java:110)
        at jetbrains.exodus.gc.BackgroundCleaningJob.execute(BackgroundCleaningJob.java:78)
        at jetbrains.exodus.core.execution.Job.run(Job.java:99)
        at jetbrains.exodus.core.execution.ThreadJobProcessor.executeJob(ThreadJobProcessor.java:132)
        at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doExecuteJob(JobProcessorQueueAdapter.java:238)
        at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doTimedJobs(JobProcessorQueueAdapter.java:190)
        at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doJobs(JobProcessorQueueAdapter.java:161)
        at jetbrains.exodus.core.execution.ThreadJobProcessor.run(ThreadJobProcessor.java:110)
        at jetbrains.exodus.core.execution.ThreadJobProcessor$1.run(ThreadJobProcessor.java:43)
        at java.lang.Thread.run(Thread.java:745)
2019-11-19 19:15:49,898 13409 [Exodus shared background cleaner] ERROR j.exodus.gc.BackgroundCleaner - DataIterator: no more bytes available, address = 6019, file = 00000000000.xd
jetbrains.exodus.ExodusException: DataIterator: no more bytes available, address = 6019, file = 00000000000.xd
        at jetbrains.exodus.log.DataIterator.next(DataIterator.java:60)
        at jetbrains.exodus.log.Log.readNotNull(Log.java:440)
        at jetbrains.exodus.tree.btree.BTreeBase.getLoggable(BTreeBase.java:119)
        at jetbrains.exodus.tree.btree.BTreeBase.compareLeafToKey(BTreeBase.java:170)
        at jetbrains.exodus.tree.btree.BasePageImmutable.binarySearch(BasePageImmutable.java:169)
        at jetbrains.exodus.tree.btree.BTreeTraverser.getNextSibling(BTreeTraverser.java:141)
        at jetbrains.exodus.tree.btree.LeafNode.reclaim(LeafNode.java:154)
        at jetbrains.exodus.tree.btree.BTreeMutable.reclaim(BTreeMutable.java:295)
        at jetbrains.exodus.env.StoreImpl.reclaim(StoreImpl.java:160)
        at jetbrains.exodus.gc.GarbageCollector.cleanSingleFile(GarbageCollector.java:377)
        at jetbrains.exodus.gc.GarbageCollector.doCleanFiles(GarbageCollector.java:288)
        at jetbrains.exodus.gc.GarbageCollector.cleanFiles(GarbageCollector.java:215)
        at jetbrains.exodus.gc.BackgroundCleaningJob.cleanFiles(BackgroundCleaningJob.java:139)
        at jetbrains.exodus.gc.BackgroundCleaningJob.doCleanLog(BackgroundCleaningJob.java:110)
        at jetbrains.exodus.gc.BackgroundCleaningJob.execute(BackgroundCleaningJob.java:78)
        at jetbrains.exodus.core.execution.Job.run(Job.java:99)
        at jetbrains.exodus.core.execution.ThreadJobProcessor.executeJob(ThreadJobProcessor.java:132)
        at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doExecuteJob(JobProcessorQueueAdapter.java:238)
        at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doTimedJobs(JobProcessorQueueAdapter.java:190)
        at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doJobs(JobProcessorQueueAdapter.java:161)
        at jetbrains.exodus.core.execution.ThreadJobProcessor.run(ThreadJobProcessor.java:110)
        at jetbrains.exodus.core.execution.ThreadJobProcessor$1.run(ThreadJobProcessor.java:43)
        at java.lang.Thread.run(Thread.java:745)

What Xodus version do you use? Did you experience any hardware/system failures? — Vyacheslav Lukianov, Nov 19 '19 at 13:35
Xodus version is 1.0.6. I think it was working more or less ok for maybe a year without any issue. I don't see any other indication that there are any issues with hardware/software. Everything is working as usual, except 1 store — user12384512, Nov 19 '19 at 18:13

score 2 · Answer 1 · answered Nov 21 '19 at 11:27

2

Looks like one of the environments got corrupted for some reason.

In most cases it is possible to fix corrupted environment using the Reflect tool. Please download the latest released version of xodus-tools.jar from https://mvnrepository.com/artifact/org.jetbrains.xodus/xodus-tools. Let's say sourceDir is a path where corrupted environment is. Prepare empty targetDir, path where the tool will create copy of the environment. Run:

java -jar <path to jar>/xodus-tools.jar reflect -c sourceDir targetDir

What does the tool do? It creates in targetDir fixed and compressed copy of environment from sourceDir. It tries to copy as many key/value pairs as it can.

Try to run your application against fixed environment. If it's ok update Xodus version to the latest one.

answered Nov 21 '19 at 11:27

Vyacheslav Lukianov

1,913
8
12

What are the most common reasons that could make the Xodus database corrupted? – quarks Nov 27 '19 at 15:44
1

There are some known and fixed cases like https://youtrack.jetbrains.com/issue/XD-763. Also, there are common issues for any database, sort of hardware/system failures. E.g., Windows OS family doesn't fsync directory structure. There can appear an issue in Linux FS as well: https://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Data_Loss_with_Applications_that_do_not_use_fsync.28.29_correctly. We try fix/workaround all platform-specific issues in order to be truly platform-independent. – Vyacheslav Lukianov Nov 28 '19 at 14:18

Jetbrains xodus no more bytes available error

1 Answers1