0

I'm using Neo4j over windows for testing purposes and I'm working with a db containing ~2 million relations and about the same amount of nodes. after I had an ungraceful shutdown of neo4j while writing a batch of relations the db got corrupted.

it seems like there are some broken nodes/relations in the db and whenever I try to read them I get this error (I'm using py2neo):

Error: NodeImpl#1292315 not found. This can be because someone else deleted this entity while we were trying to read properties from it, or because of concurrent modification of other properties on this entity. The problem should be temporary.

I tried rebooting but neo4j fails to recover from this error. I found this question: Neo4j cannot read certain nodes. Throws NotFoundException. Corrupt database but the answer he got is no good for me because it involved in going over the db and redo the indexing, and I can't even read those broken nodes/relations so I can't fix their index (tried it and got the same error).

In general I've had many stability issues with neo4j (and on multiple platforms, not just windows). if no decent solution is found for this problem I will have to switch to a different database.

thanks in advance!

Community
  • 1
  • 1
Ronen Ness
  • 9,923
  • 4
  • 33
  • 50
  • I tried writing a script that query all the nodes (and obviously fail with the exception above), and from the exception string I got the id of the problematic node and delete it. I wanted to run this in a loop until there are no more errors, but when I do the delete part I'm getting this error: File "C:\Python27\lib\site-packages\py2neo\neo4j.py", line 1076, in _execute raise CustomCypherError(e) InvalidRecordException: PropertyRecord[2083536] not in use – Ronen Ness Sep 27 '14 at 22:56
  • I'm looking into using neo4j as a primary data store. Were you able to fix the corruption issue or have you since switched to a different database? Neo4j looks well suited to the app I'm building but I'm concerned about reliability. Thanks! – Venkat D. Nov 02 '15 at 22:47
  • hello Venkat D, I don't really know as I don't work on that project anymore. honestly, neo4j felt quite unstable and I wasn't too happy with it, but I don't know if the alternatives are any better. also, when working with it I was quite new with node-based dbs, so that might contributed to the chaos. – Ronen Ness Nov 06 '15 at 16:20

3 Answers3

1

I wrote a tool a while ago that allows you to copy a broken store and keeps the good records intact.

You might want to check it out. I assume you used the 2.1.x version of Neo4j.

https://github.com/jexp/store-utils/tree/21

For 2.0.x check out:

https://github.com/jexp/store-utils/tree/20

Michael Hunger
  • 41,339
  • 3
  • 57
  • 80
0

To verify if your datastore is consistent follow the steps mentioned in http://www.markhneedham.com/blog/2014/01/22/neo4j-backup-store-copy-and-consistency-check/.

Are you referring to batch inserter API when speaking of "while writing a batch of relations"?

If so, be aware that batch inserter API requires a clean shutdown, see the big fat red warning on http://docs.neo4j.org/chunked/stable/batchinsert.html.

Stefan Armbruster
  • 39,465
  • 6
  • 87
  • 97
  • I'm using 'py2neo.neo4j.WriteBatch()', which I found in this doc: http://book.py2neo.org/en/latest/batches/ is that it? no big fat red warning there.. it says "Such requests also have the advantage of being executed within a single transaction." so it sounds like its not what you sent me? – Ronen Ness Sep 28 '14 at 09:26
  • nope, that is not batch inserter API since batch inserter API has no notion of transaction. Make sure to use consistency checker. – Stefan Armbruster Sep 28 '14 at 10:20
  • what good will consistency checker do? I already know the db is corrupted and even the Ids of the broken nodes (looks like they exist but don't have a PropertyRecord which mess neo4j up). so I'd love to know how to fix it. – Ronen Ness Sep 28 '14 at 10:39
  • as the name suggests the consistency checker will check consistency. Fixing a corrupt store is a non trivial task requiring a lot of manual work based on deep knowledge of the internal data structures. The best way to learn those is reading through the unit tests at https://github.com/neo4j/neo4j – Stefan Armbruster Sep 28 '14 at 11:52
  • since I'm working in a startup and have very limited time & resources I'm looking for a simple solution and can't afford to dive into the depths of neo4j. thanks anyway. – Ronen Ness Sep 28 '14 at 14:48
0

Are the broken nodes schema indexed and are you attempting to read them via this indexed label/property? If so, it's possible you may have a broken index following the sudden shutdown.

Assuming this is the case, you could try deleting the schema subdirectory within the graph store directory while the server is not running and let the database rebuild the index on restart. While this isn't an official way to recover from a broken index, it can sometimes work. Obviously, I suggest you back up your store before trying this.

Nigel Small
  • 4,475
  • 1
  • 17
  • 15
  • thanks but there is no problem with indexing (I actually deleted the entire index they were in, just in case). even a simple query such as "START z=node(*) RETURN z" invoke the error when reaching the broken nodes. – Ronen Ness Sep 28 '14 at 21:21