16

I am having a bit of trouble with neo4j unique constraints, where a CREATE cypher statement is failing to execute due to the node already existing. Problem is, it doesn't (exist). Also, this exact process with this exact data worked yesterday.

My neo4j version is community 2.0.0 (release) on ubuntu 12.04.3. This is my current situation:

My constraints: 

tas@vtas:~$ neo4j-shell
neo4j-sh (?)$ schema
Indexes
  ON :ConsumerUser(tokens) ONLINE
  ON :Id(uuid)             ONLINE (for uniqueness constraint) #relevant
  ON :User(email)          ONLINE (for uniqueness constraint)
  ON :User(username)       ONLINE (for uniqueness constraint) 

Constraints
  ON (user:User) ASSERT user.email IS UNIQUE
  ON (user:User) ASSERT user.username IS UNIQUE
  ON (id:Id) ASSERT id.uuid IS UNIQUE                         #relevant

:Id.uuid should be unique.

I don't have any data:

neo4j-sh (?)$ dump
begin
create index on :`ConsumerUser`(`tokens`);
create index on :`Id`(`uuid`);
create index on :`User`(`email`);
create index on :`User`(`username`);
;

(also verified with cypher MATCH (n) return n;)

The problem query:

neo4j-sh (?)$ cypher 2.0 CREATE (i:Id{uuid:2});
CypherExecutionException: Node 82 already exists with label Id and property "uuid"=[2]

Things I have tried

  • tail -f /var/lib/neo4j/data/log/*.log /var/lib/neo4j/data/graph.db/*.log /var/log/neo4j/*.log for errors: nothing logged at all
  • Restarting neo4j (service neo4j-service restart)
  • The above tail while restarting (only remotely interesting line: [main] INFO org.neo4j.kernel.AutoConfigurator - WARNING! Physical memory(1017MB) is less than assigned JVM memory(4185MB). Continuing but with available JVM memory set to available physical memory)
  • deleting the indexes (/var/lib/neo4j/data/graph.db/index/ and /var/lib/neo4j/data/graph.db/index.db) and restarting
  • restoring the above, restarting
  • Search SO
  • Search neo4j's github issues

Nothing has helped so far.

Things I will not try

  • Solution here: neo4j constraint deleted node because: dropping the constraint in production isn't an option. Also, that was a few versions ago and different use case:
    1. constraint on String[] instead of Int
    2. something was logged
  • upgrade to 2.0.1 in a fingers-crossed-this-may-fix-it fashion without knowing that this has been addressed explicitly (i need to know why this is happening)

Additional Information

  • I have ulimit -n and ulimit -Hn set to 40K

  • neo4j-sh (?)$ dbinfo -g Kernel

    {
  "KernelStartTime": "Fri Feb 21 13:53:57 GMT 2014",
  "KernelVersion": "Neo4j - Graph Database Kernel (neo4j-kernel), version: 2.0.0",
  "MBeanQuery": "org.neo4j:instance=kernel#0,name=*",
  "ReadOnly": false,
  "StoreCreationDate": "Fri Feb 14 18:43:27 GMT 2014",
  "StoreDirectory": "/var/lib/neo4j/data/graph.db",
  "StoreId": "a3351846c194229c",
  "StoreLogVersion": 21
}

  • I've seen this: https://github.com/neo4j/neo4j/issues/1069 but it seems resolved.

  • This is on a VirtualBox VM on a MacOSX 10.6 host

I'm at a loss, time for my first SO question.

The easy answer is "just wipe everything and start again" (or just re-do the constraint), but that isn't really acceptable (what if this happens in production?).

Any ideas?

Community
  • 1
  • 1
Tasos Bitsios
  • 2,699
  • 1
  • 16
  • 22
  • can you try with Cypher MERGE command? basically it will avoid you that exception because MERGE will either create or just return an existing node pattern – Sumeet Sharma Feb 21 '14 at 18:24
  • 1
    `neo4j-sh (?)$ MERGE (i:Id{uuid:2});` `EntityNotFoundException: Node with id 82` – Tasos Bitsios Feb 21 '14 at 19:39
  • strange! seems your db has become corrupt. WHat does these command return can you tell start n = node(82) return n – Sumeet Sharma Feb 21 '14 at 19:43
  • 1
    Same. `neo4j-sh (?)$ start n = node(82) return n; EntityNotFoundException: Node with id 82` – Tasos Bitsios Feb 21 '14 at 19:51
  • Any idea how that could have happened? Should it have healed itself during startup (/restart), or is there a separate tool for this, that I failed to find with Google? (I don't mind about this particular dev instance, just worried about this happening in production) – Tasos Bitsios Feb 21 '14 at 19:57
  • it happened sometime during my initial neo4j days. When neo4j server would have stopped suddenly or due to some reason transaction is not properly completed or the db isnt shutdown properly. I am not sure of the root cause though – Sumeet Sharma Feb 21 '14 at 20:01
  • I see. How did you resolve it? – Tasos Bitsios Feb 21 '14 at 20:03
  • In my case as the data wasn't useful I deleted and created a new db. But that wont serve your purpose as you mentioned in the question . – Sumeet Sharma Feb 21 '14 at 20:49
  • What you could do in production is: dump the graph to a file, create a new graph, import the dump file back. – pablosaraiva May 02 '15 at 19:56
  • Hmmm...the DB is empty, and yet yes is behaving as if it's corrupt. Presumably this DB had some content in it before, and then it was deleted, or was it just cold-started from nothing and then this problem appeared? I'm guessing there was some delete that was done on content there earlier. What was the method of delete? – FrobberOfBits May 03 '15 at 14:01
  • If the DB is completely empty, junking the DB and starting over seems appropriate but yes I'd want to find the root cause too. What was happening with this DB (details on software used, transactions, etc) just prior to this problem emerging? – FrobberOfBits May 03 '15 at 14:02

1 Answers1

4

Your DB is corrupt. Internally, Neo4j has a reference to this node, but you deleted the node, so this reference points to nothing. You can't delete it, because it doesn't exist, and you can't create it, because it THINKS it exists. (This was most likely caused by in improper/unexpected shutdown of the database. Remember to make sure this machine has a battery backup in production)

This is why you ALWAYS BACKUP DATA IN PRODUCTION! If a shard becomes corrupt, than you can just purge it and reload the data. Minimal downtime, and no need to understand how it's corrupt, just that it is. If you don't have backups (and you should have offsite backups), than you will need to export your data to CSV, purge the db, and load the CSV data back in. By purge, I mean completely erase the old db directory, and let Neo4j create a new one.

(Do not try an salvage the database without doing a clean purge, as once a DB becomes corrupt, you have no way of knowing what or how it has been compromised.)

Tezra
  • 8,463
  • 3
  • 31
  • 68