How to Increment consistently in Neo4j, Cypher is too slow?

Question

Im currently using Neo4j 2.0+ and cypher to create and save sessions.

My project at times requires multiple writes per second to a node labeled 'ChildSession', and I notice that when I 'increment' the ChildSession_ID in cypher, I often have ChildSession_ID's skipping numbers or being the same number.

Not sure if neo4j/cypher is too slow for my requirements, but I doubt it since the internal Neo4j Node ID's are incremented normally.

The cypher command i'm using, to increment ChildSession is:

 match (p:ChildSession) with count(p) as Total 
 Create (b:ChildSession{ChildSession_ID:Total + 1 })

One would expect the ChildSession_ID to increment, but I get the following results, when I check the nodes in neo4j browser:

match (u:ChildSession) return u,ID(u)

Results:
ChildSession_ID 1
44997
ChildSession_ID 1
44998
ChildSession_ID 1
44999
ChildSession_ID 4
45000
ChildSession_ID 5
45001
ChildSession_ID 6
45002
ChildSession_ID 6
45003
ChildSession_ID 8
45004
ChildSession_ID 8
45005

I've been unable to get neo4j to increment accurately. I tried using redis and its hincrby command which increments and then put this variable into my cypher query's ChildSession_ID attribute. This method works, but I would rather do it with cypher instead.

The argument could be made to use redis only instead for fast writes, but I use a hiearchy of session levels and need the querying neo4j offers.

Thanks.

score 7 · Accepted Answer · answered Sep 08 '14 at 08:22

Have you added any manual locking around access to the ChildSession label? I would expect otherwise that the MATCH statement could happen on multiple threads at the same time (no read locking). The result of this would be the same value for Total being passed to your CREATE statement.

Two things spring to mind, using the COUNT is maybe not the fastest and would not guarantee unique values if you were ever to delete a ChildSession (maybe you do not care?). Maintaining the count as a property is probably faster (query time) and incurs minimal overhead.

I believe that the correct way to achieve this is to use a MERGE statement on a separate label which should lock on the Incremental Node through the whole transaction (as described in this Blog Post):

MERGE (nid:Incremental{type:ChildSession})
ON CREATE SET nid.count = 1
ON MATCH SET nid.count = nid.count + 1
RETURN nid.count

Caveat: Whilst I believe this is the correct way to do it, I'll be damned if it is working nicely in a rapid async environment, suggesting that the semantics of the locking are not what I expect.

You could of course generate a UUID if the incremental nature of your ChildSession_ID is not important?

Although the incremental works at normal speed it doesn't increment as expected when adding multiple nodes per second as i'm doing. May be neo4j issue, but I am taking the UUID approach. I was able to accomplish the increments by using redis's hincrby command. Thanks +1 — 1ManStartup, Sep 11 '14 at 00:21

How to Increment consistently in Neo4j, Cypher is too slow?

1 Answers1