Cypher Query in Neo4j Returns 'undefined'

Question

I've built a graph using Neo4j which consists of about 1000 nodes and about 270,000 relationships. The graph is a flight network (airports are nodes, and flights are relationships) so there are many relationships from node to node. I'm sure there are also many cycles in certain queries (I'm not sure if that's a problem). Anyway, certain queries work fine, like these:

start a=node(1), b=node(2) match p=a-[:flys_to*..1]->b return p;
start a=node(1), b=node(2) match p=shortestPath(a-[:flys_to]->b) return p;

But when I try other more complicated queries, there is no friendly error message; it just hangs for about 3 minutes (depending on which two nodes I search between) and returns "Undefined".

One of the queries that hangs looks like this:

start a=node(1), b=node(2) match p=a-[:flys_to*..2]->b return p;

Also, an interesting side note is that when I type the following query, it instantly returns "Undefined":

start a=node(1), b=node(2) match p=a-[:flys_to]->b return p;

Does this make sense, or does anyone have any ideas as to where I might start troubleshooting?

More Info about this Specific Problem

Here's a compressed file of the database: http://www.opensourcebanking.org/neo4j/

I'm running this in a virtual environment over VMware. The host is OSX 10.8.2 and the guest is Windows Server 2012. The JRE is 1.6.0_38. Also, I have auto indexing setup on the node property 'iata', and the relationship property "flys_to".

Here are a few Cypher queries and their related logged messages:

The following query returns 'undefined' almost immediately:

start a=node(353), b=node(295)
match p=a-->b
return p;

And here's what I get in the log:

SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
org.neo4j.helpers.ThisShouldNotHappenError: Developer: Andres claims that: Unexpected traversal state encountered
    at org.neo4j.cypher.internal.pipes.matching.BidirectionalTraversalMatcher$StepCollisionDetector.includePath(BidirectionalTraversalMatcher.scala:101)
    at org.neo4j.kernel.StandardBranchCollisionDetector.evaluate(StandardBranchCollisionDetector.java:74)
    at org.neo4j.kernel.StandardBranchCollisionDetector.evaluate(StandardBranchCollisionDetector.java:37)
    at org.neo4j.kernel.impl.traversal.BidirectionalTraverserIterator.fetchNextOrNull(BidirectionalTraverserIterator.java:157)
    at org.neo4j.kernel.impl.traversal.BidirectionalTraverserIterator.fetchNextOrNull(BidirectionalTraverserIterator.java:41)
    at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
    at scala.collection.JavaConversions$JIteratorWrapper.hasNext(JavaConversions.scala:574)
    at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:371)
    at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:371)
    at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:334)
    at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:334)
    at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:334)
    at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply$mcZ$sp(ClosingIterator.scala:36)
    at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
    at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
    at org.neo4j.cypher.internal.ClosingIterator.failIfThrows(ClosingIterator.scala:87)
    at org.neo4j.cypher.internal.ClosingIterator.hasNext(ClosingIterator.scala:35)
    at scala.collection.Iterator$class.foreach(Iterator.scala:660)
    at org.neo4j.cypher.internal.ClosingIterator.foreach(ClosingIterator.scala:31)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
    at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:128)
    at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:242)
    at org.neo4j.cypher.internal.ClosingIterator.toList(ClosingIterator.scala:31)
    at org.neo4j.cypher.PipeExecutionResult.createTimedResults(PipeExecutionResult.scala:73)
    at org.neo4j.cypher.PipeExecutionResult.dumpToString(PipeExecutionResult.scala:80)
    at org.neo4j.cypher.PipeExecutionResult.dumpToString(PipeExecutionResult.scala:125)
    at org.neo4j.cypher.javacompat.ExecutionResult.dumpToString(ExecutionResult.java:91)
    at org.neo4j.shell.kernel.apps.Start.exec(Start.java:71)
    at org.neo4j.shell.kernel.apps.ReadOnlyGraphDatabaseApp.execute(ReadOnlyGraphDatabaseApp.java:32)
    at org.neo4j.shell.impl.AbstractAppServer.interpretLine(AbstractAppServer.java:115)
    at org.neo4j.shell.kernel.GraphDatabaseShellServer.interpretLine(GraphDatabaseShellServer.java:92)
    at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:130)
    at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:114)
    at org.neo4j.server.webadmin.console.ShellSession.evaluate(ShellSession.java:101)
    at org.neo4j.server.webadmin.rest.console.ConsoleService.exec(ConsoleService.java:125)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

The following query works fine. It returns 426 rows and nothing is logged.

start a=node(353), b=node(295)
match p=a-[:flys_to*..1]->b
return p;

The following query returns 'undefined' after about 10-15 minutes:

start a=node(353), b=node(295)
match p=a-[:flys_to*..2]->b
return p;

Log:

SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: Java heap space
    at org.neo4j.cypher.internal.pipes.matching.AddedHistory.toSeq(History.scala:87)
    at org.neo4j.cypher.internal.pipes.matching.AddedHistory.toMap(History.scala:75)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.isMatchSoFar(PatternMatcher.scala:166)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.org$neo4j$cypher$internal$pipes$matching$PatternMatcher$$traverseNextNodeFromRelationship(PatternMatcher.scala:98)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher$$anonfun$4.apply(PatternMatcher.scala:150)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher$$anonfun$4.apply(PatternMatcher.scala:150)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:175)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:175)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:634)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:626)
    at scala.collection.immutable.Stream.foldLeft(Stream.scala:302)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.traverseRelationship(PatternMatcher.scala:150)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.traverseNextSpecificNode(PatternMatcher.scala:61)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.traverseNode(PatternMatcher.scala:72)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.foreach(PatternMatcher.scala:36)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:99)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:250)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.toBuffer(PatternMatcher.scala:28)
    at scala.collection.TraversableLike$class.toStream(TraversableLike.scala:571)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.toStream(PatternMatcher.scala:28)
    at scala.collection.TraversableLike$class.toIterator(TraversableLike.scala:570)
    at org.neo4j.cypher.internal.pipes.matching.PatternMatcher.toIterator(PatternMatcher.scala:28)
    at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:371)
    at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:334)
    at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:334)
    at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:334)
    at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply$mcZ$sp(ClosingIterator.scala:36)
    at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
    at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
    at org.neo4j.cypher.internal.ClosingIterator.failIfThrows(ClosingIterator.scala:87)
    at org.neo4j.cypher.internal.ClosingIterator.hasNext(ClosingIterator.scala:35)

You should provide a sample database at console.neo4j.org, for us to play with. — Werner Kvalem Vesterås, Jan 15 '13 at 06:32
maybe a simple db cleanup would help. simply erase the folder with neo4j and unpack it again and create the data again. — ulkas, Jan 15 '13 at 09:29
@WernerVesterås I'm not sure how much of the graph you wanted, so I just tried creating the whole thing @ console.neo4j.org and I'm pretty sure I broke it:/ It was working fine, but after I tried creating my graph I got the following error: An error occurred in the application and your page could not be served. Please try again in a few moments. If you are the application owner, check your logs for details. I apologize if I caused any harm. It seems to be back up now, but I'm afraid to try it again. — Brent Barbata, Jan 16 '13 at 04:17
@ulkas Thanks for the reply. I tried recreating the graph earlier today on both 1.8.1 and 1.9.M03 but, sadly, I still have the same problem. — Brent Barbata, Jan 16 '13 at 04:23
please have a look at the logs in data/log/ and data/graph.db/messages.log for any exception to share here. Can you share a zipped copy of your db somewhere? How much memory do you use for your server? — Michael Hunger, Jan 16 '13 at 07:13
@MichaelHunger Yeah, I can definitely share a compressed copy of my db. Here's a [link](http://www.opensourcebanking.org/neo4j/). I've also updated the original question with some queries and their associated logged messages (if any). I was running this machine with 2GB RAM, but after you asked me about it, I upped it to 8GB. The interesting thing is that after increasing the memory, instead of hanging for about 3 minutes and then returning 'undefined', it now hangs for about 10-15 minutes and returns 'undefined'. (The log also shows out of memory.) — Brent Barbata, Jan 16 '13 at 11:43
I think the problem here is that there are so many relationships to and from nodes that I'm running into a A LOT of "cycles." By cycles, I mean that while the path/relationship from node to node is different, the same node is appearing in the path. Here's an example with the console: http://console.neo4j.org/?id=k34uet. Is there a way to terminate the path once it encounters two of the same nodes? I've seen the example using the "WITH ALL...", but I'm afraid it won't solve the out of memory problem since MATCH runs before WITH. Is this correct? — Brent Barbata, Jan 22 '13 at 20:48
actually match tries to pull in as many expressions as it can to reduce the match. — Michael Hunger, Feb 20 '13 at 09:02

score 0 · Answer 1 · answered Feb 20 '13 at 09:55

Did you try to use shortestPath or allShortestPaths for your queries?

 start a=node(1), b=node(2) match p=allShortestPaths(a-[:flys_to*..2]->b) return p;


+--------------------------------------------------------------------------------------------------------+
| p                                                                                                      |
+--------------------------------------------------------------------------------------------------------+
| [Node[1]{iata:"ABE"},:flys_to[224044] {},Node[546]{iata:"SFO"},:flys_to[24211] {},Node[2]{iata:"ABI"}] |
| [Node[1]{iata:"ABE"},:flys_to[224044] {},Node[546]{iata:"SFO"},:flys_to[24210] {},Node[2]{iata:"ABI"}] |
....
| [Node[1]{iata:"ABE"},:flys_to[66032] {},Node[146]{iata:"DFW"},:flys_to[1] {},Node[2]{iata:"ABI"}]      |
| [Node[1]{iata:"ABE"},:flys_to[66032] {},Node[146]{iata:"DFW"},:flys_to[0] {},Node[2]{iata:"ABI"}]      |
+--------------------------------------------------------------------------------------------------------+
28 rows
20 ms
 start a=node(1), b=node(2) match p=shortestPath(a-[:flys_to*..2]->b) return p;

+--------------------------------------------------------------------------------------------------------+
| p                                                                                                      |
+--------------------------------------------------------------------------------------------------------+
| [Node[1]{iata:"ABE"},:flys_to[224044] {},Node[546]{iata:"SFO"},:flys_to[24211] {},Node[2]{iata:"ABI"}] |
+--------------------------------------------------------------------------------------------------------+
1 row
1 ms

yeah I did try those (and they always worked super fast), but I was hoping to be able to set a maximum amount of hops and then get all paths under that limit. For example, if set to a max of 2 hops, I'd like it to return all paths with 1 hop as well as 2 hops. — Brent Barbata, Feb 20 '13 at 12:15

score 0 · Answer 2 · answered Feb 20 '13 at 09:56

0

For me this returns 426 rows:

start a=node(353), b=node(295)
match p=a-->b
return p;

answered Feb 20 '13 at 09:56

Michael Hunger

41,339
3
57
80

This is so weird. Every time I try this query I get the error I posted above. Do you think it could have anything to do with the environment I'm running it in (Win 2012 Svr and Java 6)? – Brent Barbata Feb 20 '13 at 12:20
So I've updated to Neo4j M04 and Java 7u17 and now this query seems to work fine. It's returning 426 rows now. But for those bigger queries that route through the same node multiple times, I'm still running into the out of memory problem. I'm going to try adding more properties to the relationships (like flight days and times) to reduce the number of possible paths. We'll see how that works. – Brent Barbata Mar 05 '13 at 04:26

Cypher Query in Neo4j Returns 'undefined'

2 Answers2