0

I have the following pagination query:

g.V().hasLabel('my-label').flatMap(out().order().by(id)).range(15000, 15299)

and I getting the following error:

[ERROR] GremlinServerError: 500: {"code":"MemoryLimitExceededException","requestId":"...","detailedMessage":"Query cannot be completed due to memory limitations."}

So is order() causing the db to load all nodes into memory, and caused memory issue? How can I get around this?

stephen mallette
  • 45,298
  • 5
  • 67
  • 135
user1187968
  • 7,154
  • 16
  • 81
  • 152
  • It's possible there is not enough memory to perform the `order` depending on the amount of vertices yielded by the `out`. I don't think the `flatMap` is adding any value here so trying without it may also help with query optimization. What instance size are you using? – Kelvin Lawrence Aug 18 '21 at 22:00
  • @KelvinLawrence Using instance size db.t3.medium – user1187968 Aug 18 '21 at 22:01
  • A bigger instance size will have more memory for each query execution worker and may help. It really depends how much data that `out` step is returning. – Kelvin Lawrence Aug 18 '21 at 22:44

1 Answers1

0

Adding an answer in case others find this discussion. At time of writing, the db.t3.medium instance size is the smallest one supported by Amazon Neptune. The Neptune architecture is such that each instance has a worker pool of threads that is typically equal to twice the number of vCPUs on the instance. Memory and other resources have to be divided up amongst those threads. A large chunk of the memory is also taken up by the cache that is used to keep graph data local to the instance as much as possible. Larger instances have more vCPU and memory and therefore a query has more resources. In many cases out of memory errors are a sign that you just need to deploy a larger instance.

You can use the /profile API to see how many traversers your query is spawning and that is often a good indication of how much memory it is taking up. You may need to use a limit to get a profile if you want to try it on the T3 instance.

Kelvin Lawrence
  • 14,674
  • 2
  • 16
  • 38