1

I have a few AppEngine operations that take more time than expected, so I've been looking at the trace logs on the GAE console to get some insights.

One of them consists in a simple query to sort entities, such as 

(data, cursor, more) = query.order(-cls.created).fetch_page(RESULTS_PER_PAGE, start_cursor=cursor)

but according to the logs, and if I understand the trace correctly (screenshot below), the operation run 2 queries: one to get the data (RunQuery), one to find if there's more data (Next). Neither take too much time, but there's a 600ms delay between both.

I'm wondering why this delay both is too long? I've seen some issues about Next() where queries took longer (How do I prevent application calling datastore_v3.next() when calling get_multi?), but the time for Next() in my case is very minimal. The bottleneck seems to be the time between both. This happens in lots of cases - the previous NDB query is just an example.

So:

  1. Can I avoid calls to Next() or
  2. Can I minimize the delay between RunQuery and Next()

I'm using a standard instance, and don't have any specific setup in my app.yaml file, besides an inbound warmup.

enter image description here

Community
  • 1
  • 1
apassant
  • 167
  • 3
  • 11

2 Answers2

1

use .fetch() instead of .fetch_page() ... the former fetches up to a limit and stops... it doesn't return any information about whether there are more results

Nicholas Franceschina
  • 6,009
  • 6
  • 36
  • 51
  • Yes, that's also what I was advise on a separate conversation - as there's no way to control the latency between the RunQuery() and Next(). However, it won't work if I need to know when I have more results (hence the use of fetch_page()) – apassant Jun 05 '15 at 08:33
  • consider using a query iterator... there is a special `probably_has_next()` call which uses some shortcuts for efficiency, but I'm not an authority on it: https://cloud.google.com/appengine/docs/python/ndb/queries#iterators – Nicholas Franceschina Jun 05 '15 at 15:12
0

This is just speculation, but I think the reason that query results are broken into multiple chunks (.Next) is because of the size of each chunk. If you can reduce the size of your entities or use Projection queries to shrink the results, you may see better performance.

Brent Washburne
  • 12,904
  • 4
  • 60
  • 82