Timeouts in datastore queries

Question

I am using objectify v5.1.11 from auto scaled app engine instances in java8 runtime environment.

I have an API which IOT devices call periodically to upload statistics information. In this API, I insert an entity to datastore to store the statistics information. This entity uses the auto generated IDs of datastore. Entity definition is as follows:

@Entity(name = "Stats")
public class StatsEntity {
    @Id
    private Long statisticsId;

    @Index
    private Long deviceId;

    @Index
    private String statsKey;

    @Index
    private Date creationTime;
}

But I had a requirement of checking for duplicates before inserting the entity. I switched to custom generated (String) Ids. I came up with a mechanism of appending the deviceId to statsKey (unique for each statistic within device) string provided by the device to generate the ID.

This is to avoid the eventual consistency behaviour if I use query to check if the entity already exists. Since get by ID is strongly consistent I can use it to check for duplicates.

There is another API to fetch the statistics uploaded by a device. In this API, I list the entities by filtering on deviceId and order by creationTime in descending order (newest first) with a page size of 100. This request times out since the request exceeds the 60s limit of appengine. I see the following exception in the logs:

Task was cancelled.
java.util.concurrent.CancellationException: Task was cancelled.
    at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1355)
    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:555)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:436)
    at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:99)
    at com.google.appengine.tools.development.TimedFuture.get(TimedFuture.java:42)
    at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:93)
    at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:69)
    at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33)
    at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:243)
    at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:180)
    at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:173)
    at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:70)
    at com.googlecode.objectify.impl.KeysOnlyIterator.hasNext(KeysOnlyIterator.java:29)
    at com.google.common.collect.Iterators$5.hasNext(Iterators.java:580)
    at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
    at com.googlecode.objectify.impl.ChunkIterator.hasNext(ChunkIterator.java:39)
    at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50)
    at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50)
    at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1105)
    at com.googlecode.objectify.impl.ChunkingIterator.hasNext(ChunkingIterator.java:51)
    at com.ittiam.cvml.dao.repository.PerformanceStatsRepositoryImpl.list(PerformanceStatsRepositoryImpl.java:154)
    at com.ittiam.cvml.service.PerformanceStatsServiceImpl.listPerformanceStats(PerformanceStatsServiceImpl.java:227)

The statsKey provided by the device is based on time and hence monotonically increasing (step increase of 15 mins) which is bad as per this link. But my traffic is not large enough to warrant this behaviour. Each device makes 2 to 3 requests every 15 minutes and there are about 300 devices. When I try to list entities for devices which haven't made any request since I made this switch to custom ID, I still observe this issue.

Edit

My code to list the entity is as follows:

Query<StatsEntity> query = ofy().load().type(StatsEntity.class);

List<StatsEntity> entityList =
        new ArrayList<StatsEntity>();

query = query.filter("deviceId", deviceId);

query = query.order("-creationTime");

query = query.limit(100);

QueryResultIterator<StatsEntity> iterator = query.iterator();

while (iterator.hasNext()) {
    entityList.add(iterator.next());
}

Answer 1

This error usually occurs because of the write contention. The logic behind this is simple if you're having multiple transactions such as writing and reading some stuff from the same entity group concurrently.
There are various approaches to solve this problem:

• A query lives for only 30 secs but you can extend it by converting your API into a task queue. Usually handling such write contention issues you should always use a task queue which lasts for around 10 mins.
• If possible, make your entity group smaller.

You can find more approaches here.

Hope this answers your question!!!