Missing results from cosmosDB queries over a few hours

Question

We came accross a recurring problem querying our cosmosDB during the day of the 6th of august.

We had queries with missing results, but we were still receiving a "valid" response. Batches of results were not returned in some very critic queries for our application. For instance, a request that should be returning 250 docs was returning 240. Multiple queries accross a 5-6 hours frame had those incomplete results. The data queried was undeniably present in the database.

We were querying using a snippet similar to this, using a paging of 200 docs:

public static List<T> ToList<T>(this IDocumentQuery<T> documentQuery)
{
    List<T> documents = new List<T>();
    while (documentQuery.HasMoreResults)
    {
        documents.AddRange(documentQuery.ExecuteNextAsync<T>().Result);
    }
    return documents;
}

the IDocumentQuery is initialized by casting a IQueryable, a bit like this:

queryable.AsDocumentQuery()

The solution used is similar to something found in the microsoft doc: https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.linq.documentqueryable.asdocumentquery?view=azure-dotnet

We are currently moving our data solution towards cosmosDB and this is how we noticed the problem. Do you have any ideas what could've cause this? Should we expect more problems of the same type? Do you know how we can identify that we are receiving incomplete results?

score 0 · Answer 1 · answered Aug 12 '19 at 08:38

I'm afraid that you need to set MaxItemCount property when you use CreateDocumentQuery method,otherwise the MaxItemCount default value is -1. If the data is too large, the data result set may be truncated within the throughput limit.

You could refer to this case:Pagination in Cosmos DB using Page Size and Page Number you could get some clues from the Continuation Token Example in that case.

private static async Task<KeyValuePair<string, IEnumerable<CeleryTask>>> QueryDocumentsByPage(int pageNumber, int pageSize, string continuationToken)
    {
        DocumentClient documentClient = new DocumentClient(new Uri("https://{CosmosDB/SQL Account Name}.documents.azure.com:443/"), "{CosmosDB/SQL Account Key}");

        var feedOptions = new FeedOptions {
            MaxItemCount = pageSize,
            EnableCrossPartitionQuery = true,

            // IMPORTANT: Set the continuation token (NULL for the first ever request/page)
            RequestContinuation = continuationToken 
        };

        IQueryable<CeleryTask> filter = documentClient.CreateDocumentQuery<CeleryTask>("dbs/{Database Name}/colls/{Collection Name}", feedOptions);
        IDocumentQuery<CeleryTask> query = filter.AsDocumentQuery();

        FeedResponse<CeleryTask> feedRespose = await query.ExecuteNextAsync<CeleryTask>();

        List<CeleryTask> documents = new List<CeleryTask>();
        foreach (CeleryTask t in feedRespose)
        {
            documents.Add(t);
        }

        // IMPORTANT: Ensure the continuation token is kept for the next requests
        return new KeyValuePair<string, IEnumerable<CeleryTask>>(feedRespose.ResponseContinuation, documents);
    }

As mentionned in the original post, we are paging by chunks of 200 items, which we do using the MaxItemCount property. — Theophane Fremond-Guilbault, Aug 13 '19 at 14:29

Missing results from cosmosDB queries over a few hours

1 Answers1