Slow performance on Azure DocumentDB

Question

I'm currently facing quite slow response times from Azure DocumentDB (first time trying it).

There are 31 objects in a collection, which I am going to fetch and return to the caller. The code I am using is this:

public async Task<List<dynamic>> Get(string collectionName = null)
{
    // Lookup from Dictionary, takes literally no time
    var collection = await GetCollectionAsync(collectionName);

    var sw = Stopwatch.StartNew();

    var query = await
        _client.CreateDocumentQuery(collection.DocumentsLink, 
            new FeedOptions { MaxItemCount = 1000 })
            .AsDocumentQuery()
            .ExecuteNextAsync();

    Trace.WriteLine($"Get documents: {sw.ElapsedMilliseconds} ms");

    return query.ToList();
}

To instantiate the client, I'm using the following code:

_client = new DocumentClient(new Uri(endpoint), authKey, new ConnectionPolicy
{
    ConnectionMode = ConnectionMode.Direct,
    ConnectionProtocol = Protocol.Tcp
});

The response times I am getting from the Stopwatch is between 360ms and 1200ms to return 31 objects. For me, that is quite slow. Without the custom ConnectionPolicy the average response time is ca 950ms.

Am I doing something wrong here? Is it possible to speed these requests up somehow?

Here is the output from the Trace, printing out the Stopwatch's elapsed time:

Get documents: 1984 ms
Get documents: 1252 ms
Get documents: 1246 ms
Get documents: 359 ms
Get documents: 356 ms
Get documents: 356 ms
Get documents: 351 ms
Get documents: 1248 ms
Get documents: 1314 ms
Get documents: 1250 ms

Have you tried running the code in the same data center as your DocumentDB instance? I'm very disappointed running operations from my system on a fast internet connection (250ms minimum for even one operation), but the latency is under 10ms when I run in the same data center. If you don't want to do the experiment to push it to the data center, then try a query with 10x the amount of data. My suspicion is that you'll see only slightly increased numbers. If so, that would give you evidence that it is the latency of calling it from outside the data center. — Larry Maccherone, Aug 29 '15 at 00:39
I do think there is a case to be made that crossing the data center boundary is too expensive from a latency perspective. The ping time on my internet connection is only 50-70ms. Even doubling that doesn't explain a minimum 250ms latency. — Larry Maccherone, Aug 29 '15 at 00:40
I can say this much: retrieving 31 rows from a SQL server (Azure) in the same data center is _much_ faster than this. Even when the rows have more data than the objects retrieved from DocumentDB. — Andre Andersen, Aug 29 '15 at 01:06
Are you saying that you are running this code in the same data center and still getting those numbers? If you retrieve 31 rows from DocumentDB in the same data center, you should get much faster response. If not, then it must be something else. It would be interesting to see how Azure SQL compares from a latency perspective when everything is in the same data center. — Larry Maccherone, Aug 29 '15 at 02:35
I can't comment on your code because I use docdb on a NodeJS application and have no experience with the .Net sdk, but I've run rather sophisticated queries against collections with over 5K documents that return within 50ms. Perhaps your index policy is not properly configured? What does the x charge header of your queue return? — Luis Delgado, Aug 30 '15 at 17:39

score 26 · Accepted Answer · edited Jan 22 '17 at 18:23

Updated to reflect latest service changes (1/22/2017): DocumentDB guarantees p99 read latency < 10 ms and p99 write latency < 15 ms with SLAs on the database side. The tips below still apply to achieve low latency reads using the SDKs**

Updated to reflect latest service changes (6/14/2016): There is no need to cache self-links when using routing via user-defined ids. Also added a few more tips.**

Reads typically take <1 ms on the DocumentDB storage partition itself; and the bottleneck is often the network latency between the application and the database. Thus, it is best to have the application running in the same datacenter as the database.

Here are some general tips on SDK usage:

Tip #1: Use a singleton DocumentDB client for the lifetime of your application

Note that each DocumentClient instance is thread-safe and performs efficient connection management and address caching when operating in Direct Mode. To allow efficient connection management and better performance by DocumentClient, it is recommended to use a single instance of DocumentClient per AppDomain for the lifetime of the application.

~~Tip #2: Cache document and collection SelfLinks for lower read latency~~

In Azure DocumentDB, each document has a system-generated selfLink. These selfLinks are guaranteed to be unique and immutable for the lifetime of the document. Reading a single document using a selfLink is the most efficient way to get a single document. Due to the immutability of the selfLink, you should cache selfLinks whenever possible for best read performance.

Document document = await client.ReadDocumentAsync("/dbs/1234/colls/1234354/docs/2332435465");

Having said that, it may not be always possible for the application to work with a document’s selfLink for read scenarios; in this case, the next most efficient way to retrieve a document is to query by the document’s user provided Id property. For example:

IDocumentQuery<Document> query = (from doc in client.CreateDocumentQuery(colSelfLink) where doc.Id == "myId" select document).AsDocumentQuery(); 
            Document myDocument = null;
            while (query.HasMoreResults)
            {
                FeedResponse<Document> res = await query.ExecuteNextAsync<Document>();
                if (res.Count != 0) {
                    myDocument = res.Single();
                    break;
                }
           }

Tip #3: Tune page size for queries/read feeds for better performance

When performing a bulk read of documents using read feed functionality (i.e. ReadDocumentFeedAsync) or when issuing a DocumentDB SQL query, the results are returned in a segmented fashion if the result set is too large. By default, results are returned in chunks of 100 items or 1 MB, whichever limit is hit first.

In order to reduce the number of network round trips required to retrieve all applicable results, you can increase the page size using x-ms-max-item-count request header to up to 1000. In cases where you need to display only a few results, e.g., if your user interface or application API returns only ten results a time, you can also decrease the page size to 10 in order to reduce the throughput consumed for reads and queries.

You may also set the page size using the available DocumentDB SDKs. For example:

IQueryable<dynamic> authorResults =
client.CreateDocumentQuery(documentCollection.SelfLink, "SELECT p.Author FROM Pages p WHERE p.Title = 'About Seattle'", new FeedOptions { MaxItemCount = 1000 });

A few more tips (6/14/2016):

Use point-reads (e.g. read document instead of query document) for lookup by id
Configure the DocumentDB client (using ConnectionPolicy) to use direct connectivity over gateway
Collocate clients in the same Azure Region as your database
Call OpenAsync() to prevent higher first call latency
You can debug LINQ queries by calling ToString() on the queryable to see the SQL query sent over the wire

For more performance tips, check out this blog post.

A more detailed list on performance imporvements: https://learn.microsoft.com/en-us/azure/cosmos-db/performance-tips — Jochen Kalmbach, Mar 29 '18 at 07:22
@Andy - tip2 is no longer relevant as using selflinks is deprecated: https://azure.microsoft.com/en-us/blog/azure-documentdb-bids-fond-farewell-to-self-links/ — Imre Pühvel, Jul 09 '18 at 08:35
Also you can forget using direct mode if using the node SDK. They have been promising to add support but still nothing and still slow. — Tony Gutierrez, Feb 14 '19 at 19:00

Slow performance on Azure DocumentDB

1 Answers1

Linked