2

I am trying to use cosmos db change feed (I'm referring to https://learn.microsoft.com/en-us/azure/cosmos-db/change-feed-processor and https://github.com/Azure/azure-cosmos-dotnet-v2/tree/master/samples/code-samples/ChangeFeedProcessorV2).

When I start a multiple instances of a consumer, the observer seems to see only 1 partition key range. I only see a message - Observer opened for partition Key Range 0 and it starts receiving the change feed. So, the feed is received by only 1 consumer at any given point. If I close one consumer, the next one picks up happily.

I can't seem to understand the partition keys / ranges in cosmos db. In cosmos db, I've created a database and a collection within it. I've defined a partition key - /myId. I store a unique guid in myId. I've saved about 10000 transactions in the collection.

When I look at partition key ranges using api (/dbs/db-name/colls/coll-name/pkranges), I see only node under PartitionKeyRanges. Below is the output I see

{
    "_rid": "LEAgAL7tmKM=",
    "PartitionKeyRanges": [
        {
            "_rid": "LEAgAL7tmKMCAAAAAAAAUA==",
            "id": "0",
            "_etag": "\"00007d00-0000-0000-0000-5c3645e70000\"",
            "minInclusive": "",
            "maxExclusive": "FF",
            "ridPrefix": 0,
            "_self": "dbs/LAEgAA==/colls/LEAgAL7tmKM=/pkranges/LEAgAL7tmKMCAAAAAAAAUA==/",
            "throughputFraction": 1,
            "status": "online",
            "parents": [],
            "_ts": 1547060711
        }
    ],
    "_count": 1
}

Shouldn't this show more partition key ranges? Is this behavior expected?

How do I get multiple consumers to receive data as shown under https://learn.microsoft.com/en-us/azure/cosmos-db/change-feed-processor?

perf-guru
  • 91
  • 3
  • 12

1 Answers1

1

TL;DR - you should be able to ignore partition key ranges and the number of them you have and just let Change Feed Processor manage that for you.

The partition key ranges is an implementation detail we currently leak. The short answer is we add new partition key ranges when we want to restructure how your data is stored in the backend. This can happen for lots of reasons, like you add more data, you consume a lot of RUs for a subsection of that data, or we just want to shuffle things around. Theoretically, if you kept adding data, we'd eventually split the range in two.

We're working on some updates for the v3 SDKs that are currently in preview to abstract this a bit further, since even the answer I have given above is pretty hand wavey and we should have a more easily understood contract for public APIs.

Chris Anderson
  • 8,305
  • 2
  • 29
  • 37
  • Thanks! Could you confirm my interpretation - 1. As of now, I can't control the number of active consumers. 2. CosmosDb will indirectly control that through partition key ranges. 3. Will v3 SDKs allow me to control the number of active consumers? – perf-guru Jan 11 '19 at 01:43
  • 1
    1. A given Change Feed Processor Host Id will assign 1 or many PK Ranges to a given Change Feed Processor Host instance. You can control the number of instances, but you can't effectively exceed the PK range count. So yes, you can control it up to a max. 2. You can't control directly the number of PK Ranges. We control that. 3. V3 will just further abstract away the PK Range concept so you only know the max number of consumers you're currently able to effectively use. It will not give you more control over the # of PK Ranges. We have no current plans to give control for that. – Chris Anderson Jan 11 '19 at 21:38
  • So just to be clear, what you are saying is that I cannot decide how much parallelism I want for a given change feed processor, this will be decided for me by Cosmos DB. Then that means if my change feed processor is slow at processing the feed then I cannot increase the speed by just increasing the parallelism through increasing the number of instances? In this documentation https://learn.microsoft.com/en-us/azure/cosmos-db/change-feed-processor it seems the change feed processor are scaleable, but they really are not then? :( – svenskmand Aug 27 '20 at 09:53