0

I'm using Change Feed with Azure Functions. I have a CosmosDBTrigger assigned to each individual container with a unique LeaseCollectionPrefix. This seems to be working fine, but I'm not quite clear on what happens when the consumption-based Functions runtime decides to scale and create multiple active workers.

I need to ensure that multiple workers do not receive changes for the same logical partition key because I update some aggregate information and don't want concurrency to mess this up. Does the Change Feed push model automatically assign logical partition key ranges per active worker? Or will changes be delivered to the first available worker, thus, making it possible for the same logical partition key to appear in multiple workers?

user246392
  • 2,661
  • 11
  • 54
  • 96

1 Answers1

0

The Change Feed trigger uses the Change Feed Processor, see this article for more details.

For simplicity, let's scope it down to 1 container. For a particular container, you have a set of leases in your lease collection.

When the dynamic scaling decides to add or remove instances, basically the leases are equally distributed across the instances. Each lease represents a range of partition key values in your container. A lease can only be owned by one instance, and an instance can own multiple leases.

This means that, in normal circumstances, a change will always be delivered to one instance (the one owning the lease for the range that includes the document's partition key value).

Though, the trigger has an "at least once delivery" guarantee, because if a lease is being processed (reading a set of changes) at the same moment a new instance comes up and the load balancing kicks in, there is a chance that that same change is read on the new instance.

Bottom line is, you should always account for the possibility of the change happening in two instances in a distributed system, even if that possibility is remote.

Matias Quaranta
  • 13,907
  • 1
  • 22
  • 47
  • Thanks. This is exactly what I was looking for. Is there any room for configurability? For one of my containers, I need serialized processing (meaning one lease and one instance only even if the scaling decides to add more instances). How can I accomplish this? – user246392 Oct 29 '20 at 22:09
  • Sadly there is no configuration that defines a strict once delivery, and this is due to the distributed system nature if many instances are involved. If your instance count remains constant, meaning no reshufling of the leases, then you should receive items only once. But if instances start to come up/down, then duplications are possible. – Matias Quaranta Oct 29 '20 at 22:52