0

We are having a strange issue with the change feed subscriber isnt picking up new records after the cosmos db is redeployed in a different service. My assumption was when the feed is down it would then pick up from where it left off once the container becomes available again?

Was curious to know if anyone else has seen this behaviour before or if its potentially something to do with the configuration of the feed?

To add a bit of context manually deleting and recreating the containers seems to fix the issue so it seems deployment-related?

This is what our change feed configuration looks like.

 var changeFeedProcessorBuilder = (await _cosmosProvider
                .GetContainer(_config.DatabaseName, _config.SourceContainerName))
                .GetChangeFeedProcessorBuilder<CosmosAggregateEventEntity>(_projector.Name, OnMessage)
                .WithInstanceName($"Processor{Guid.NewGuid().ToString().Replace("-", string.Empty)}")
                .WithLeaseContainer(await _cosmosProvider.GetContainer(_config.DatabaseName, CosmosChangeFeedConfiguration.LeaseContainerName, "/id"))
                .WithStartTime(DateTime.MinValue.ToUniversalTime());

            if (_defaultPollingTimeOverride != null)
                changeFeedProcessorBuilder.WithPollInterval((TimeSpan)_defaultPollingTimeOverride);

            _processor = changeFeedProcessorBuilder.Build();

            await _processor.StartAsync();
Chris Marshall
  • 740
  • 1
  • 9
  • 25
  • What do you mean by "cosmos db is redeployed in a different service"? – Mark Brown May 27 '20 at 18:27
  • So in this case we have two microservices here one is called ProductStore which is the read API and the second is called ProductCatalog which it the write API. When something is created it gets added to the event store in the ProductCatalog and the ProductStore listens to the changefeed which ads it to the stores inside ProductStore which then creates projectsions of the data that was created in the write API. When we redeploy ProductCatalog we experience issues with the change feed where we manually have to delete the containers before it will pick up new entries which is odd. Hope this helps – Chris Marshall May 27 '20 at 18:35
  • Are you deleting and recreating the source container? What are the issues you are seeing? The Change Feed Processor, if stopped and then restarted, will pick up the state that was previously saved on the leases if the monitored and lease containers are the same and were not deleted/recreated. – Matias Quaranta May 27 '20 at 22:14
  • Interesting you say that as I've just had a look at our pipeline and it looks like during a fresh deployment it updates the cosmos db account but it doesnt delete the db or containers so it should still be able to use that. But.im now wondering if cosmos cant be updated after something has subscribed – Chris Marshall May 27 '20 at 22:17
  • Technically, the lease container contains the lease checkpoints for the monitored container. if you do an update on the Cosmos DB account, that's a control plane operation and does not affect collections, nor data. So the Change Feed Processor should be able to keep working. What is the actual error you are getting? – Matias Quaranta May 28 '20 at 18:48
  • 1
    So after another day of debugging it turns out we were deploying later in the pipeline which was creating a new provider's as everytime the service starts it generates another one which was the issue so were currently reworking this so that's not the case – Chris Marshall May 28 '20 at 20:05
  • It would be great if you can share your finding/solution as an answer in case anyone runs into something similar in the future :) – Matias Quaranta May 29 '20 at 00:06
  • Will do once it's all in working order and we understand the issue properly. – Chris Marshall May 29 '20 at 13:48

0 Answers0