Azure Service Fabric How to rerun RunAsync method when completed?

Question

I have a stateless service running a background process inside RunAsync method.

This background process must run forever. What it does is irrelevant, but it essentially polls a database every 60 seconds.

Unlike a Worker Role or WebJob, the RunAsync method in a Service Fabric service can run to completion and the service instance will stay up.

This has been observed to happen when Service Fabric signals the cancellation token passed to RunAsync.

Once Service Fabric decides to gracefully return from RunAsync method, how can I rerun my polling routine again?

And how can I determine why Service Fabric signalled cancellation token in the first place?

My current workaround is to throw an exception so the service is forced to restart.

    public class Stateless1 : StatelessService
    {       
        protected override async Task RunAsync(CancellationToken cancellationToken)
        {
                try
                {
                    await Start(cancellationToken);
                }
                catch(OperationCanceledException ex)
                {
                     throw; 

    //If an OperationCanceledException escapes from
    //RunAsync(CancellationToken) and Service Fabric runtime has requested
    //cancellation by signaling cancellationToken passed to
    //RunAsync(CancellationToken), Service Fabric runtime handles this exception and
    //considers it as graceful completion of RunAsyn(CancellationToken).                     

                }
                catch(Exception ex)
                {
                    //  log error
                    throw; 
 // force service to restart, If an exception of any other type escapes from
 // RunAsync(CancellationToken) then the process that is hosting this service
 // instance is brought down
                }

        }

        private async Task Start(CancellationToken cancellationToken)
        {
            while(true)
            {
                   cancellationToken.ThrowIfCancellationRequested(); // honour cancellation token
                try
                {


                    await PollDatabase(); 
                }

                catch(Exception ex)
                {
                    //  log error
                    throw; 
                }
                finally
                {
                    for (var i = 0; i < 12; i++)
                    {
                        cancellationToken.ThrowIfCancellationRequested();  // honour cancellation token

                        await Task.Delay(TimeSpan.FromSeconds(5), cancellationToken);
                    }
                }

            }               
        }
    }

I should add that when Service Fabric signals cancellation, no exception had occurred in PollDatabase() — puri, Jul 20 '17 at 09:14
There is a `CloseAsync`, however the fact that `RunAsync` gets a `CancellationToken` and a `CloseAsync` exists is confusing. Why would you need both? — The Muffin Man, Jul 21 '17 at 01:57

score 0 · Answer 1 · answered Jul 19 '17 at 22:31

0

Could you have the polling method return Task<T> instead of Task? You could return the reason in an object of your choosing.

https://blog.stephencleary.com/2012/02/async-and-await.html

answered Jul 19 '17 at 22:31

I actually need to know why Service Fabric needs to shut down this service instance. This does not occur due to pooling method throwing an exception. – puri Jul 20 '17 at 09:22

score 0 · Answer 2 · answered Aug 15 '17 at 14:05

Once Service Fabric decides to gracefully return from RunAsync method, how can I rerun my polling routine again?

As far as I can understand, RunAsync will be called again once it returns. On Reliable services lifecycle overview you can read the following.

For stateful reliable services RunAsync() would be called again if the service were demoted from primary and then promoted back to primary.

You can read there also what is the lifecycle of primary replicas being demoted and promoted, which does not include OnCloseAsync.

And how can I determine why Service Fabric signalled cancellation token in the first place?

I am afraid you need to search for thrown exceptions. Note that I am not sure about this answer, it is only my suspicion.

Azure Service Fabric How to rerun RunAsync method when completed?

2 Answers2