Data Migration into Service Fabric Stateful Service

Question

I have a stateful service that stores a bunch of data about my users that is stored in a reliable dictionary and obviously also retrieves it from there too.

However, I also have a SQL database that used to store this info. On initialization of a new stateful service instance, I need to migrate that info from my SQL database into the new reliable storage mechanism. From that point on, the stateful service is the source of truth. Ideally, I'd like to delay availability of my stateful service until this initialization process is completed.

Are there any suggestions on an approach for how to do this?

Would it be acceptable to just return an error state or throw an exception when the service is called while the state is not yet imported? That would be by far the easiest to implement. Or you can try to use a Task or TaskCompletionSource that awaits the Task before opening the communication listeners. — Peter Bons, May 21 '18 at 05:55
@PeterBons - yes, I would be happy for the service to return a "not ready" or similar exception (like what SF currently does). Which function do I override in my service to do as you suggest? — Reddog, May 21 '18 at 07:34

Peter Bons · Answer 1 · 2018-05-21T08:03:34.257

Something like does will do the trick:

public interface IStateful1 : IService
{
    Task MyMethod();
}

internal sealed class Stateful1 : StatefulService, IStateful1
{
    private bool isReady = false; 

    public Stateful1(StatefulServiceContext context)
        : base(context)
    { }

    public Task MyMethod()
    {
        if(!isReady)
            throw new NotImplementedException(); // Probably throw or return something more meaningful :-) 

        return Task.CompletedTask; // Do your thing here
    }

    protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
    {
        return new ServiceReplicaListener[0];
    }

    protected override async Task RunAsync(CancellationToken cancellationToken)
    {
        await Task.Run(() => {
            // Simulation of some work
            Thread.Sleep((int)TimeSpan.FromMinutes(5).TotalMilliseconds);
        });

        isReady = true;
    }
}

In this setup the import from the DB into the reliable collection is done in the RunAsync method.

Unfortunately, AFAIK, there is not way to plug in the communication listeners at a later time. That would make things way easier.

If CreateServiceReplicaListeners would be an async operation we could await the initialization task here, but we can't right now. Using .Wait() is not going to work as it will report that the instance is taking to long to get running and will mark the instance as unhealthy.

A complete overview of the lifecycle of a service can be found in the docs

score 0 · Answer 2 · answered Jun 06 '18 at 14:00

I am not sure if I got you right. But based on your comment I would suggest the following solution for returning the 'Not ready' response during the migration.

public interface IMigrationService
{
    bool IsDone();
}

public class MigrationService : IMigrationService
{
    private bool migrating = tu;

    public bool BeginMigration()
    {
        this.migrating = true;
    }
    public bool EndMigration()
    {
        this.migrating = false;
    }
    public bool IsDone()
    {
        return this.migrating;
    }
}

// WebHost startup class
public class Startup
{
    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        // Register a middle-ware that would short circuit responses
        // while migration is in progress.
        app.Use(
            async (context, next) =>
            {
                var migrationService = 
                    context.RequestServices.GetService<IMigrationService>();
                if (!migrationService.IsDone())
                {
                    /* short circuit the response with approriate error message */
                }

                await next();
            });

        app.UseMvc();
    }
}

public class Stateful : StatefulService
{
    private readonly IMigrationService migrationService;

    public Stateful(StatefulServiceContext context)
      : base(context)
    { 
        this.migrationService = new MigrationService();
    }

    protected override IEnumerable<ServiceReplicaListener>
        CreateServiceReplicaListeners()
    {
        /* 
            Create a listener here with WebHostBuilder

            Use Startup class with the middle-ware defined and 
            add configure services -> .ConfigureServices() 
            with services.AddSingleton<IMigrationService>(this.migrationService)
        */
    }

    protected override async Task 
        RunAsync(CancellationToken cancellationToken)
    {
        this.migrationService.StartMigration();

        /* Migration code */

        this.migrationService.EndMigration();
    }
}

This would allow you to roll-out a new version of the service that would short circuit all requests with appropriate error message while the migration is in progress.

Hope this helps.

Data Migration into Service Fabric Stateful Service

2 Answers2