2

Background: We have an Azure .NET application where we need to register a single "front end" user with multiple back end providers. Since this registration takes longer, we offload it to worker role and there are multiple worker roles. All data is stored in Azure SQL and we're using Entity Framework 5.0 as our ORM. The way we're currently setup, we read from SQL dB => process in worker role code => write/update to SQL dB to flag completion. Essentially I need to solve the traditional "multithreaded + shared data writes" problem but instead of OS scale it's at the cloud scale.

Concern: We have a race condition with multiple workers if the first worker take longer than the visibility timeout. For example, assuming two worker roles, I've marked below how both would read from SQL, think that the processing is still pending and both would proceed. It results in a last-writer-wins race condition and also creates orphaned and extra accounts on the external service providers.

Question: How can I modify this to take care of this situation elegantly? I can alter the data flow or use a per-user "cloud" lock for Mutex. Without trying to constraint anyone's thinking, in the past I speculated having a SQL based cloud lock, but couldn't really get it working in EF5.0. Here I'm trying to explore any answers, SQL based locks or not.

// Read message off Service Bus Queue
// message invisible for 1 min now, worker must finish in 1 min
BrokeredMessage qMsg = queueClient.Receive();

// Extract UserID Guid from message
Guid userProfileId = DeserializeUserIdFromQMsg(qMsg); 

// READ PROFILE FROM SQL
UserProfile up = (from x in myAppDbContext.UserProfiles select x).SingleOrDefault(p => p.UserProfileId == userProfileId);

if (up != null)
{
    List<Task> allUserRegTasks = new List<Task>();

    string firstName = up.FirstName; // <== WORKER ROLE #2 HERE
    string lastName = up.LastName;
    string emailAddress = up.Email;

    // Step 1: Async register User with provider #1, update db
    if (String.IsNullOrEmpty(up.Svc1CustId)) 
    {
        // <== WORKER ROLE #1 HERE
        Svc1RegHelper svc1RegHelper = new Svc1RegHelper(); 
        Task svc1UserRegTask = svc1RegHelper.GetRegisterTask(userProfileId, firstName, lastName, emailAddress);
        svc1UserRegTask.Start(); // <== SQL WRITE INSIDE THIS (marks "up.Svc1CustId")
        allUserRegTasks.Add(svc1UserRegTask);
    }

    // Step 2: Async register User with provider #2, update db
    if (String.IsNullOrEmpty(up.Svc2CustId))
    {
        Svc2RegHelper svc2RegHelper = new Svc2RegHelper();
        Task svc2UserRegTask = svc2RegHelper.GetRegisterTask(userProfileId, firstName, lastName, emailAddress);
        svc2UserRegTask.Start(); // <== SQL WRITE INSIDE THIS (marks "up.Svc2CustId")
        allUserRegTasks.Add(svc2UserRegTask);
    }

    Task.WaitAll(allUserRegTasks.ToArray());

    // Step 3: Send confirmation email to user we're ready for them!
    // ...

}
Community
  • 1
  • 1
DeepSpace101
  • 13,110
  • 9
  • 77
  • 127

1 Answers1

4

You can put a mutex in blob storage via blob lease. Put a try/catch around the whole thing as the AcquireLease() will fail if mutex is used by someone else

                var lockBlobContainer = cloudClient.GetContainerReference("mutex-container");
                var lockBlob = lockBlobContainer.GetBlobReference("SOME_KNOWN_KEY.lck");
                lockBlob.UploadText(DateTime.UtcNow.ToString(CultureInfo.InvariantCulture)); //creates the mutex file
                var leaseId = lockBlob.AcquireLease();

                try
                {
                    // Do stuff
                }
                finally
                {
                    lockBlob.ReleaseLease(leaseId);
                }
Igorek
  • 15,716
  • 3
  • 54
  • 92
  • If I have one blob container that represents a lock on the entire SQL DB table, then all my other workers will be sitting idle despite no data dependency. They should be free to process other user profile rows in the table as the data dependency is per-user (profile) level. So if I have N users, I'd need N blob containers (since 1 blob container = 1 lock) with that scheme. I don't like the idea of linking # of users to # of blob storage containers. – DeepSpace101 Jan 08 '13 at 23:46
  • Yes, you would need N blobs, not blob containers. Not sure why you don't like it, but it's one of the simpler ways to do a mutex in Azure. *shrug* – Igorek Jan 09 '13 at 00:54
  • I agree it's simpler in *this* code block. But it adds complexity in another part where we'd have to remove/add blobs as when users signup/go away to match N blobs with N users as # of user varies. It might be a necessary evil but I'll explore some other options in the meantime. Appreciate your answer BTW! – DeepSpace101 Jan 09 '13 at 02:27