3

I'm using async/await to make multiple requests per second to a service using their API. The problem I am running into is when I need to refresh the token (it expries every hour). After the token has expired, I get a 401 unauthorized error back from the service. That's when I refresh the token, and retry the failed request again. The token gets refreshed fine, but what I'm finding is that even after the token is refreshed, a lot of subsequent requests are still sent with the old token. Following are the methods that are used in this functionality. Wondering if anything standouts as being the culprit for this unintended behavior.

public void Process(id)
{
    var tasks = items.Select(async item =>
    {
        var response = await SendRequestAsync(() => CreateRequest(item.Url));
        //do something with response
        await Process(item.subId); //recursive call to process sub items.
    }).ToList();

    if (tasks.Any())
        await Task.WhenAll(tasks);      

}

public HttpRequestMessage CreateRequest(string url)
{
    var request = new HttpRequestMessage(HttpMethod.Get, url);
    request.Headers.Add("Authorization", "Bearer " + AppSettings.AccessToken); 
    return request;
}

public async Task<HttpResponseMessage> SendRequestAsync(Func<HttpRequestMessage> funcReq)
{   
    var response = await ExecuteRequestAsync(funcReq());

    while (response.StatusCode == HttpStatusCode.Unauthorized)
    {
        await RefreshTokenAsync();
        return await ExecuteRequestAsync(funcReq());  //assuming func ensures that CreateRequest is called each time, so I'll always have a new request with the updated token.
    }  

    return response;            
}

private async Task<HttpResponseMessage> ExecuteRequestAsync(HttpRequestMessage request)
{
    var client = new HttpClient();
    var response = await client.SendAsync(request);
    return response;    
}

public async Task RefreshTokenAsync()
{
    await semaphoreSlim.WaitAsync();
    try
    {      
        if ((DateTime.Now - refreshTime).TotalMinutes < 60) //tokens last for an hour, so after the refresh is made by the first request that failed, subsequent requests should have the latest token.
            return; 

        Token newToken = GetNewToken();
        AppSettings.AccessToken = newToken.AccessToken //AppSettings is a singleton wrapper class for app.cofig app settings
        refreshTime = DateTime.Now
    }
    finally
    {
        semaphoreSlim.Release();
    }
}
Prabhu
  • 12,995
  • 33
  • 127
  • 210
  • 1
    How much longer are requests being executed with the same token? Have you eliminated the possibility that those are requests which were in progress/started before the token was refreshed? – Marcel N. Aug 11 '14 at 19:53
  • @MarcelN. At first I get a bunch of 401 errors--that's expected. But my semaphore should ensure that only the first failed request will refresh the token. The remaining ones should then have the updated token by the time they are sent out. Though some requests do go out with the new token, a lot of requests are still going out with the old one. Some retry requests are going with the old token for more than an hour after the refresh. – Prabhu Aug 11 '14 at 20:21
  • How long do your requests take? It sounds very reasonable to assume what @MarcelN said about previous requests not finishing execution. Also, how are you initalizing your `SemaphoreSlim`? – Yuval Itzchakov Aug 11 '14 at 20:22
  • I don't think that's entirely true. Some requests may already be at this point `var response = await ExecuteRequestAsync(funcReq());`, after `funcReq` evaluated, when the token gets refreshed. By this time it's too late for them to get the new token. They will get executed with the old one. – Marcel N. Aug 11 '14 at 20:23
  • @YuvalItzchakov But for any given request, I can expect that the second time that same request is sent out, that they should have the latest token, right? – Prabhu Aug 11 '14 at 20:24
  • @YuvalItzchakov new SemaphoreSlim(1) in the constructor. – Prabhu Aug 11 '14 at 20:25
  • @MarcelN. You're right, but that just means that several of my requests will have the old token when they are fired for the first time. But when a request fails, they have to go through the RefreshToken routine and should have the updated token when they are sent out the second time right? My concern is only for retry requests to have the updated token. – Prabhu Aug 11 '14 at 20:27
  • @Prabhu: Agreed, but I am not sure if there's any proof there are requests that execute one or more times with the same token. That would mean this `() => CreateRequest(item.Url)` is cached, which cannot happen. – Marcel N. Aug 11 '14 at 20:29
  • Have you debugged requests that go through `RefreshTokenAsync` and still get the old token? Also, why are you using a `Func` instead of simply calling `CreateRequest` yourself? – Yuval Itzchakov Aug 11 '14 at 20:29
  • @YuvalItzchakov the CreateRequest is actually in a different class, I just simplified it for the discussion here, so in my real case I use funcs so I can invoke it back from the called class. As for debugging, I can't seem to isolate the issue. It seems to work as expected when debugging. But I think I'll have to try again. I did try probing the value of AppSettings.AccessToken just before client.SendAsync, and it is always the latest token. – Prabhu Aug 11 '14 at 20:38
  • Have you maybe considered a locking mechanism during your update of the token? So all other callers will have to wait for the updated token and cant proceed until then? – Yuval Itzchakov Aug 11 '14 at 20:40
  • @MarcelN. So anyhow do I have the right approach? Is there a better way to handle the token refresh and hand over the new token to all subsequent requests? Is there a way to easily update the request header for all pending requests? – Prabhu Aug 11 '14 at 20:40
  • @YuvalItzchakov Isn't the semaphore in the RefreshTokenAsync method enough to ensure exclusive/serial access? Since this is an async situation, I assumed that using a semaphore would be more appropriate than using readerwriterlocks. – Prabhu Aug 11 '14 at 20:43
  • 2
    @Prabhu: I'm not sure about better, but you could **not wait** for a 401 to refresh the token. Instead, refresh it every 60 minutes and serialize access only when regenerating. This implies that in `CreateRequest` you won't use `AppSettings.AccessToken` anymore, but a getter which does the job. This way you can be positive you always have the latest token. The downside is if the API changes the interval you'll have to adjust. – Marcel N. Aug 11 '14 at 20:50
  • @Prahbu No, it doesn't. Using the semaphore ensures only one caller can access `RefreshTokenAsync`, but they can freely access `AppSettings.AccessToken` as much as they like, even **while** the token is being updated – Yuval Itzchakov Aug 11 '14 at 20:55
  • @MarcelN. Interesting, I never thought of proactively refreshing the token. I think that's a good idea. I'll still need to handle the 429s because the 60 mins is not a guarantee. Could you please elaborate on the getter? I am not seeing how that would ensure the latest token as opposed to AppSettings.AcessToken. – Prabhu Aug 11 '14 at 20:58
  • @YuvalItzchakov if only one caller can access RefreshTokenAsync at a time, then only one caller can update it, right? And all callers that access it the moment after it's updated should get the latest value, isn't it? Am I forgetting something basic? – Prabhu Aug 11 '14 at 21:00
  • But while the `RefreshTokenAsync` is executing, they're still consuming the old value. – Yuval Itzchakov Aug 11 '14 at 21:02
  • @YuvalItzchakov According to my code, all failed requests MUST go through the RefreshToken routine, and since entry to RefreshToken is serialized, all retry requests will access the latest value of AppSettings.AccessToken by the time they get to it, isn't it? I'm interested in knowing more of your thoughts because that may be the reason for my issue. – Prabhu Aug 11 '14 at 21:07
  • @Prabhu Consider checking the tokens validity before making the request, perhaps in `CreateRequest`. That way, the first request to encounter an expired token locks, and updates the token. That way, you save yourself the unnessacery 401 – Yuval Itzchakov Aug 11 '14 at 21:08
  • All failed request will go through your token refresh. But while the the first 401 occurs and tries to update the token, other `CreateRequest` method are running in the background and could still be consuming the old value. Should all of them refresh the token? Or is it global? – Yuval Itzchakov Aug 11 '14 at 21:11
  • @YuvalItzchakov The CreateRequest gets invoked only after awaiting RefreshToken, so I'm assuming that no CreateRequests are happening in the background when the token is being refreshed, is my assumption not correct? I'm only referring to second-time requests after the first attempt failed. My whole concern is only about retry requests having the old access token so I don't really mind if the initial first-attempt requests have the old token. But once a requestfails, I want to make sure it has the new token before it goes out the second time. – Prabhu Aug 11 '14 at 21:24

2 Answers2

1

This is not an answer. It's just I don't have where to post the code discussed in one of the comments.

Prabhu, I think something like this should work in order to have the token updated prior to getting a 401. This works only if you can make an assumption on how often tokens expire.

public HttpRequestMessage CreateRequest(string url)
{
    var request = new HttpRequestMessage(HttpMethod.Get, url);
    request.Headers.Add("Authorization", "Bearer " + GetUpToDateAccessToken()); 
    return request;
}

private Token GetUpToDateAccessToken()
{
    _readWriteLockSlim.EnterReadLock();
    try
    {      
        return _latestToken;
    }
    finally
    {
        _readWriteLockSlim.ExitReadLock();
    }  
}

Updating the token can be done with a Timer every 60 minutes. Synchronization is done with a read-write lock. This would be the Timer's tick handler (you can use a System.Timers.Timer).

private void UpdateToken()
{  
  _readWriteLockSlim.EnterWriteLock();
  try 
  {
     if ((DateTime.Now - refreshTime).TotalMinutes >= 60) 
     {
         Token newToken = GetNewToken();
         _latestToken = newToken.AccessToken;
          refreshTime = DateTime.Now;
     }
  }
  finally
  {
    _readWriteLockSlim.ExitWriteLock();
  }
}

As you mentioned, if the 60 minute expiry period is not guaranteed then this won't work as expected. Maybe you can re-generate the token every 5 minutes or so, to make sure you don't make requests with invalid tokens.

Finally, to handle 401s because they can still occur, you can modify the while loop in SendRequestAsync to:

if (response.StatusCode == HttpStatusCode.Unauthorized)
{
   UpdateToken();
   return await ExecuteRequestAsync(funcReq());
}
Marcel N.
  • 13,726
  • 5
  • 47
  • 72
  • @YuvalItzchakov: Thanks for the edit, but I was just adding a `_readWriteLockSlim.EnterWriteLock();` where the blank line was :). – Marcel N. Aug 11 '14 at 21:16
  • LOL, Sorry about that :) – Yuval Itzchakov Aug 11 '14 at 21:18
  • @Marcel Thank you. Could I use this along with handling 401 in case I do get it, so that I can be sure that I don't lose any requests? – Prabhu Aug 11 '14 at 21:20
  • 1
    @Prabhu: If you do get a 401 then I guess you could call `UpdateToken` which enters a critical section anyway. But the weird "race condition" can still occur, less often though, if two or more requests are started before one 401s. – Marcel N. Aug 11 '14 at 21:21
  • @MarcelN. As long as I can at least ensure that every single request made the second time will have the updated token, I don't mind the occasional race condition. – Prabhu Aug 11 '14 at 21:32
  • @MarcelN. Just to wrap up my thoughts around this. So, where do you think is the hole in my current code that could let a second attempt request go out with the old access token? – Prabhu Aug 11 '14 at 21:34
  • @Prabhu: I don't think there's ever a possibility to have a request go out at a second attempt with the old access token in your current code. There is possible, however, to have too many requests start with an expired token. – Marcel N. Aug 11 '14 at 21:41
  • @MarcelN. it seems from my logs that there are several reattempt requests that went out with the old access token. Is something else going on? – Prabhu Aug 11 '14 at 21:57
  • 1
    @Prabhu: The only thing that crosses my mind now is this condition `if ((DateTime.Now - refreshTime).TotalMinutes < 60)`. If the server rejects the requests **AND** the difference is less than 60 minutes then requests will still use the old token. It's the only way. – Marcel N. Aug 11 '14 at 22:09
  • @MarcelN. I think there is another likely scenario where that could happen. Since the refresh token logic is in a while loop, when the first batch of 401s are received, they'll be sent out with the renewed token, but if the token gets refreshed a second time before some of the retry requests go out, the "new" token they (the retry requests) have would have become stale too. Is my thinking valid? – Prabhu Aug 11 '14 at 23:42
  • @Prabhu: Which one was it? – Marcel N. Aug 12 '14 at 16:42
  • @MarcelN. Another question--since this is an async scenario where I am using ConfigureAwait(false) (so the caller's context may be lost), I should really be using SemaphoreSlim for synchronization, right, rather than readerwritelocks? – Prabhu Aug 18 '14 at 16:00
1

I will suggest the following workflow (added to the currently suggested from Marcel N) (pseudo code) :

// Manage the expiration token yourself in the Application or Db
var token = GetTokenFromDbOrApplicationWithExpirationDateTime()
// Expiration on your application can be a little bit less than real, so instead of 60 can be 50 minutes.
if (token.isExpired)
    token = RequestNewToken()
    Db.SaveChanges(token);
}

CallMethod1Async(token)
CallMethod2Async(token)
CallMethod3Async(token)

you also may want to check if the CallMethodAsync returns a response with invalid token as Marcel N provided.

Bart Calixto
  • 19,210
  • 11
  • 78
  • 114