I have long running jobs that use Azure MSI tokens to access ADLS Gen 2 storage. I am running into Azure server side token caching issue. This is my setup:
- I request access token for a user managed identity from Azure, the default TTL is 8 hours.
- That token is then given a long running job, say the job runs for 5 hours and finishes successfully.
- Then my code requests a new token from Azure IMDS endpoint. Azure, returns me the same access token (now with effective TTL of 3 hours).
- This token is given to a new long running job which fails after 3 hours.
I could not find any documentation on how to force a token refresh, so that when I request the second token it's expiry time is 8 hours after the second request. In fact I found a blog that says that force refresh is not possible.
This does not look right there should be a way to force a token refresh (get a new token with new expiry). Has anybody come across this issue and how did you solve it.
NOTE: The processes the job runs on does not have the ability to request tokens from Azure endpoint.