Azure DevOps Services - self hosted agents have gone Offline

Question

When starting up the windows service for the agent following error is logged in the event log:

Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.
(Level: Error; Source: VstsAgentService)

The agents are configured using a PAT. The windows services are run as Network Service. Configured according to: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops

The agents have been running successfully for weeks but s omething has broken our agents. Of course it will afterwards be connected to some type of change. Thanks in advance for any troubleshooting hints.

Extractions from Agent_20200421-125954-utc.log.

[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2020-04-21 12:59:58Z WARN VisualStudioServices] Authentication failed with status code 401.
...
[2020-04-21 12:59:58Z ERR  VisualStudioServices] POST request to https://dev.azure.com/{organization}/_apis/distributedtask/pools/1/sessions is not authorized. Details: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR  MessageListener] Catch exception during create session.
[2020-04-21 12:59:58Z ERR  MessageListener] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z INFO MessageListener] Non-retriable exception: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR  Terminal] WRITE ERROR: Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.

score 1 · Answer 1 · answered Apr 22 '20 at 05:33

Something has broken our agents.

Since completely know nothing to what are the changes and how's the broken of agent now, just give some possible suggestion and you can have a try.

Go D:\agent5, and open .credentials with text.

Make sure its content is like below:

{
  "scheme": "OAuth",
  "data": {
    "clientId": "xxxxxx", # This are generated automatically by     
    "authorizationUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token",
    "oauthEndpointUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token"
  }
}

Then save it, and try to re-run agent again.

If this way still could not make your agent available, or the .credentials file has contained the contents I mentioned above, now I guess the broken should caused on .credentials_rsaparams file.

.credentials_rsaparams is a file which has a RSA private key for OAuth token exchange between agent and server. The server hold the public key, and the server will send a new token to agent encrypted by the public key every 50 mins.

It is the file which generated by server automatically, which means you can not modified it manually since you don't know what you should input.

At present, you'd better re-configure this agent by following this steps:

1) Run Command line/Powershell line by using Run as Administrator, then cd D:\agent5.

2) Run .\config remove. And do agent configuration steps after the remove succeed.

Thanks. We have found some additional facts for delimiting the problem. No agents are working in the OLD agent pool. Even if we remove and config an old agent in the OLD agent pool it won't start to work. When creating a brand new agent (in a totally new folder on server) in the OLD agent pool it won't start to work. When I create a brand NEW agent pool and agent, that agent is immediately Online and working! What makes the difference with old and new agent pool (so we can learn, and avoid same problem in future)? Any recognition? — B Cederholm, Apr 23 '20 at 06:30

Azure DevOps Services - self hosted agents have gone Offline

1 Answers1