0

No token received using an azure compute cluster as a system-assigned managed identity in a training job

Package Name & Version: azure-identity==1.12.0 azure-keyvault-secrets==4.6.0

**Python Version: **:

3.9

Describe the bug I am trying to run a training job on an Azure ML compute cluster (Terraform resource (https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/machine_learning_compute_cluster). The compute cluster has been set up with a system assigned managed identity. As part part of the training job, I am trying to retrieve secrets stored in an Azure key vault. Using the Azure portal I have given get permission for the key vault to the compute cluster.

I am trying to obtain credentials using the following code:

def GetCredentials(): cred = DefaultAzureCredential(additionally_allowed_tenants=['*'], logging_enable=True, exclude_visual_studio_code_credential=True, exclude_shared_token_cache_credential=True) cred.get_token("https://management.core.windows.net/.default" return cred

This is the error message I am getting: error message

This is odd because the process works in another environment with an analogous but different set of Azure resources.

Would you be able to direct me in how to solve this issue?

Thank you :)

Expecting no error message. Successful run logs

jackc
  • 1
  • 1
  • Are you running this code from compute cluster itself? – SiddheshDesai May 19 '23 at 07:32
  • Yes. It is part of a scheduled training job that runs on the ml compute cluster – jackc May 19 '23 at 08:43
  • I have just uploaded the logs of a successful run in an analogous environment. – jackc May 19 '23 at 10:52
  • Can you add the system managed identity of compute atleast contributor role at your key vault and ML resource or the resource group in which they exist and then try to run the code in cluster? :- https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal – SiddheshDesai May 19 '23 at 11:34
  • Also, Just try adding the contributor role to the system managed identity at the subscription level itself and check? – SiddheshDesai May 19 '23 at 11:34
  • Thank you for your help on this Siddhesh. None of the solutions have proved successful thus far. – jackc May 19 '23 at 12:30
  • Instead of using DefaultAzureCredentials, Can you use - from azure.identity import ManagedIdentityCredential credential = ManagedIdentityCredential() and check the output? – SiddheshDesai May 19 '23 at 13:33
  • And retrieve the key vault secrets using this code with managed identity? -from azure.keyvault.secrets import SecretClient from azure.identity import ManagedIdentityCredential credential = ManagedIdentityCredential() secret_client = SecretClient(vault_url="https://.vault.azure.net/", credential=credential) secret = secret_client.get_secret("") – SiddheshDesai May 19 '23 at 13:36

1 Answers1

0

I guess it was you that also asked this on github. :)

Instead of managed identity does service principal work for you?

Service principal should work in AML managed environment.

(I work for Microsoft in the Azure SDK team)

Xiang Yan
  • 194
  • 6
  • Hi Xiang. Yes it was :). How do I make this change from managed identity to service principal to find out? – jackc May 20 '23 at 07:05