I am getting by policy Error while invoking Azure ML Batch end point via python SDK V2. Can anyone explain how to resolve?

Question

I am trying to invoke an azure ml batch end point vis python sdk v2

job = ml_client.batch_endpoints.invoke(
    endpoint_name="test-batch-v2",
    input=my_test_data_input,
    deployment_name="test-deploy1"
)

I am getting below error, can anyone explain how to resolve this?

error_endpoint_invoke

I am trying to create a batch end point deployment job via invoke api and it failed

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\azure\ai\ml\_telemetry\activity.py:263, in monitor_with_activity..monitor..wrapper(*args, **kwargs)
    260 @functools.wraps(f)
    261 def wrapper(*args, **kwargs):
    262     with log_activity(logger, activity_name or f.__name__, activity_type, custom_dimensions):
--> 263         return f(*args, **kwargs)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\azure\ai\ml\operations\_batch_endpoint_operations.py:316, in BatchEndpointOperations.invoke(self, endpoint_name, deployment_name, inputs, **kwargs)
    309     headers[EndpointInvokeFields.MODEL_DEPLOYMENT] = deployment_name
    311 response = self._requests_pipeline.post(
    312     endpoint.properties.scoring_uri,
    313     json=BatchJobResource(properties=batch_job).serialize(),
    314     headers=headers,
    315 )
--> 316 validate_response(response)
    317 batch_job = json.loads(response.text())
    318 return BatchJobResource.deserialize(batch_job)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\azure\ai\ml\_utils\_endpoint_utils.py:120, in validate_response(response)
    117         r_json = response.json()
    118     except ValueError:
    119         # exception is not in the json format
--> 120         raise Exception(response.content.decode("utf-8"))
    121 failure_msg = r_json.get("error", {}).get("message", response)
    122 error_map = {
    123     401: ClientAuthenticationError,
    124     404: ResourceNotFoundError,
    125     409: ResourceExistsError,
    126 }

Exception: BY_POLICY

Answer 1

I had similar error in the past. I couldn’t create any type of endpoint as it was blocked by policy which was set by azure admin. Just ask someone who has admin access to allow creation of endpoints.

Answer 2

The error message "BY_POLICY" indicates that the batch endpoint is not configured to allow invocations from your current identity.

As Azure Active Directory authentication, or aad_token, is supported by batch endpoints. This indicates that a valid Azure Active Directory authentication token must be presented to the batch endpoint URI in order to invoke a batch endpoint.

Make sure the ml_client object is created with valid credentials. As the credentials you are using should have the correct permissions to access the endpoint.

For example, if you want to execute a batch endpoint using a service principal already created in Azure Active Directory.

from azure.ai.ml import MLClient
from azure.identity import EnvironmentCredential

os.environ["AZURE_TENANT_ID"] = "<TENANT_ID>"
os.environ["AZURE_CLIENT_ID"] = "<CLIENT_ID>"
os.environ["AZURE_CLIENT_SECRET"] = "<CLIENT_SECRET>"

subscription_id = "<subscription>"
resource_group = "<resource-group>"
workspace = "<workspace>"

ml_client = MLClient(EnvironmentCredential(), subscription_id, resource_group, workspace)

You can refer to this documentation: Authorization on batch endpoints which give more details on how run jobs using different types of credentials (user's credentials and managed identity)

Also, do validate your batch endpoint creation by following the sample provided in this documentation.