I have tried following the documentation instructions here (see my code below), and the pipeline seems to run okay. However, when I view it on ML Studio, it says the pipeline has failed because the container does not exist.
Worse, if I log into Microsoft Azure Storage Explorer, the default datastore appears to be corrupted somehow and displays the following message: The specified container does not exist..
Before running this I was able to add files and folders to the container.
I have now tried this on two separate ML instances.
Does anyone know why?
I need to persist some data, so am using an OutputFileDatasetConfig object, and I am running code below on an Azure ML compute instance.
from azureml.core import Workspace, Dataset, Datastore
from azureml.core.compute.compute import ComputeTarget
import azureml.core
from azureml.core import Workspace, Experiment
from azureml.core.environment import Environment
from azureml.pipeline.core import Pipeline
from azureml.pipeline.steps import PythonScriptStep
from azureml.core.runconfig import RunConfiguration
from azureml.data import OutputFileDatasetConfig
interactive_auth = InteractiveLoginAuthentication(tenant_id=tenant_id)
workspace = Workspace(
subscription_id, resource_group, workspace_name, auth=interactive_auth)
output_datastore = Datastore(
workspace=workspace,
name=resource_manager.get_output_datastore())
output_config = OutputFileDatasetConfig(
destination=(output_datastore, 'DomainConfiguration'))
step1 = PythonScriptStep(
name="Script",
script_name="script.py",
compute_target=compute_target,
source_directory=source_directory,
arguments=[
"--output_configuration",
output_config.as_mount(),
],
allow_reuse=True,
runconfig=runconfig,
)
steps.append(step1)
pipeline = Pipeline(workspace=workspace, steps=steps)
pipeline.validate()
experiment = Experiment(workspace,'ExperimentName')
run = experiment.submit(pipeline, regenerate_outputs=False)
