1

I have lots of run files created by running PyTorch estimator/ ScriptRunStep experiments that are saved in azureml blob storage container. Previously, I'd been viewing these runs in the Experiments tab of the ml.azure.com portal and associating tags to these runs to categorise and load the desired models.

However, a coworker recently deleted my workspace. I created a new one which is connected to the previously-existing blob container, the run files therefore still exist and can be accessed on this new workspace, but they no longer show up in the Experiment viewer on ml.azure.com. Neither can I see the tags I'd associated to the runs.

Is there any way to load these old run files into the Experiment viewer or is it only possible to view runs created inside the current workspace?

Sample scriptrunconfig code:

data_ref = DataReference(datastore=ds,
                         data_reference_name="<name>",        
                         path_on_datastore = "<path>")
args = ['--data_dir',   str(data_ref),     
        '--num_epochs', 30,     
        '--lr',         0.01,          
        '--classifier', 'int_ext' ]  

src = ScriptRunConfig(source_directory='.',                       
                      arguments=args,                      
                      compute_target = compute_target,                       
                      environment = env,                       
                      script='train.py') 
src.run_config.data_references = {data_ref.data_reference_name: 
                                  data_ref.to_config()}
walexand3r
  • 13
  • 3
  • 1
    ouch! that sucks! you're right that the data is still stored in blob. Can you share about the "files" you are interested in? How did you used to access them from the experiments view, from the "Outputs + Logs" tab of a Run page? – Anders Swanson Mar 25 '21 at 21:49
  • Thanks for replying so quick! Yeah I was using "Outputs + Logs" as well as "Metrics" and "Details" tabs. Ideally I'd like to be able to see the run's metrics, as well as its tags as these described the arguments I'd given to the script. But looking at the ExperimentRun.dcid folder now, I can't see where metrics and tags are actually stored. I just have azureml-logs and my output dir which has the model.pth in. So maybe it's not possible for me to load this information from file? – walexand3r Mar 26 '21 at 10:06
  • I guess my priority now is to just find and register the model.pths I need, based on the tags I set previously, and leave the runs behind. – walexand3r Mar 26 '21 at 10:09
  • can you provide a mock of the `ScriptRunConfig`s that you normally provide? I'm interested in the `DataReference` or `Dataset` objects you're supplying – Anders Swanson Mar 26 '21 at 19:37
  • you might want to edit that code into your answer, as comments can sometimes get deleted and are hard to format. – Anders Swanson Mar 30 '21 at 20:48
  • You should definitely try copying the contents of the old container into the new container first. and report back to me – Anders Swanson Mar 30 '21 at 20:52
  • 1
    Have added the code to my question. I created a new workspace and copied over the whole azureml blob but the new workspace shows no runs recorded in experiments tab unfortunately. I've opened an support ticket though so will let you know what they say. – walexand3r Mar 31 '21 at 11:06

1 Answers1

0

Sorry for your loss! First, I'd make absolutely sure that you can't recover the deleted workspace. Definitely worthwhile to open an priority support ticket with Azure.

Another thing you might try is:

  1. create a new workspace (which will create a new storage account for you for the new workspace's logs)
  2. copy your old workspace's data into the new workspace's storage account.
Anders Swanson
  • 3,637
  • 1
  • 18
  • 43
  • 1
    Azure support advised I do same thing you suggested but this hasn't worked. They came back and said I have to just recreate the experiments manually :( Thanks for your help though, really appreciate it! – walexand3r Apr 12 '21 at 09:33
  • @walexand3r ouch! sorry to hear it. You've taught me to be a lot more careful about who has contributor permission to our RGs! – Anders Swanson Apr 12 '21 at 22:15