7

I'm trying to submit an experiment to Azure ML using a Python script.

The Environment being initialised uses a custom Dockerfile.

env = Environment(name="test")
env.docker.base_image = None
env.docker.base_dockerfile = './Docker/Dockerfile'
env.docker.enabled = True

However the DockerFile needs a few COPY statements but those fail as follow:

Step 9/23 : COPY requirements-azure.txt /tmp/requirements-azure.txt
COPY failed: stat /var/lib/docker/tmp/docker-builder701026190/requirements-azure.txt: no such file or directory

The Azure host environment responsible to build the image does not contain the files the Dockerfile requires, those exist in my local development machine from where I initiate the python script.

I've been searching for the whole day of a way to add to the environment these files but without success.

Below an excerpt from the Dockerfile and the python script that submits the experiment.

FROM mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04 as base
COPY ./Docker/requirements-azure.txt /tmp/requirements-azure.txt # <- breaks here

[...]

Here is how I'm submitting the experiment:

from azureml.core.environment import Environment
from azureml.core import Workspace
from azureml.core.model import Model
from azureml.core import Workspace, Experiment
from azureml.core.compute import ComputeTarget

from azureml.core import Experiment, Workspace
from azureml.train.estimator import Estimator
import os

ws = Workspace.from_config(path='/mnt/azure/config/workspace-config.json')
env = Environment(name="test")
env.docker.base_image = None
env.docker.base_dockerfile = './Docker/Dockerfile'
env.docker.enabled = True
compute_target = ComputeTarget(workspace=ws, name='GRComputeInstance')
estimator = Estimator(
    source_directory='/workspace/',
    compute_target=compute_target,
    entry_script="./src/ml/train/main.py",
    environment_definition=env
)
experiment = Experiment(workspace=ws, name="estimator-test")
run = experiment.submit(estimator)
run.wait_for_completion(show_output=True, wait_post_processing=True)

Any idea?

Giuseppe Romagnuolo
  • 3,362
  • 2
  • 30
  • 38
  • Hi Can you please print oath before line 9/23? And share the details of the path script is running from? I am suspecting it could be just that. – A Modgil Aug 08 '20 at 13:35

2 Answers2

0

I think the correct way to setup the requirements.txt for your project is using Define an inference configuration as:

name: project_environment
dependencies:
    - python=3.6.2
    - scikit-learn=0.20.0
    - pip:
        # You must list azureml-defaults as a pip dependency
    - azureml-defaults>=1.0.45
    - inference-schema[numpy-support]

See this

LinPy
  • 16,987
  • 4
  • 43
  • 57
  • Thanks @LinPy but I'm trying to get training to work using a custom Dockerfile. See https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-environments#enable-docker, it specifically talks about training _"The DockerSection of [..] allows you to finely customize [..] the guest operating system on which you run your training."_ I'd like to define a single training environment that can run locally or on Azure by leveraging a same Docker image. This article gives some details: https://towardsdatascience.com/unifying-remote-and-local-azureml-environments-bcea1292e37f – Giuseppe Romagnuolo Aug 06 '20 at 14:39
  • did you see the warning regarding using "user_managed_dependencies" , your way providing the requirment file is not the right way .... – LinPy Aug 11 '20 at 19:23
0

I think you need to look for "using your own base image", e.g. in the Azure docs here. For building the actual Docker image you have two options:

  1. Build on Azure build servers. Here you need to upload all required files together with your Dockerfile to the build environment. (Alternatively, you could consider making the requirements-azure.txt file available via HTTP, such that the build environment can fetch it from anywhere.)
  2. Build locally with your own Docker-installation and upload the final image to the correct Azure registry.

This is just the broad outline, at the moment I can't give more detailed recommendations. Hope it helps anyway.

Jan E.
  • 1
  • 1