The same environment is used for my pipeline steps and they are running fine but the Script Run Config fails every time with same error. We are using the SDK v1.
"Creating conda environment failed with exit code: 1"
Python version used- 3.8 added as dependency while making the environment.
Code sample: [enter image description here][1]
Output log: Streaming azureml-logs/60_control_log.txt
=========================================
[2023-02-08T22:03:43.622773] Using urllib.request Python 3.0 or later
Streaming log file azureml-logs/60_control_log.txt
Running: ['/bin/bash', '/tmp/azureml_runs/score_test/azureml-environment-setup/conda_env_checker.sh']
Starting the daemon thread to refresh tokens in background for process with pid = 12605
Materialized conda environment not found on target: /home/azureuser/.azureml/envs/azureml
[2023-02-08T22:03:43.744540] Logging experiment preparation status in history service.
Running: ['/bin/bash', '/tmp/azureml_runs/score_test/azureml-environment-setup/conda_env_builder.sh']
Running: ['conda', '--version']
conda 4.10.3
Creating conda environment...
Running: ['conda', 'env', 'create', '-p', '/home/azureuser/.azureml/envs/azureml', '-f', 'azureml-environment-setup/mutated_conda_dependencies.yml']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
==> WARNING: A newer version of conda exists. <==
current version: 4.10.3
latest version: 23.1.0
Please update conda by running
$ conda update -n base -c defaults conda
Downloading and Extracting Packages
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/test/run_test.sh. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/recipe/patches/0001-check-for-existence-of-type-before-setting.patch. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/recipe/conda_build_config.yaml. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/recipe/configure_includedir_option.patch. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/recipe/meta.yaml. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/recipe/meta.yaml.template. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/recipe/recipe_log.txt. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/recipe/build.sh. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/run_exports.json. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/index.json. Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename /anaconda/pkgs/libffi-3.3-he6710b0_2/info/repodata_record.json. Please remove this file manually (you may need to reboot to free file handles)
Show More...
TypeError('not all arguments converted during string formatting')
TypeError('not all arguments converted during string formatting')
Attempting to clean up partially built conda environment: /home/azureuser/.azureml/envs/azureml
Creating conda environment failed with exit code: 1
[2023-02-08T22:04:05.439184] Logging error in history service: Failed to run ['/bin/bash', '/tmp/azureml_runs/score_test_/azureml-environment-setup/conda_env_builder.sh']
Exit code 1
Details can be found in azureml-logs/60_control_log.txt log file.
---------------------------------------------------------------------------
ActivityFailedException Traceback (most recent call last)
<ipython-input-127-bc407f33209f> in <module>
2 score_run = score_exp.submit(score_config)
3 print(score_run)
----> 4 score_run.wait_for_completion(show_output=True)
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/run.py in wait_for_completion(self, show_output, wait_post_processing, raise_on_error)
810 file_handle=sys.stdout,
811 wait_post_processing=wait_post_processing,
--> 812 raise_on_error=raise_on_error)
813 return self.get_details()
814 except KeyboardInterrupt:
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/run.py in _stream_run_output(self, file_handle, wait_post_processing, raise_on_error)
1052 file_handle.write("\n")
1053 else:
-> 1054 raise ActivityFailedException(error_details=json.dumps(error, indent=4))
1055
1056 file_handle.write("\n")
ActivityFailedException: ActivityFailedException:
Message: Activity Failed:
{
"error": {
"code": "UserError",
"message": "Creating conda environment failed with exit code: 1",
"messageParameters": {},
"details": []
},
"time": "2023-02-08T22:04:05.33789Z"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Activity Failed:\n{\n \"error\": {\n \"code\": \"UserError\",\n \"message\": \"Creating conda environment failed with exit code: 1\",\n \"messageParameters\": {},\n \"details\": []\n },\n \"time\": \"2023-02-08T22:04:05.33789Z\"\n}"
}
}
[1]: https://i.stack.imgur.com/gZ40g.png
