1

I am trying to work with file data in Azure ML prompt flow and resorted to this article:

Access data from Azure cloud storage during interactive development

It said to run this command in my runtime compute instance to install azureml.fsspec and mltable: pip install -U azureml-fsspec mltable

After everything was installed, I kept getting this error whenever I ran the python prompt flow step that had fsspec

===============================

I actually had to install fsspec and mltable with specific versions: fsspec 1.0.0 and mltable 1.3.0. This is because the updated versions doesn't work with the azureml-dataprep library together. This is what happened with the updated versions:

ERROR: azureml-dataset-runtime 1.51.0 has requirement azureml-dataprep<4.11.0a,>=4.10.0a, but you'll have azureml-dataprep 4.11.1 which is incompatible.

Or these two errors:

ERROR: mltable 1.4.1 has requirement azureml-dataprep[parquet]<4.12.0a,>=4.11.1, but you'll have azureml-dataprep 4.10.9 which is incompatible.
ERROR: azureml-fsspec 1.1.1 has requirement azureml-dataprep<4.12.0a,>=4.11.1, but you'll have azureml-dataprep 4.10.9 which is incompatible.

So apparently dataprep version has to be <4.11 for dataset-runtime but 4.11.1 for fsspec and mltable.

=============================

So I have fixed that issue by using older versions of azureml-fsspec and mltable but I still get the same error saying that there's no fsspec module whenever I run the python prompt flow step.

Has anyone ever run into this issue or know how to solve it?

Arnab Biswas
  • 4,495
  • 3
  • 42
  • 60
Keenan Grant
  • 13
  • 3
  • Can you provide more details such as: what specific task you're trying to do? Are you running the code in azureml notebook? And other relevant details which can help in reproduce the issue. – RishabhM Jul 19 '23 at 08:10
  • @RishabhM I am currently having compute instance issues so I can't show the python step I was trying to run. But I was trying to run this line of code inside of the prompt flow python step: `from azureml.fsspec import AzureMachineLearningFileSystem` But kept getting the "No module named azureml.fsspec" error. Even though I installed it in the compute instance terminal. – Keenan Grant Jul 19 '23 at 17:55
  • Welcome to AzureML! I have faced several bugs while I was trying to use Azure ML SDK v1. For PromptFlow, I will wait for it to come out of preview mode. – Arnab Biswas Aug 29 '23 at 05:20

1 Answers1

0

Based on the provided information, I have reproduced the issue.

enter image description here

When we are creating runtime for Prompt flow it will assign a default runtime environment which doesn't contain the required package.

To solve this, you can install the packages using subprocess.

import subprocess
package_name = "azureml-fsspec"
subprocess.check_call(["python", "-m", "pip", "install", package_name])

enter image description here

With this I was able to resolve the issue.

enter image description here

Another possible solution is to use custom environment with required packages. For more details, please refer to this documentation.

RishabhM
  • 525
  • 1
  • 5