0

I have a python script that runs programs in parallel using joblib and it works just fine (100% cpu consumption on local machine).

Lately, I've migrated the python script to data science virtual machine (DSVM) on azure but found that the multiprocessing logic does not work (cpu consumption remains low level 5% on average).

Does anyone know whether azure VM supports parallelization? if yes, what python package I should use? "joblib" is currently used.

if no, any alternatives you can suggest for parallelization on azure VM?

Thank you in advance and stay healthy!!!

Regards, Kelvin

Kelvin Ho
  • 61
  • 3

1 Answers1

0

What is your local computer OS (Windows, Linux, distro, version)? was the test run in real HW or virtual? what virtualization technology (Hyper-V, vmware, other)? The virtualization offered by Azure should match real HW so I would first look at the OS and dependencies in regards to parallelization, i.e. process or thread base, etc. I'm not pro on Python however think there should be a way for you to identify what is blocking one process or thread to start another, kind of bottleneck. Knowing this should lead in the right direction to resolve the issue. Cesar

Cesar Solis
  • 1