0

I have trouble running my a little bit complex python program in a remote directory, mounted by SSHFS. It takes a few seconds to perform imports when executing in a remote directory and a fraction of a second in a local directory. The program should not access anything in the remote directory on its own, especially in the import phase.

By default, there is current (remote) directory I sys.path, but when I remove it before (other) imports, speed does not change. I confirmed with python -vv that this remote directory is not accessed in the process of looking for modules. Still, I can see a stable flow of some data from the network with an external network monitor during the import phase.

Moreover, I can't really identify what exectly it is doing when consuming most time. It seems to happen after one import is finished, according to my simple printouts, and before a next import is started...

I'm running Fedora 25 Linux

HDJEMAI
  • 9,436
  • 46
  • 67
  • 93
Lech Wiktor Piotrowski
  • 99
  • 6
  • 1
    Yes, sshfs is slow. If you want to know exactly which OS-level operations are eating your time, I'd suggest using sysdig or a similar high-performance syscall-tracing tool. strace will also collect information about what syscalls are being made, but it's got so much overhead that you'll have a big performance hit just from using strace, so it's good for "what's hitting this filesystem?", but not so good for "what are my actual pain points in terms of performance?". (Though since you haven't specified your operating system, we don't actually know that it's one for which sysdig is available...) – Charles Duffy Mar 03 '17 at 04:10
  • (...but then, on many of the places where sysdig *isn't* available, you've got compelling competitors like dtrace that will also fill the niche). – Charles Duffy Mar 03 '17 at 04:12
  • Thanks, I'll try to look with the tool. In general sshfs seems for me to be as fast as NFS, etc. when I am doing some very demanding read/write operations with a C program in the same directory. While just initializing python script is dead slow. That made me wondering... – Lech Wiktor Piotrowski Mar 03 '17 at 07:18

1 Answers1

-1

In my case it were the Cern ROOT libraries import. When importing, they look in the current directory, no matter what I do. So the solution is to

  1. store the current directory
  2. cd to some really local directory, like "/" or "/home" before imports
  3. come back to the stored directory after imports
Umair
  • 6,366
  • 15
  • 42
  • 50
Lech Wiktor Piotrowski
  • 99
  • 6