2

My goal is to concurrently fetch files from multiple servers including multiple files from the same server. I am given a dict of file paths to fetch.

I'm thinking of something like this: paramiko multi ssh but with SFTP.

I have a list of opened pysftp.Connection objects. I can reuse these connection handles, so at least if I need to interact with the same server, the initialization process is skipped. I've been using this without errors. When I use threading for invoking the function that fetches the files however, it stops working, because multiple threads are trying to take over the same SFTP connection which is not supported by pysftp.

Do you know any workaround for this?

Community
  • 1
  • 1
Rápli András
  • 3,869
  • 1
  • 35
  • 55
  • Have you tried multiprocessing, i.e. https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor-example – Klemen Tusar Feb 14 '17 at 15:27
  • I cannot use multiprocessing because this is a daemon which can't fork processes. This is a self-defense of the multiprocessing library to prevent forked processes becoming zombies. Also, I need the shared memory space in which the SFTP connection objects are in. I'm using Python 2.7 – Rápli András Feb 14 '17 at 15:27
  • You have multiple dictionaries with the same key? How can you be given {'1.1.1.1':..., '1.1.1.1'}? If you reorganize your data so that you have only one key per connection, with a list of file names, then it's simple - run your threading on each connection, and have each thread loop over getting each file in the list. – Scott Mermelstein Feb 14 '17 at 15:36
  • Okay. I was making up that example fast. I'll try that, sounds like a good idea – Rápli András Feb 14 '17 at 15:38
  • 1
    did you get it, Do you have your code – udit kanotra Jun 05 '19 at 12:06

0 Answers0