Download multiple files in different SFTP directories to local

Question

I have a scenario where we need to download certain image files in different directories in SFTP server to local.

Example : 
/IMAGES/folder1 has img11, img12, img13, img14
/IMAGES/folder2 has img21, img22, img23, img24
/IMAGES/folder3 has img31, img32, img33, img34
And I need to download img12, img23 and img34 from folder 1, 2 and 3 respectively

Right now I go inside each folder and get the images individually which takes an extraordinary amount of time(since there are 10,000s of images to download).

I have also found out that downloading a single file of the same size(as that of multiple image files) takes a fraction of the time.

My question is, is there a way to get these multiple files together instead of downloading them one after another ?

One approach I came up with was to copy all the files to a temp folder in sftp server and then download the directory but sftp does not allow 'copy', and I can not use 'rename' because then I will be moving the files to temp directory

tdelaney · Accepted Answer · 2022-07-14T16:10:52.477

You could use a process pool to open multiple sftp connections and download in parallel. For example,

from paramiko import SSHClient
from multiprocessing import Pool

def download_init(host):
    global client, sftp
    client = SSHClient()
    client.load_system_host_keys()
    client.connect(host)
    sftp = ssh_client.open_sftp()

def download_close(dummy):
    client.close()

def download_worker(params):
    local_path, remote_path = *params
    sftp.get(remote_path, local_path)

list_of_local_and_remote_files = [
    ["/client/files/folder1/img11", "/IMAGES/folder1/img11"],
]

def downloader(files):
    pool_size = 8
    pool = Pool(8, initializer=download_init, 
        initargs=["sftpserver.example.com"])
    result = pool.map(download_worker, files, chunksize=10)
    pool.map(download_close, range(pool_size))

if __name__ == "__main__":
    downloader(list_of_local_and_remote_files)

Its unfortunate that Pool doesn't have a finalizer to undo what was set in the initializer. Its not usually necessary - the exiting process is cleanup enough. In the example I just wrote a separate worker function that cleans things up. By having 1 work item per pool process, they each get 1 call.

Download multiple files in different SFTP directories to local

1 Answers1