0

I am writing Python code for Windows and it connects to SFTP via paramiko. In the SFTP there are around 100k individual .txt files, 4kb in size each. The current python process will read each file from SFTP and write to local which are very slow.

Is there a way to compress those files for example using zip into a single file and then download and un-compress/extract in local drive?

Thank you

4 Leave Cover
  • 1,248
  • 12
  • 40
  • 83
  • Something like this? `c.exec_command("zip files.zip a.txt b.txt c.txt... and so on...")` but I have around 100k files is it possible to put those filenames in single string command? thanks – 4 Leave Cover Jun 10 '21 at 19:17
  • command line has restriction for line length so you can't put all filenames in command. Probably `wildcard` would have also problem because `bash` may try to replace `wildcard` with list of all matching filenames and again it will create too long line. If server has Python then you could upload Python script which would use `for`-loop to compress all files in zip. – furas Jun 11 '21 at 00:21
  • @furas Indeed, but I believe/hope you can pass the wildcards directly to `zip` (by escaping them), for `zip` to resolve them internally. Like `zip file.zip -i \*.txt`. + Alternatively, one can also feed Python commands to remote Python process to avoid uploading the script. – Martin Prikryl Jun 11 '21 at 05:06
  • @MartinPrikryl I don't know if zip can resolve it internally - I never check it. I use LInux and in Linux usually `bash` converts wildcard to list of files and programs don't have to do it. Feeding commands to remote Python is also some kind of uploading code :) But this uploading is hidden. – furas Jun 11 '21 at 05:21
  • The [`zip` man](https://linux.die.net/man/1/zip) page shows this example: `zip -r foo . -i dir/\*`. + Yes, indeed, it's hidden uploading, but it avoids creating temporary files. – Martin Prikryl Jun 11 '21 at 05:25

0 Answers0