My task is to perform some action on data queried from a database and then store the output on an SFTP server.
The script performing all of this is executed on a VM running Windows Server 2016 Standard through task scheduler.
There are a total of 4 files which I should copy at the end of the script from some location on a shared drive (accessible from the VM that's running the script) to the SFTP server. These files have size:
- ~25 MB
- ~55 MB
- ~900 MB
- ~2 GB
Here's the final part of the script, where the copying process takes place:
hostname = 'ftp_server_name'
username = 'username'
password = 'password'
keypath = 'Sftp_Key'
known_hosts = 'known_hosts.txt'
import paramiko
mykey = paramiko.RSAKey.from_private_key_file(keypath, password)
ssh_client = paramiko.SSHClient()
ssh_client.load_host_keys(known_hosts)
ssh_client.connect(
hostname=hostname,
username=username,
allow_agent=True,
pkey=mykey,
disabled_algorithms={'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}
)
tr = ssh_client.get_transport()
tr.default_max_packet_size = 10000000000
tr.default_window_size = 10000000000
sftp_client = ssh_client.open_sftp()
sftp_client.put(
r'\\path\to\origin\on\shared\drive\file_1.csv',
'/root/folder/path/to/destination/file_1.csv'
)
sftp_client.put(
r'\\path\to\origin\on\shared\drive\file_2.csv',
'/root/folder/path/to/destination/file_2.csv'
)
sftp_client.put(
r'\\path\to\origin\on\shared\drive\file_3.csv',
'/root/folder/path/to/destination/file_3.csv'
)
sftp_client.put(
r'\\path\to\origin\on\shared\drive\file_4.csv',
'/root/folder/path/to/destination/file_4.csv'
)
print(sftp_client.listdir())
sftp_client.close()
ssh_client.close()
As you can see, I am
- spelling out entirely the destination file name (as indicated in the documentation and as suggested everywhere on SO)
- getting the transport and increasing the sizes (as suggested here)
It all goes well until the final one (file_4), at which I get the following error
Traceback (most recent call last):
File "H:\folder\scripts\test_sftp_copy.py", line 45, in <module>
ftp_client.put(
File "C:\Users\MeUser\Anaconda3\lib\site-packages\paramiko\sftp_client.py", line 759, in put
return self.putfo(fl, remotepath, file_size, callback, confirm)
File "C:\Users\MeUser\Anaconda3\lib\site-packages\paramiko\sftp_client.py", line 716, in putfo
size = self._transfer_with_callback(
File "C:\Users\MeUser\Anaconda3\lib\site-packages\paramiko\sftp_client.py", line 679, in _transfer_with_callback
writer.write(data)
File "C:\Users\MeUser\Anaconda3\lib\site-packages\paramiko\file.py", line 405, in write
self._write_all(data)
File "C:\Users\MeUser\Anaconda3\lib\site-packages\paramiko\file.py", line 522, in _write_all
count = self._write(data)
File "C:\Users\MeUser\Anaconda3\lib\site-packages\paramiko\sftp_file.py", line 208, in _write
t, msg = self.sftp._read_response(req)
File "C:\Users\A-AChiap\Anaconda3\lib\site-packages\paramiko\sftp_client.py", line 874, in _read_response
self._convert_status(msg)
File "C:\Users\A-AChiap\Anaconda3\lib\site-packages\paramiko\sftp_client.py", line 907, in _convert_status
raise IOError(text)
OSError: Failure.
The weird thing is that, once the code breaks after attempting to put the fourth file (raising the OSError: failure), I cannot put any other file in the server any more.
So, for instance, if I try copying again file_1.csv, I would get the OSError: failure error right away.
In this case, I need to wait several hours or try again the next day.
This feels like an issue on the SFTP side.
Can anybody with more experience or knowledge of the matter comment? Thank you
