Python sftp error with paramiko

Question

Why i get a connection dropped error with paramiko when i invoke get function ?

class AllowAnythingPolicy(paramiko.MissingHostKeyPolicy):
    def missing_host_key(self, client, hostname, key):
        return

client = paramiko.SSHClient()
client.set_missing_host_key_policy(AllowAnythingPolicy())
client.connect('', username='',password='')
sftp.get('','')

i have a file of 70 mb, the function download 20mb after i get an error. this function worke fine when the size file is under 20mb

this is paramiko log file :

DEB [20161115-10:25:47.792] thr=1   paramiko.transport: starting thread (client mode): 0x472a3d0
DEB [20161115-10:25:47.793] thr=1   paramiko.transport: Local version/idstring: SSH-2.0-paramiko_2.0.2
DEB [20161115-10:25:47.793] thr=1   paramiko.transport: Remote version/idstring: SSH-2.0-SshServer
INF [20161115-10:25:47.794] thr=1   paramiko.transport: Connected (version 2.0, client SshServer)
DEB [20161115-10:25:47.795] thr=1   paramiko.transport: kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa'] client encrypt:['aes256-ctr', 'aes256-cbc'] server encrypt:['aes256-ctr', 'aes256-cbc'] client mac:['hmac-sha1', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com'] server mac:['hmac-sha1', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com'] client compress:['none'] server compress:['none'] client lang:['en-US'] server lang:['en-US'] kex follows?False
DEB [20161115-10:25:47.795] thr=1   paramiko.transport: Kex agreed: diffie-hellman-group1-sha1
DEB [20161115-10:25:47.796] thr=1   paramiko.transport: Cipher agreed: aes256-ctr
DEB [20161115-10:25:47.796] thr=1   paramiko.transport: MAC agreed: hmac-sha2-256
DEB [20161115-10:25:47.796] thr=1   paramiko.transport: Compression agreed: none
DEB [20161115-10:25:48.054] thr=1   paramiko.transport: kex engine KexGroup1 specified hash_algo <built-in function openssl_sha1>
DEB [20161115-10:25:48.054] thr=1   paramiko.transport: Switch to new keys ...
DEB [20161115-10:25:48.057] thr=1   paramiko.transport: userauth is OK
INF [20161115-10:25:48.137] thr=1   paramiko.transport: Authentication (password) successful!
DEB [20161115-10:25:57.677] thr=2   paramiko.transport: [chan 0] Max packet in: 32768 bytes
DEB [20161115-10:25:57.680] thr=1   paramiko.transport: [chan 0] Max packet out: 32768 bytes
DEB [20161115-10:25:57.681] thr=1   paramiko.transport: Secsh channel 0 opened.
DEB [20161115-10:25:57.682] thr=1   paramiko.transport: [chan 0] Sesch channel 0 request ok
INF [20161115-10:25:57.684] thr=2   paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
DEB [20161115-10:25:57.685] thr=2   paramiko.transport.sftp: [chan 0] stat(b'/GEO/OUT')
DEB [20161115-10:25:57.688] thr=2   paramiko.transport.sftp: [chan 0] normalize(b'/GEO/OUT')
DEB [20161115-10:25:57.690] thr=2   paramiko.transport.sftp: [chan 0] listdir(b'/GEO/OUT/.')
DEB [20161115-10:26:02.008] thr=2   paramiko.transport.sftp: [chan 0] stat(b'/GEO/OUT/test.csv')
DEB [20161115-10:26:02.012] thr=2   paramiko.transport.sftp: [chan 0] open(b'/GEO/OUT/test.csv', 'rb')
DEB [20161115-10:26:02.016] thr=2   paramiko.transport.sftp: [chan 0] open(b'/GEO/OUT/test.csv', 'rb') -> b'2f47454f2f4f55542f746573742e637376'
DEB [20161115-10:28:10.626] thr=1   paramiko.transport: EOF in transport thread
DEB [20161115-10:28:10.626] thr=2   paramiko.transport.sftp: [chan 0] close(b'2f47454f2f4f55542f746573742e637376')

and Python error :

EOFError                                  Traceback (most recent call last)
C:\Anaconda3\lib\site-packages\paramiko\sftp_client.py in _read_response(self, waitfor)
    759             try:
--> 760                 t, data = self._read_packet()
    761             except EOFError as e:

C:\Anaconda3\lib\site-packages\paramiko\sftp.py in _read_packet(self)
    172     def _read_packet(self):
--> 173         x = self._read_all(4)
    174         # most sftp servers won't accept packets larger than about 32k, so

C:\Anaconda3\lib\site-packages\paramiko\sftp.py in _read_all(self, n)
    158             if len(x) == 0:
--> 159                 raise EOFError()
    160             out += x

EOFError: 

During handling of the above exception, another exception occurred:

SSHException                              Traceback (most recent call last)
<ipython-input-49-b52d34c6bd07> in <module>()
----> 1 sftp.get('/GEO/OUT/test.csv','ESRI_OUT/te.csv')

C:\Anaconda3\lib\site-packages\paramiko\sftp_client.py in get(self, remotepath, localpath, callback)
    719         """
    720         with open(localpath, 'wb') as fl:
--> 721             size = self.getfo(remotepath, fl, callback)
    722         s = os.stat(localpath)
    723         if s.st_size != size:

C:\Anaconda3\lib\site-packages\paramiko\sftp_client.py in getfo(self, remotepath, fl, callback)
    697             fr.prefetch(file_size)
    698             return self._transfer_with_callback(
--> 699                 reader=fr, writer=fl, file_size=file_size, callback=callback
    700             )
    701 

C:\Anaconda3\lib\site-packages\paramiko\sftp_client.py in _transfer_with_callback(self, reader, writer, file_size, callback)
    596         size = 0
    597         while True:
--> 598             data = reader.read(32768)
    599             writer.write(data)
    600             size += len(data)

C:\Anaconda3\lib\site-packages\paramiko\file.py in read(self, size)
    209                 read_size = max(self._bufsize, read_size)
    210             try:
--> 211                 new_data = self._read(read_size)
    212             except EOFError:
    213                 new_data = None

C:\Anaconda3\lib\site-packages\paramiko\sftp_file.py in _read(self, size)
    163         size = min(size, self.MAX_REQUEST_SIZE)
    164         if self._prefetching:
--> 165             data = self._read_prefetch(size)
    166             if data is not None:
    167                 return data

C:\Anaconda3\lib\site-packages\paramiko\sftp_file.py in _read_prefetch(self, size)
    143             if self._prefetch_done or self._closed:
    144                 break
--> 145             self.sftp._read_response()
    146             self._check_exception()
    147         if offset is None:

C:\Anaconda3\lib\site-packages\paramiko\sftp_client.py in _read_response(self, waitfor)
    760                 t, data = self._read_packet()
    761             except EOFError as e:
--> 762                 raise SSHException('Server connection dropped: %s' % str(e))
    763             msg = Message(data)
    764             num = msg.get_int()

SSHException: Server connection dropped:

Show us WinSCP and [paramiko log files](http://stackoverflow.com/q/27587716/850848). — Martin Prikryl, Nov 15 '16 at 09:08
when i use get function : EOFError: During handling of the above exception, another exception occurred: SSHException SSHException: Server connection dropped: — slimoo, Nov 15 '16 at 09:17
Log file please! And of course, include the full exception message to the question (there's "edit" link). — Martin Prikryl, Nov 15 '16 at 09:21

score 2 · Answer 1 · answered Nov 15 '16 at 10:25

2

the solution to my problem is :

tr = client.get_transport()
tr.default_max_packet_size = 100000000
tr.default_window_size = 100000000

answered Nov 15 '16 at 10:25

slimoo

123
2
14

Marius · Answer 2 · 2016-11-14T17:21:27.177

I am guessing you get resource errors? Feel free to update the Question with the actual error as wel..

If the error is related to resources i would try to use open instead of get, i would use open and read, like this:

class AllowAnythingPolicy(paramiko.MissingHostKeyPolicy):
    def missing_host_key(self, client, hostname, key):
        return

client = paramiko.SSHClient()
client.set_missing_host_key_policy(AllowAnythingPolicy())
client.connect('', username='',password='')
file = sftp.open('','')
file.read # Will get you the file.

This is due to the problem that SFTP can't really stream files.

Another possibility is to set the block size for transferring files, something like this:

remotefile = sftp.open('','')
with remotefile.file(remote_file_path, mode='w') as sftpfile:
    sftpfile.MAX_REQUEST_SIZE = 1024
    sftpfile.write(data)

Python sftp error with paramiko

2 Answers2