Python ftplib cannot use STOR in callback function following RETR

Question

Here is what I need to accomplish: - connect to FTP - get contents of test.txt - write new contents into test.txt right after getting the results

In the real case scenario I need to get previos modification time, stored in a txt file and then upload to FTP only those files which were modified after that time without checking every file specifically (there are thousands of them, that would be too long).

Here is where I'm stuck.

def continueTest(data, ftp):
    print(data, ftp)
    with open('test.txt', 'w+') as file:
        file.write('test')
    with open('test.txt', 'rb') as file:
        ftp.storbinary('STOR htdocs/test.txt', file)

def test():
    host_data=FTP_HOSTS['planz-norwegian']
    ftp = ftplib.FTP(host=host_data['server'],
                     user = host_data['username'],
                     passwd = host_data['password'])
    print('connected to ftp')
    ftp.retrbinary('RETR htdocs/test.txt', lambda data:continueTest(data, ftp))


if __name__=='__main__':
    test()

This outputs:

connected to ftp
b'test' <ftplib.FTP object at 0x0322FAB0>
Traceback (most recent call last):
  File "C:\Python33\Plan Z Editor SL\redistdb.py", line 111, in <module>
    test()
  File "C:\Python33\Plan Z Editor SL\redistdb.py", line 107, in test
    ftp.retrbinary('RETR htdocs/test.txt', lambda data:continueTest(data, ftp))
  File "C:\Python33\lib\ftplib.py", line 434, in retrbinary
    callback(data)
  File "C:\Python33\Plan Z Editor SL\redistdb.py", line 107, in <lambda>
    ftp.retrbinary('RETR htdocs/test.txt', lambda data:continueTest(data, ftp))
  File "C:\Python33\Plan Z Editor SL\redistdb.py", line 99, in continueTest
    ftp.storbinary('STOR htdocs/test.txt', file)
  File "C:\Python33\lib\ftplib.py", line 483, in storbinary
    with self.transfercmd(cmd, rest) as conn:
  File "C:\Python33\lib\ftplib.py", line 391, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "C:\Python33\lib\ftplib.py", line 351, in ntransfercmd
    host, port = self.makepasv()
  File "C:\Python33\lib\ftplib.py", line 329, in makepasv
    host, port = parse227(self.sendcmd('PASV'))
  File "C:\Python33\lib\ftplib.py", line 873, in parse227
    raise error_reply(resp)
ftplib.error_reply: 200 Type set to I.

If I don't use STOR in a callback, everything works fine, But then, how am I supposed to get data from RETR command? I know possible solutions, but I'm sure there must be a more elegant one: - use urllib.request instead of RETR (what if there's no HTTP on the server?) - reinitialize FTP connection in callback function (may be slower than expected because of waiting for the server to reconnect) - user ftp.set_pasv(False) (callback launches, but the script does not end and cannot use ftp.quit() or ftp.close())

score 2 · Answer 1 · answered Sep 01 '15 at 20:12

According to the documentation of retrbinary:

The callback function is called for each block of data received, with a single string argument giving the data block.

This suggests that the callback is called while the data connection to retrieve the file is still open and the STOR command is not yet completed. It is not possible with FTP to create a new data connection (in the same FTP session) while another is still active. Additionally it looks like ftplib gets confused and considers the response to TYPE I beeing the response for PASV:

File "C:\Python33\lib\ftplib.py", line 873, in parse227
    raise error_reply(resp)
ftplib.error_reply: 200 Type set to I.

What you should do instead is to call STOR only after the RETR is completed, i.e. let the callback store everything in the file but then open the file only after retrbinary returned.

But then, how am I supposed to get data from RETR command?

In your current callback you store the data inside a file and then you read the file. The callback should still store the data in the file but reading and calling STOR should be done outside the callback, right after retrbinary. You cannot RETR and STOR data in parallel.

Storing the received data in a file is nice, but isn't it possible to store it in a variable instead, at least a global one? It's merely a single date with time, which hardly justifies involving file system to server as a buffer. Thank you for your reply! — Megan Caithlyn, Sep 01 '15 at 22:52
@HelgaIliashenko: sure it is possible, but since you stored the stuff in a file I thought this is what you want. For example code see http://stackoverflow.com/questions/21783551/get-ftp-retrlines-to-print-a-files-content-to-a-string — Steffen Ullrich, Sep 02 '15 at 04:12

Python ftplib cannot use STOR in callback function following RETR

1 Answers1