Python3 ftplib not fetching larger binary files

Question

I have the following script to fetch files from a remote FTP site and download them locally:

for i in ftp.nlst():
  ftp.retrbinary("RETR " + i, open(local_file_path + "/" + i, "wb").write)

The debug information shows that the file is being fetched, and I can see it's being streamed to the local system. After some time the download will just halt and seems like it's paused...

get "150 Opening BINARY mode data connection for 'filename' (163479928 bytes).\n"
resp "150 Opening BINARY mode data connection for 'filename' (163479928 bytes)."

score 1 · Answer 1 · answered Feb 03 '15 at 19:36

1

Have you tried to experiment with maxblocksize parameter? For example:

ftp.retrbinary("RETR " + i, open(i, "wb").write, maxblocksize=1048576)

I'm not sure what caused your issue, but if your files are big it probably connected with buffer size on FTP server side.

answered Feb 03 '15 at 19:36

artemdevel

641
1
9
21

Thanks i'll give this a try. btw in Python 3 it's *blocksize* instead of *maxblocksize* ;) https://hg.python.org/cpython/file/3.4/Lib/ftplib.py#l426 – Brandon Davis Feb 03 '15 at 21:13
Sorry, I completely missed you asked about python 3 :) I tried to reproduce your issue on my test ftp server (I use pure-ftpd) and used python 3 but everything just worked.. – artemdevel Feb 04 '15 at 08:14

score 1 · Answer 2 · answered Feb 03 '15 at 19:39

1

If your are retrieving multiple files, I would list the names and make sure that each is closed properly.

for name in ftp.nlst():
  print(name)  # for debugging
  with  open(local_file_path + "/" + name, "wb") as f:
      ftp.retrbinary("RETR " + i, f.write)

answered Feb 03 '15 at 19:39

Terry Jan Reedy

18,414
3
40
52

I should of added that I have set the set_debuglevel(2), which will log every request sent to and from the server. I can see when the file is opened, but it never closes.. – Brandon Davis Feb 03 '15 at 21:24

score 1 · Answer 3 · answered Feb 04 '15 at 18:23

The bottom line for this issue was the internet connection (download speed 1.45MB) and how the ftplib handles connection drops. When using retrbinary and the connection is dropped it is never resumed. The ftplib needs to support block mode (RFC 959), so if the connection is dropped it can resume starting at the latest block received.

Python3 ftplib not fetching larger binary files

3 Answers3