2

I am using Python's multiprocessing to do bulk downloads using FTP. However, when I try to open more than 5 ftp sessions, an EOFError is raised, meaning the host is disconnecting me for opening too many sessions.

The only solution I see is to open a single FTP object and pass it to the necessary methods. The problem is that because multiprocessing uses pickling to move objects around, and FTP objects can't be pickled, this is not possible. My question is thus whether it is possible to work around this by finding a way to pickle FTP objects?

My code is of the following form:

def get_file(name):
    #code here    

def worker(name_list, out_q):
    lst = []
    for name in name_list:
        lst.append(get_file(name))
    out_q.put(lst)

if __name__ == '__main__':

    #est ftp cnxn
    ftp = FTP('ftp.blah.blah', 'anonymous', 'meow')

    #multiprocessing code here

The get_file def needs access to the ftp connection, and if I put it outside of the if __name__ == '__main__' block, then a new ftp connection is created each time a process runs through the code.

aensm
  • 3,325
  • 9
  • 34
  • 44

2 Answers2

2

You might be able to work around the problem by creating a pickleable class that wraps the FTP objects. Essentially you bind the FTP constructor arguments in your wrapper class then once it's deserialized on the remote host, the FTP object is instantiated there.

Rakis
  • 7,779
  • 24
  • 25
2

I don't really understand why you would want to do that:

  • create a bunch of processes to download stuff in parallel
  • but only use one FTP object, in effect serializing the download

How exactly does this solve your problem?

But, instead of serializing the FTP object, create a process for FTP requests and devise a mini-language for communicating with that process - let your other processes send (easily pickleable) messsages of the form get src dst.

EDIT: Just checked the documentation for [ftplib][1]. Nowhere does it say it can handle multiple calls. Assume it doesn't!

So, I would do this:

  • create MAX_CONNECTIONS (e.g. 5) FTP worker processes that
  • contact a master process that has a queue of files to retrieve
  • worker processes retrieve an task from the queue, downloads the file and checks master for new stuff to do
  • repeat until the work is done
Daren Thomas
  • 67,947
  • 40
  • 154
  • 200
  • to answer your question: I'm assuming downloading in parallel will be much faster, even with only one FTP connection open. Is this not the case? – aensm Jul 11 '12 at 14:46
  • what then is the difference between starting all the downloads in the same thread? – Daren Thomas Jul 11 '12 at 14:49
  • If I were doing all the downloads in the same thread, then it would be much slower than breaking up the download list into chunks and having multiple processes concurrently download files. Are you implying that a single FTP session is only capable of one download process at a time? Because if that were the case then, yes, this would pretty much defeat the purpose. – aensm Jul 11 '12 at 14:57
  • 1
    @aensm, right. I don't think the session can handle multiple downloads at the same time. It might be able to (read the source), but I don't think so. In fact, I don't remember the FTP protocol itself being able to do this, but that is rather fuzzy in my mind. See my edit for how I would parallelize this. – Daren Thomas Jul 11 '12 at 15:01
  • I was hoping to open more than 5 processes at a time. Do you think it would be possible to allow ftplib to handle more calls? Also, could it be the case that it's the host that's refusing multiple connections rather than a problem with ftplib? – aensm Jul 11 '12 at 15:06
  • 1
    I want to add that you can only read one file in one session like Daren says. If you want simultaneous downloads, you have to open multiple sessions. The amount of sessions you can open might not be determined by the client, but by the server. So watch out with the amount of sessions you plan to use, because the server might reduce them one day. – W. Goeman Jul 11 '12 at 15:11