0

I have a script that copies 50 or so files linked to by urls. The files average about 20 MB in size and I am using pycurl from within a custom file_copy function to copy them. I started several file_copy threads in order to increase speed, but with either 4 or 8 threads I only see about a 33% speed improvement versus using a single thread.

Are threaded pycurl functions the best choice for this use case?

Craig Burgler
  • 1,749
  • 10
  • 19
  • 1
    Threads in Python have their performance issues, but my biggest hunch here is that your operations are IO-bound anyway. – Makoto Oct 03 '16 at 14:58
  • @Makoto Yes, this was it. I was already near max on my bandwidth so there was not much room for improvement with threading. I was able to increase the bandwidth, and multiple threads (up to about 16) now make a significant performance improvement (~10X) versus a single-threaded approach. – Craig Burgler Oct 04 '16 at 01:59

0 Answers0