1

In Python, I'm trying to run 150-200 processes. I have these 150 things in an array, and I've split this array up into multiple arrays of 10 elements each.

Now, I run a Multiprocessing Map, with 10 elements at a time. Once all 10 are complete, we go onto the next 10, and so on.

Now, the problem: The ninth and tenth process are almost ALWAYS slower than the rest. Is there a reason for that? Am I not doing this the most efficient way?

** I won't be able to share the code for this. So do you have any ideas as to why this may be happening?

Thanks in advance.

TajyMany
  • 527
  • 7
  • 20
  • Do you happen to have 4 or 8 processor cores? – L3viathan Feb 05 '16 at 22:10
  • A Core i7 Processor with 4 cores. However, from what I've heard, it could use 4 cores as 8? Is that true? Thanks! – TajyMany Feb 05 '16 at 22:20
  • Exactly, the i7s do hyper-threading to have 8 virtual cores, so that *might* be the reason. – L3viathan Feb 05 '16 at 22:24
  • So, would there be a way to fix that? – TajyMany Feb 05 '16 at 22:25
  • 1
    Well yes, using 8 instead of 10 parallel processes. – L3viathan Feb 05 '16 at 22:27
  • Thank you! It seemed to work! – TajyMany Feb 05 '16 at 22:29
  • There's a new problem: I can run the script 3 times, and it works fine, but when I run it the fourth time, the same problem arises. Any ideas? – TajyMany Feb 05 '16 at 22:34
  • 1
    Are all your elements to be processed supposed to be processed in an equivalent duration ? Or maybe some elements require an harder computation ? (or to wait from from a remote answer, etc. ?). In addition to setting the number of process to 8, did you try other multiprocessing methods such as `imap_unordered` to see if you have the same issue ? – mgc Feb 05 '16 at 23:41
  • Or is it possible that you are starting to be short of RAM at the end of your run, causing it dumping on hard drive (and generating a delay ?) – mgc Feb 05 '16 at 23:49
  • I haven't yet tried the other methods, but I will, thanks, but no, it's not a RAM issue, I have 64 GB of RAM, and I have 55 GB available during the process. – TajyMany Feb 06 '16 at 00:25
  • Well, I tried out `imap_unordered`, and it worked better, I can run it ~10 times, smoothly, but then it gets VERY slow after that (in fact, it can take up to 100 seconds to complete all 150). But yes, it does wait for a response from IBM Watson servers, which usually return within a second or two. – TajyMany Feb 06 '16 at 00:33

1 Answers1

1

This is due to the way Pool.map distributes the data among the workers. Use chunksize=1 as a parameter, i.e. map(...,..., chunksize=1).

A similar problem is explained in: python multiprocessing map mishandling of last processes

Community
  • 1
  • 1
Luis
  • 63
  • 6