1

I have a Python script that is currently using multiprocessing to perform tasks in parallel. My advisor recommended me to use GNU Parallel to speed it up since Python programs always execute on a single core. Should I keep the multiprocessing script as it is and use GNU Parallel on top of it? Or should I remove the multiprocessing part and then use GNU Parallel? Does it make a difference?

raffayatiq
  • 159
  • 9
  • Check [this question](https://stackoverflow.com/questions/57993330/gnu-parallel-vs-multiprocessing) and see if it helps – mgmussi Apr 09 '22 at 11:25
  • multiprocessing as the name suggests uses multiple core already – Epsi95 Apr 09 '22 at 11:25
  • "*since Python programs always execute on a single core*" This is wrong. Each process is indeed sequential (assuming you do not use a module that is parallel which is rare) but multiprocessing creates multiple process. Parallelizing a parallel process generally results in a slower execution (sometime huge regarding the platform) as it massively cause process *pre-emptions*. It is a bit like trying to do a lot of things at the same time as a human. *Context switches* are expensive. You can check the CPU usage to see how much your program already use multiple cores. – Jérôme Richard Apr 09 '22 at 11:57

1 Answers1

0

Does it make a difference?

There is a really simple answer: Try it and measure.

The performance of parallelization these days depends on so many factors and it really depends on your application (and sometimes on your hardware).

Ole Tange
  • 31,768
  • 5
  • 86
  • 104
  • I have ran the script without GNU parallel and with GNU parallel. With GNU parallel, since I have 4 cores, I ran the script with 4 different inputs. The 4 scripts with GNU parallel complete altogether in about the same time as a single script without GNU parallel. So I think there is no right answer and you really do have to measure it and see. – raffayatiq Apr 10 '22 at 22:56