I have a Python script that is currently using multiprocessing to perform tasks in parallel. My advisor recommended me to use GNU Parallel to speed it up since Python programs always execute on a single core. Should I keep the multiprocessing script as it is and use GNU Parallel on top of it? Or should I remove the multiprocessing part and then use GNU Parallel? Does it make a difference?
Asked
Active
Viewed 132 times
1
-
Check [this question](https://stackoverflow.com/questions/57993330/gnu-parallel-vs-multiprocessing) and see if it helps – mgmussi Apr 09 '22 at 11:25
-
multiprocessing as the name suggests uses multiple core already – Epsi95 Apr 09 '22 at 11:25
-
"*since Python programs always execute on a single core*" This is wrong. Each process is indeed sequential (assuming you do not use a module that is parallel which is rare) but multiprocessing creates multiple process. Parallelizing a parallel process generally results in a slower execution (sometime huge regarding the platform) as it massively cause process *pre-emptions*. It is a bit like trying to do a lot of things at the same time as a human. *Context switches* are expensive. You can check the CPU usage to see how much your program already use multiple cores. – Jérôme Richard Apr 09 '22 at 11:57
1 Answers
0
Does it make a difference?
There is a really simple answer: Try it and measure.
The performance of parallelization these days depends on so many factors and it really depends on your application (and sometimes on your hardware).

Ole Tange
- 31,768
- 5
- 86
- 104
-
I have ran the script without GNU parallel and with GNU parallel. With GNU parallel, since I have 4 cores, I ran the script with 4 different inputs. The 4 scripts with GNU parallel complete altogether in about the same time as a single script without GNU parallel. So I think there is no right answer and you really do have to measure it and see. – raffayatiq Apr 10 '22 at 22:56