5

I have a simple program to run 8 processes, It remarkably reduces the script running time by using multiprocessing, however, I am not sure how many processes should I put to maximum my CPU utilization. Currently my cpu is 6 cores with only 1 physical cpu as it is a VPS. :

def spider1():
def spider2():
def spider3():
def spider4():
def spider5():
def spider6():
def spider7():
def spider8():
if __name__ == '__main__':
    p1 = multiprocessing.Process(target=spider1,)
    p2 = multiprocessing.Process(target=spider2,)
    p3 = multiprocessing.Process(target=spider3,)
    p4 = multiprocessing.Process(target=spider4, )
    p5 = multiprocessing.Process(target=spider5, )
    p6 = multiprocessing.Process(target=spider6, )
    p7 = multiprocessing.Process(target=spider7, )
    p8 = multiprocessing.Process(target=spider8, )
    p1.start()
    p2.start()
    p3.start()
    p4.start()
    p5.start()
    p6.start()
    p7.start()
    p8.start()
jonrsharpe
  • 115,751
  • 26
  • 228
  • 437
Cook
  • 333
  • 1
  • 3
  • 8
  • you could use https://pypi.org/project/psutil/ to get the cpu utilization and increase or reduce the number of Processes – Radan Sep 13 '18 at 10:08
  • Thanks Radan, I will have a try and adjust the processes number. However, I heard the limitation depends on the CPU core numbers. For example, if I have 6 core, then the best processes should be 6, if exceeds 6, it will not increase the CPU utilization efficiency ... not sure if it is true – Cook Sep 13 '18 at 10:14
  • not sure if it is true, but modern CPU's are really good in multi threading, Also there is no guarantee that they would run only python processes at a given time, because there will always be other background process on the machine. – Radan Sep 13 '18 at 10:20
  • 1
    Have you heard of loops? They are a nice way to write repeptitive code more concisely. – The function name "spider" sounds like you are scraping web pages. This will most likely be limited by your internet connection, not your CPU, so you should increase the number of processes until you saturate your connection. – Sven Marnach Sep 13 '18 at 10:24
  • @Cook this will give you some information https://stackoverflow.com/questions/42103367/limit-total-cpu-usage-in-python-multiprocessing – Radan Sep 13 '18 at 10:25
  • @Radan Make the difference between processes and threads. Each python process can run one thread at any given moment in time due to the Global Interpreter Lock (GIL). For pure pythonic operation, you can use multiprocessing to do parallel computing. If some of the computation is switched to C for instance (numpy, OpenCV), then multithreading becomes interesting. – Mathieu Sep 13 '18 at 10:27
  • 1
    I would say it depends on the load on each process. If each process has 10 minutes computation to perform, then it is best to spawn N or N-1 process, with N being the number of physical cores on your computer. (N-1 to avoid switching back and forth between python computation and OS background tasks). If each process do not have a lot of computation to perform, and the CPU load is low, then it is best to spawn a lot of them. Try to estimate that load. – Mathieu Sep 13 '18 at 10:28
  • @Mathieu The GIL is a non-issue if the code is not CPU bound. The vast majority of the use cases for threads are for UI programming and IO, and for these cases threads in Python work just fine. Since the OP seems to be writing a web crawler, the GIL is irrelevant here as well. – Sven Marnach Sep 13 '18 at 10:34
  • As people are saying, it depends what the bottleneck is. If you are spending most of the time at very low CPU usage, like waiting for a page to download, then look at running multiple threads on a single core. If the script speed is bound by the speed of the CPU and you are doing millions of calculations per second, then go for multiprocessing. – Peter Sep 13 '18 at 10:38
  • @Mathieu sorry, all i was saying was processors would be multi-threading, sorry if it was bit confusing. I do think N-1 like you suggested could be a good idea, but i am not fully convinced that the schedulers would just allocate one core for each python process. may be you are not suggesting that. – Radan Sep 13 '18 at 10:39
  • @Radan I'm using this on heavy computation application. Each iteration needs several minutes. I'm using N process, and the OS scheduler is indeed allocating 1 core / process, with sometimes a bit of switch towards the background, but negligible in my case. As Sven says, it depends on the application. – Mathieu Sep 13 '18 at 11:24

2 Answers2

8

If you want to use the number of cpu's to calculate number of process to spawn, use cpu_count to find the number of cpu's,

psutil.cpu_count()

But using the CPU utilization to calculate the number of spawned processes could be a better approach, to check the CPU utilization, you could do something like, 

import psutil
psutil.cpu_times_percent(interval=1, percpu=False)

this will give you the cpu usage and for example you could use that information to decide if you want to spawn a new process or not. It might be a good idea to keep an eye on memory and swap too.

I think this answer might be useful to look at, Limit total CPU usage in python multiprocessing

Radan
  • 1,630
  • 5
  • 25
  • 38
1

For a recommendation you have to give much more information about your use case. Multi-processing and the associated communication primitives like queues introduce overhead. Additionally, reasoning about such an issue using a VPS introduces many variables that might heavily skew experimental results.

  1. Learn about concurrency and parallelism if you haven't already.
  2. Generally: IO is a slow operation and the variable dominating that decision.
  3. I would use this really low res rule of thumb: Go with the number of cores N and multiply by a factor starting with 1.0 that increases with independent IO load and decreases asymptotically to 1/N with dependent IO load of your tasks.

This means that, if for example your parallel tasks fight over one limited resource, like a spinning harddisk, decrease parallelism (lockout cost) and concurrency (task switching cost by seektime) down to one. No IO leaves you with the number of cores that you can then use on full burn. With IO that is independent this rule would lead you to increase the number of tasks running in parallel, so the CPU cores can switch to another task when one runs into an IO operation.

AndreasT
  • 9,417
  • 11
  • 46
  • 60