How do I get cpu_times of process run with psutil.Popen after it is finished but before it is terminated

Question

I would like to use the cpu_times() method of a psutil.Popen object to find the cumulative values after it has finished. I first tried the following:

p = psutil.Popen("stress --cpu 3 -v -t 15", shell=True)
p.wait()
cpuTimes = p.cpu_times()

However, this results in a NoSuchProcess exception because wait() doesn't return until the process is terminated. I next tried putting the call to cpu_times() before the call to wait():

p = psutil.Popen("stress --cpu 3 -v -t 15", shell=True)
cpuTimes = p.cpu_times()
p.wait()

However, this yields a response of all zeros. I presume this is because it gets called immediately after the process starts. So, I added a call to time.sleep() that would just outlast the process:

p = psutil.Popen("stress --cpu 3 -v -t 15", shell=True)
time.sleep(15.) # b/c -t 15 in subprocess means it will take 15 seconds
cpuTimes = p.cpu_times()
p.wait()

This actually yields the expected value of 45 seconds (3 CPUs at 100% utilization for 15 seconds).

For a general process, however, I will not know how long it will take to finish. I would like to avoid adding an arbitrarily large sleep time just to make sure the process has finished before I make my query.

Is there a way to know that the process is done without calling wait() or other such methods upon whose return the process has terminated?

tintedFrantic · Accepted Answer · 2019-03-26T14:40:59.553

Alright, I found a solution. It was actually pretty simple, so I probably should have thought of it before posting my question, but I guess that's the way these things sometimes go.

p = psutil.Popen("stress --cpu 3 -v -t 15", shell=True)
cpuTimes = p.cpu_times()
while True:
    time.sleep(1e-4)
    cpuTimes = p.cpu_times()  # Make sure to do this before the call to poll()
    retCode = p.poll()
    if retCode is not None:
        break

Here, I have added a loop after starting the subprocess that contains calls to the cpu_times() and poll() methods for the process.

The poll() method is a non-blocking way to check whether the subprocess has finished. By calling cpu_times() right before the call to poll(), a minuscule amount of time will have passed, so we can, for all intents and purposes, consider the last call to cpu_times() to yield an accurate result.

Note that we would get an exception when calling cpu_times() after poll() if the process has finished. This is another reason to put the call to cpu_times() before the call to poll().

The argument to the sleep() method is not super important but does influence how much time will elapse after the subprocess finishes and before the loop will terminate. You could probably also add some sort of generous timeout to guard against an infinite loop if the subprocess never finishes.

Thank you! This needs to pulled into psutil's docs. This is the best example for psutil.Popen — chaostheory, Feb 21 '20 at 18:32
Won't this cause a performance issue, polling 10,000 times per second? — Elliott B, Sep 01 '20 at 23:58
@ElliottB, you are probably correct. I imagine it could cause some non-negligible performance issue. As mentioned above, I believe the argument to `sleep()` will mostly serve to control how much time elapses after the process terminates. Thus, one could pick a larger value than I have, causing one to `poll()` less often if that is a concern. — tintedFrantic, Sep 10 '20 at 16:21

How do I get cpu_times of process run with psutil.Popen after it is finished but before it is terminated

1 Answers1