-2

I'm trying to get the value of the CPU consumption and the RAM consumption of my Python App on a Linux embedded environnement using Psutil and Memory_Profiler. The thing is I can't get it done, or more accurately I think I don't understand exactly what I'm getting as a result. I find Psutil quite unclear about the values returned by the methods so I don't know exactly what I should use in my case.

I've written this but I think this can't give me what I actually want:

f = open("mem_info.txt", "w") 
txt = "Memory RAM : " + str(process.memory_info().vms) + " bytes"
txt3 = "RAM details : " + str(virtual_memory())
txt2 = "Memory Percent : " + str(process.memory_percent(memtype="vms")) + " %"
txt4 = "CPU Time  : " + str(process.cpu_times()) + " s"
TXT = txt + "\n" + txt2 + "\n\n" + txt3 + "\n\n" + txt4 + "\n"
f.write(TXT)
f.close()

Output :

Memory RAM : 605085696 bytes
Memory Percent : 19.297597042513864 %

RAM details : svmem(total=3135549440, available=2531147776, percent=19.3, used=417918976, free=104349696, active=1263509504, inactive=1590898688, buffers=13201408, cached=2600079360, shared=17809408, slab=109674496)

CPU Time  : pcputimes(user=0.54, system=0.07, children_user=6.99, children_system=0.34, iowait=0.61) s

Could someone tell me if I should use an other method ? Or If I used the right one, what result should I look at ? Should I like sum the different results printed in CPU Time ? But if I do so it's way to long for being the time of CPU used... PLease Help me ! :(

Lalastro
  • 151
  • 2
  • 10

2 Answers2

1

https://psutil.readthedocs.io/en/latest/index.html?highlight=oneshot#psutil.Process.oneshot

Utility context manager which considerably speeds up the retrieval of multiple process information at the same time. Internally different process info (e.g. name(), ppid(), uids(), create_time(), …) may be fetched by using the same routine, but only one value is returned and the others are discarded. When using this context manager the internal routine is executed once (in the example below on name()) the value of interest is returned and the others are cached. The subsequent calls sharing the same internal routine will return the cached value. The cache is cleared when exiting the context manager block. The advice is to use this every time you retrieve more than one information about the process. If you’re lucky, you’ll get a hell of a speedup.

I ran this:

import psutil
import time

p = psutil.Process(pid=10500)
with p.oneshot():
    while p.status() == 'running':
        print(p.name())  # execute internal routine once collecting multiple info
        print(p.cpu_times())  # return cached value
        print(p.memory_info().vms)  # return cached value
        print(p.cpu_percent())  # return cached value
        print(p.create_time())  # return cached value
        print(p.ppid())  # return cached value
        print(p.status())  # return cached value
        time.sleep(1)

On a multi-threaded python.exe (I'm on Windows). It returns information about the process and seems to be accurate when comparing with the Resource Monitor. What is it that you don't understand about your output?

Also, regarding cpu_times():

Return system CPU times as a named tuple. Every attribute represents the seconds the CPU has spent in the given mode. The attributes availability varies depending on the platform:

user: time spent by normal processes executing in user mode; on Linux this also includes guest time
system: time spent by processes executing in kernel mode
idle: time spent doing nothing

To find a process id (in this case we're looking for a python script) use:

{p.pid: p.info for p in psutil.process_iter() if p.info['name'] == 'python.exe'}

user: time spent in user mode.
system: time spent in kernel mode.

So be sure to use a pid, and you will be looking for the user output, as you are running a user program.

This is my output for two iterations of monitoring a multi-threaded python.exe using the code from the first cell in this answer:

python.exe
pcputimes(user=8.546875, system=20.046875, children_user=0.0, children_system=0.0)
436965376
0.0
1642441782.2310572
15272
running
python.exe
pcputimes(user=8.546875, system=20.046875, children_user=0.0, children_system=0.0)
436965376
100.1
1642441782.2310572
15272
running
python.exe
pcputimes(user=8.546875, system=20.046875, children_user=0.0, children_system=0.0)
436965376
100.0
1642441782.2310572
15272
running

cpu_percent is > 100 because I have multiple cores. One thing I should note, is that user does not change in this loop. I'm running in a jupyter notebook cell and it only changes when I stop the cell and restart it. So it'll take some tinkering using the psutil lib.

shullaw
  • 96
  • 1
  • 5
  • Thank you for your answer ! About cpu_time, I would like to know what to look at exactly if I want to know the total amount of CPU time consumed by my app. I actually have no use for those separated values "user", "system" and so on. So I was wondering if there was an other way, or maybe if I should just calculate the sum of all the returned values, or if the total cpu time my app consumes is just one of the returned values ?... – Lalastro Jan 17 '22 at 10:31
  • Also even with the code you suggested I only get 0% returned by `cpu_percent` and I can't get it print something else. I tried different things for that already as setting `perpu` to `true` or to `false`, to call it once before printing a value as the first called is supposed to be 0 but no matter what it still returns zero. Do you get this output too ? – Lalastro Jan 17 '22 at 10:47
  • @Lalastro the first iteration always gives me 0, but after that I get a `cpu_percent`. Make sure you use a pid. – shullaw Jan 17 '22 at 17:45
1

You can use the top command also, to get the most accurate stats

Run " top " directly in your shell to get interactive view like windows task manager

OR for command line to save output in txt file

top -n 1 -b > top-output.txt

you can cron this by saving it as a shell script,

Fairly extensive documentation, included in almost all Linux distributions

https://man7.org/linux/man-pages/man1/top.1.html
a-y-u-1
  • 13
  • 5
  • Thank you for the advice ! I've wanted to do so too but I'm on an embedded device so I can only open one terminal for it. And I don't think I can use top while having my application running unfortunately. In the console if I run it, I won't be able to execute top while it's still running... That's why I was rather looking for some python library that would allow me either to run both my application and the library's method from the terminal in one line or to add some code directly in my app to profile it. – Lalastro Jan 17 '22 at 10:11
  • you can create a shell script out of this.. and set it on crontab, that way it can execute in the background every minute to a pattern you define in cron, with any intervention, and the same file you can use as monitoring of your environment. – a-y-u-1 Jan 18 '22 at 04:37