Loadavg over possible value

Question

Due to my simplified knowledge loadavg on N-core CPU should show max value N.

I've maximized CPU usage to 100% (8 x Temrminal with: $ node --eval 'while(1){}'

Linux built-in loadavg differs from my expectations and my own implementation (details on image)

Question - is my CPU really fully used?:

1. On my 8-core CPU native linux loadavg shows value over 9 (expected max is 8)
2. During test my PC was still working smoothly (youtube video run smoothly).
3. Check details below: .. Thread(s) per core: 2 .. - it still tells nothing because I was able to achievie bigger loadavg than 16, even 20:
4. 

Does it means that during test I used only 50% of my CPU potential? That /proc/loadavg tells the truth and system monitor (UI tool) lied about CPU usage?

PC details:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:    20.04
Codename:   focal

$ cat /proc/version
Linux version 5.13.0-27-generic (buildd@lgw01-amd64-045) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #29~20.04.1-Ubuntu SMP Fri Jan 14 00:32:30 UTC 2022

$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           140
Model name:                      11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
Stepping:                        1
CPU MHz:                         1700.000
CPU max MHz:                     4700,0000
CPU min MHz:                     400,0000
BogoMIPS:                        3379.20
Virtualization:                  VT-x
L1d cache:                       192 KiB
L1i cache:                       128 KiB
L2 cache:                        5 MiB
L3 cache:                        12 MiB
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constan
                                 t_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sd
                                 bg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 i
                                 nvpcid_single cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rd
                                 t_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect 
                                 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bit
                                 alg avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear flush_l1d arch_capabilities

Answer 1

Does it means that during test I used only 50% of my CPU potential? That /proc/loadavg tells the truth and system monitor (UI tool) lied about CPU usage?

loadavg and os.cpus()[n].times provide different information, meaning you can not compute one from the other. The former measures the number of tasks in the queue to get the CPU, the latter measures the load of the CPU itself. Indeed, for a very busy system you should expect loadavg to show greater values than the number of CPUs available (or even Thread availables) since the system will be CPU-bounded and more tasks will be waiting in the queue. See this for good answers explaining loadavg.

Question - is my CPU really fully used?

Your CPU is fully used as long as os.cpus()[n].times.idle does not increase. They come from /proc/stat. That is what your GUI is probably using (something like this for example). Your PC runs smoothly because Linux makes a good job providing a responsive system. Important and urgent tasks are given higher priority so that you can still watch videos on youtube while launching 8 infinite loops. This means that your code was not the only application that kept your CPU busy during your tests.

Answer 2

@Fusho

1. Posts you mentioned told me nothing new at all:

See this for good answers about loadavg.

---

2. However thak you for your answer because it pushed me to investigate topic one more time - I found article that explains loadavg and helps with deep understanding of this topic: https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

---

3. My custom loadavg implementation is loadavg-windows for nodejs - now I can see that name cpu-loadavg fits it better.

---

4. My conclusion:

To monitor application performance and bottlenecks we should measure two different statistics:

• CPU load
• HD I/O load (I have not found any tools that allows you to measure "disk I/O load" from OS perspective. From OS perspective means time that OS awaits for hard drive I/O divided by total time passed).
• Optionally we should take care of network I/O but this topic should be part of other post.

Imo. the need of monitoring of other I/O-s (other than HD and Network) from perspective of Application development is to be discussed in future because it have no such big impact on performance as CPU load and mentioned I/O-s.