What I observe is the following:
➜ ~ cat /proc/diskstats && sleep 1 && cat /proc/diskstats
7 0 loop0 7 0 18 1 0 0 0 0 0 4 1 0 0 0 0 0 0
7 1 loop1 115 0 7496 19 0 0 0 0 0 328 19 0 0 0 0 0 0
7 2 loop2 7 0 16 2 0 0 0 0 0 4 2 0 0 0 0 0 0
7 3 loop3 14 0 48 5 0 0 0 0 0 12 5 0 0 0 0 0 0
7 4 loop4 7 0 16 1 0 0 0 0 0 4 1 0 0 0 0 0 0
7 5 loop5 223 0 3898 47 0 0 0 0 0 216 47 0 0 0 0 0 0
7 6 loop6 3232 0 259098 180 0 0 0 0 0 10192 180 0 0 0 0 0 0
7 7 loop7 22 0 84 5 0 0 0 0 0 24 5 0 0 0 0 0 0
259 0 nvme0n1 1566443 40470 417347309 1963895 4223539 5309069 527859912 5056858 16 1608032 7020754 0 0 0 0 0 0
259 1 nvme0n1p1 1566393 40470 417344957 1963863 4223539 5309069 527859912 5056858 16 1608028 7020721 0 0 0 0 0 0
7 8 loop8 41 0 688 15 0 0 0 0 0 16 15 0 0 0 0 0 0
7 9 loop9 43 0 690 13 0 0 0 0 0 12 13 0 0 0 0 0 0
7 10 loop10 11 0 28 0 0 0 0 0 0 4 0 0 0 0 0 0 0
7 0 loop0 7 0 18 1 0 0 0 0 0 4 1 0 0 0 0 0 0
7 1 loop1 115 0 7496 19 0 0 0 0 0 328 19 0 0 0 0 0 0
7 2 loop2 7 0 16 2 0 0 0 0 0 4 2 0 0 0 0 0 0
7 3 loop3 14 0 48 5 0 0 0 0 0 12 5 0 0 0 0 0 0
7 4 loop4 7 0 16 1 0 0 0 0 0 4 1 0 0 0 0 0 0
7 5 loop5 223 0 3898 47 0 0 0 0 0 216 47 0 0 0 0 0 0
7 6 loop6 3232 0 259098 180 0 0 0 0 0 10192 180 0 0 0 0 0 0
7 7 loop7 22 0 84 5 0 0 0 0 0 24 5 0 0 0 0 0 0
259 0 nvme0n1 1567711 40470 417613045 1965083 4226968 5313406 528356288 5061487 2 1609036 7026571 0 0 0 0 0 0
259 1 nvme0n1p1 1567661 40470 417610693 1965051 4226968 5313406 528356288 5061487 2 1609032 7026539 0 0 0 0 0 0
7 8 loop8 41 0 688 15 0 0 0 0 0 16 15 0 0 0 0 0 0
7 9 loop9 43 0 690 13 0 0 0 0 0 12 13 0 0 0 0 0 0
7 10 loop10 11 0 28 0 0 0 0 0 0 4 0 0 0 0 0 0 0
➜ ~ iostat -x
Linux 5.15.0-1043-aws (ip-172-31-9-1) 09/01/23 _x86_64_ (16 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
18.37 0.01 5.15 11.39 0.00 65.08
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz aqu-sz %util
loop0 0.00 0.00 0.00 0.00 0.14 1.29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.05 1.59 0.00 0.00 0.17 32.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
loop10 0.00 0.01 0.00 0.00 0.00 1.27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.00 0.00 0.00 0.29 1.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.01 0.01 0.00 0.00 0.36 1.71 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 0.14 1.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop5 0.09 0.83 0.00 0.00 0.21 8.74 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
loop6 1.37 54.88 0.00 0.00 0.06 40.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.43
loop7 0.01 0.02 0.00 0.00 0.23 1.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop8 0.02 0.15 0.00 0.00 0.37 8.39 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop9 0.02 0.15 0.00 0.00 0.30 8.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 665.74 88551.00 17.15 2.51 1.25 133.01 1795.68 112170.48 2256.46 55.69 1.20 62.47 0.00 0.00 0.00 0.00 0.00 0.00 2.98 68.33
To summerize:
- According to /proc/diskstats, the read iops is 1567711-1566443=1268, the write iops is 4226968-4223539=3429
- According to iostat, read iops is 665 while write iops is 1795
- According to node_exporter, read iops and write iops is added up to 3.6k.
I found neither of those three metrics corresponds. I wonder why, and how can I get the actual iops of the nvme disk?
Meanwhile, I also want to know how can I tell if the disk has reach its "top iops"? I feel it has reached its top iops now, since the w_await takes about 1ms, which is quite long.
