16

How do I write a query that outputs average memory usage for instances over the past 24 hours? The following query displays the current memory usage

100 * (1 - ((node_memory_MemFree + node_memory_Cached + node_memory_Buffers) / node_memory_MemTotal))

For CPU, I was able to use irate

100 * (1 - avg(irate(node_cpu[24h])) BY (instance))

How do I use irate and avg for memory?

pdna
  • 541
  • 1
  • 8
  • 17

4 Answers4

28

average memory usage for instances over the past 24 hours

You can use avg_over_time:

100 * (1 - ((avg_over_time(node_memory_MemFree[24h]) + avg_over_time(node_memory_Cached[24h]) + avg_over_time(node_memory_Buffers[24h])) / avg_over_time(node_memory_MemTotal[24h])))

For CPU, I was able to use irate

irate only looks at the last two samples, and that query is the inverse of how many modes you have and will be constant (it's always 0.1 on my kernel). You want

100 - (avg by (instance) (rate(node_cpu{job="node",mode="idle"}[5m])) * 100)

Note that this is a 5 minute moving average and you can change [5m] to whatever period of time you are looking for such as [24h].

Michael
  • 3
  • 3
brian-brazil
  • 31,678
  • 6
  • 93
  • 86
16

Update commands

CPU Usage :

(1 - avg(irate(node_cpu_seconds_total{mode="idle"}[10m])) by (instance)) * 100

Memory Usage :

100 * (1 - ((avg_over_time(node_memory_MemFree_bytes[10m]) + avg_over_time(node_memory_Cached_bytes[10m]) + avg_over_time(node_memory_Buffers_bytes[10m])) / avg_over_time(node_memory_MemTotal_bytes[10m])))
Xetiras
  • 201
  • 2
  • 5
  • Your Memory Usage query shows percentage of Memory Available, not percentage of Memory Usage. To get the "Memory Usage" you need to delete the `-1`. – krzk Feb 25 '23 at 12:14
8

Also, we can query with accounts;

CPU Usage for Windows

#Average
100 - (avg by (instance) (irate(windows_cpu_time_total{job=~"dev",mode="idle"}[2m])) * 100)
#Minimum
100 - (max by (instance) (irate(windows_cpu_time_total{job=~"dev",mode="idle"}[2m])) * 100)
#Maximum
100 - (min by (instance) (irate(windows_cpu_time_total{job=~"dev",mode="idle"}[2m])) * 100)

CPU Usage for Linux

#Average
100 - (avg by (instance) (irate(node_cpu_seconds_total{job="$job",mode="idle"}[5m])) * 100)
#Minimum
100 - (max by (instance) (irate(node_cpu_seconds_total{job="$job",mode="idle"}[5m])) * 100)
#Maximum
100 - (min by (instance) (irate(node_cpu_seconds_total{job="$job",mode="idle"}[5m])) * 100)

Memory Usage for Windows

#Average
100.0 - 100 * avg_over_time(windows_os_physical_memory_free_bytes{job=~"prod"}[30d]) / avg_over_time(windows_cs_physical_memory_bytes{job=~"prod"}[30d])
#Maximum
100.0 - 100 * min_over_time(windows_os_physical_memory_free_bytes{job=~"prod"}[30d]) / min_over_time(windows_cs_physical_memory_bytes{job=~"prod"}[30d])
#Minimum
100.0 - 100 * max_over_time(windows_os_physical_memory_free_bytes{job=~"prod"}[30d]) / max_over_time(windows_cs_physical_memory_bytes{job=~"prod"}[30d])

Memory Usage for Linux

#Avarege:
100 * (1 - ((avg_over_time(node_memory_MemFree_bytes{job=~"prod"}[30d]) + avg_over_time(node_memory_Cached_bytes{job=~"prod"}[30d]) + avg_over_time(node_memory_Buffers_bytes{job=~"prod"}[30d])) / avg_over_time(node_memory_MemTotal_bytes{job=~"prod"}[30d])))

#Maximum:
100 * (1 - ((min_over_time(node_memory_MemFree_bytes{job=~"prod"}[30d]) + min_over_time(node_memory_Cached_bytes{job=~"prod"}[30d]) + min_over_time(node_memory_Buffers_bytes{job=~"prod"}[30d])) / min_over_time(node_memory_MemTotal_bytes{job=~"prod"}[30d])))

#Minimum
100 * (1 - ((max_over_time(node_memory_MemFree_bytes{job=~"prod"}[30d]) + max_over_time(node_memory_Cached_bytes{job=~"prod"}[30d]) + max_over_time(node_memory_Buffers_bytes{job=~"prod"}[30d])) / max_over_time(node_memory_MemTotal_bytes{job=~"prod"}[30d])))

Note:- {job=~"prod"}, is account name. you can use your account names

jawad846
  • 683
  • 1
  • 9
  • 21
1

In newer versions you probably add _bytes in all queries, like the examples above

mancier
  • 11
  • 1