5

Today the disk space used by Loki has filled up. Is there a dashboard or function in Loki to identify which PODSs generated the most logs?

Something besides loki-canary?

I have never used this tool and I don't know if it is right for me

TylerH
  • 20,799
  • 66
  • 75
  • 101
Janka
  • 1,908
  • 5
  • 20
  • 41

1 Answers1

3

You can analyse the logs generation with the following LogQL:

topk(10,count_over_time({filename=~".+"}[1h]))

Change the "{filename=~".+"}" stream selector and the "1h" log range as desired.

Aleksandr Dubinsky
  • 22,436
  • 15
  • 82
  • 99
Marcelo Ávila de Oliveira
  • 19,950
  • 3
  • 39
  • 50
  • A quick explanation about how to interpret those results would be appreciated. All I see if a few lines referencing the loki components. – Dunge Aug 09 '23 at 19:48
  • `{app_kubernetes_io_component="single-binary", app_kubernetes_io_instance="loki", app_kubernetes_io_name="loki", app_kubernetes_io_part_of="memberlist", cluster="loki", container="loki", controller_revision_hash="loki-988b7cd84", filename="/var/log/pods/monitoring_loki-0_2cdc4a00-961d-47d2-a0a3-6466ba1e0fc2/loki/0.log", job="monitoring/loki-single-binary", namespace="monitoring", pod="loki-0", statefulset_kubernetes_io_pod_name="loki-0", stream="stderr"}` – Dunge Aug 09 '23 at 19:48
  • Ok, by increasing 10 to an higher value I see my own pods below and not just loki itself. "single-binary" seems to be an aggregate of all others and to be ignored. I think "canary" is what taking all my volume space, so I'll disable that feature. – Dunge Aug 09 '23 at 21:15