3

I have loki + grafana + prometheus running in EKS using official helm charts.

I want to send alerts based on error log messages and for that i'm trying to use the loki ruler.

following is the ruler config

ruler:
storage:
  type: local
  local:
    directory: /rules
rule_path: /tmp
alertmanager_url: http://prometheus-alertmanager.prometheus.svc.cluster.local
ring:
  kvstore:
    store: inmemory
enable_api: true
enable_alertmanager_v2: true

and i have following alert rules/groups

alerting_groups:
 - name: testrule
interval: 60s
rules:
  - alert: SnowplowError
    expr: sum(count_over_time({namespace="snowplow",app="stream-collector"}|="INFO" [30m]))>0
    for: 1m
    labels:
      source: loki
      target: easystack
    annotations:
      summary: High request latency
      message: '{{ $labels.clustername }} alert: {{ $labels.message }}.'

the deployement goes successful while i do not get any email alerts. the logs from the pod show that query is being executed.

level=info ts=2022-05-04T18:07:18.823133872Z caller=metrics.go:92 org_id=fake latency=fast query="(sum(count_over_time({namespace=\"snowplow\", app=\"stream-collector\"} |= \"INFO\"[30m])) > 0)" query_type=metric range_type=instant length=0s step=0s duration=47.290222ms status=200 limit=0 returned_lines=0 throughput=0B total_bytes=0B

and i'm not getting any alerts or any error. it just running with no error and no alerts. i'm suspecting that the limit 0 in query might be the culprit but i dont know how to change that limit.

would be really glad if someone could help. been stuck here for a week now.

Leo
  • 695
  • 3
  • 11
Mujahid
  • 114
  • 1
  • 3
  • 14

2 Answers2

2

Try adding enable_alertmanager_v2: true to the Loki config

Jeremy Caney
  • 7,102
  • 69
  • 48
  • 77
kvs96
  • 21
  • 2
-1

Because loki pod will send you read only filesystem error

jacklong
  • 1
  • 2
    could you please elaborate more ? – Mujahid Aug 18 '22 at 16:39
  • 2
    As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Aug 26 '22 at 05:06