Loki not alerting Alertmanager

Question

I am new with Loki and have made an alert in Loki but I don't see any notification in the Alertmanager. Loki is working fine (collecting logs), Alertmanager also (getting alerts from other sources), but the logs from loki don't get pushed to alertmanager.

Loki config:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: etc/loki/rules
  rule_path: /etc/loki/
  alertmanager_url: http://171.11.3.160:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Docker-compose Loki:

 loki:
    image: grafana/loki:2.0.0
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki/etc/local-config.yaml:/etc/loki/local-config.yaml
      - ./loki/etc/rules/rules.yaml:/etc/loki/rules/rules.yaml
    command:
      -  '--config.file=/etc/loki/local-config.yaml'

Loki rules:

groups:
  - name: rate-alerting
    rules:
    - alert: HighLogRate
      expr: |
           count_over_time(({job="grafana"})[1m]) >=0
      for: 1m

Does anybody know what's the problem?

Answer 1

I got it working atlast .

Below is my ruler config

ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rulestorage
  rule_path: /etc/loki/rules
  alertmanager_url: http://alertmanager:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  enable_alertmanager_v2: true

Created below directories

• /etc/loki/rulestorage/fake
• /etc/loki/rules/fake
• Copied alert_rules.yaml under /etc/loki/rulestorage/fake
• Gave full permission for loki user under /etc/loki/rulestorage/fake

Boom

Answer 2

The config looks good, similar as mine. I would troubleshoot it with following steps:

1. Exec to docker container and check if the rules file is not empty cat /etc/loki/rules/rules.yaml

2. Check the logs of loki. When rules are loaded properly logs like this will pop up:

level=info ts=2021-05-06T11:18:33.355446729Z caller=module_service.go:58 msg=initialising module=ruler
level=info ts=2021-05-06T11:18:33.355538059Z caller=ruler.go:400 msg="ruler up and running"
level=info ts=2021-05-06T11:18:33.356584674Z caller=mapper.go:139 msg="updating rule file" file=/data/loki/loki-stack-alerting-rules.yaml

3. During runtime loki also logs info messages about your rule (I will show you the one I am running, but slightly shortened)(notice status=200 and non-empty bytes=...):

level=info 
ts=... 
caller=metrics.go:83 
org_id=... 
traceID=... 
latency=fast 
query="sum(rate({component=\"kube-apiserver\"} |~ \"stderr F E.*failed calling webhook \\\"webhook.openpolicyagent.org\\\". an error on the server.*has prevented the request from succeeding\"[1m])) > 1" 
query_type=metric 
range_type=instant 
length=0s 
step=0s 
duration=9.028961ms 
status=200 
throughput=40MB 
total_bytes=365kB

4. Then make sure you can access alertmanager http://171.11.3.160:9093 from loki container without any issues (there can be a networking problem or you have set up basic authentication, etc.).

5. If the rule you set up (which you can test from grafana explore window) will exceed the threshold you set for 1 minute the alert should show up in alertmanager. It will be most likely ungrouped as you didn't add any labels to it.