please can anyone tell me can i set up alerts with Loki? alert should trigger to slack. my loki installation done through helm. also grafana & prometheus deployed through helm. i have dashboard to show error message and their count so , if an error message occurs it should send to slack.code is below.
loki:
enabled: true
isDefault: true
url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
ruler:
enable_api: true
enable_alertmanager_v2: true
enable_sharding: true
wal:
dir: /var/loki/wal
alert_rules:
- alert: AppError
expr: |
{namespace="health-help", pod="stgidentity-6fc88f9fb-6l5v7"} |= `request completed`
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.errorMessage }} errors from {{ $labels.namespace }}/{{ $labels.app }}"
description: "{{ $value }} errors have occurred in the last 5 minutes."
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
livenessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
datasource:
jsonData: {}
uid: ""
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: loki-s3
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: loki-s3
key: AWS_SECRET_ACCESS_KEY
config:
query_scheduler:
max_outstanding_requests_per_tenant: 2048
query_range:
parallelise_shardable_queries: false
split_queries_by_interval: 0
storage_config:
aws:
s3: s3://us-east-1/****
s3forcepathstyle: true
bucketnames: hh-stg-app-logs
region: us-east-1
insecure: false
sse_encryption: false
boltdb_shipper:
shared_store: s3
cache_ttl: 24h
here is my dashboard which i build. loki-dashboard for logs count
i need this dashboard result of logs count should send alert to slack