loki, promtail pipeline stage issue

Question

Recently I have been testin LOKI - PROMTAIL - GRAFANA for log aggregation and analysis. I'm trying to parse my logs. A sample log looks like this [2023-04-05T00:04:18.105620+00:00] INFO: {"Code":"BGCRIF902","Message":"The backup started","time":1680653058,"Level":"info","businessId":"3896176627628900352","domainId":12421,"userEmail":"peter@abc.com","userId":"3896176627628900352","resourceId":"23a34","instanceId":"3b3b","processId":1907533,"traceid":"50462d9b-c639-4697-8702-d6f4f89919a7","channelName":"Backup"}

While trying to parse it using Transform option in GRAFANA, it's not reading the timestamp properly. I searched online for this and found we can use PIPELINE STAGES in promtail to manage this.

And my promtail config looks like this.

  GNU nano 4.8                                                                         /opt/promtail/promtail-local-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki.com:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: UB
      __path__: /home/ubuntu/lustrefsx/centralizedlogs/log/UnifiedBackup_Staging/*
    pipeline_stages:
    - match:
        selector: '{job="UB"}'
        stages:
        - regex:
            expression: "^((?P<timestamp>:(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}(?:\.\d+)?))(Z|[\+-]\d{2}:\d{2})?)\\s\\-\\s(?P<logMessage>.*)$"
        - labels:
            logMessage:
        - timestamp:
            format: RFC3339Nano
            source: timestamp

I think my regex expression is wrong, the config is failing and promtail stops because of this.

Can someone help me here. Thanks in advance.

score 0 · Answer 1 · answered Apr 05 '23 at 12:20

Regex in your configuration is indeed wrong for example of log provided.

Try:

^\[(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})?)\]\s*(?P<logMessage>.*)$

Here group timestamp matches the whole content of square brackets, and logMessage - everything after ] .

Demo of matching can be seen here.

In your regex for some reason timezone was not a part of group timestamp. So be sure to check if promtail (or loki) accepts timestamps with timezone, and if not, change regex to the following:

^\[(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?)(?:Z|[+-]\d{2}:\d{2})?\]\s*(?P<logMessage>.*)$

loki, promtail pipeline stage issue

1 Answers1