Kubernetes promtail sidecar: how to get labels from the parent pod metadata

Question

I have some kubernetes applications that log to files rather than stdout/stderr, and I collect them with Promtail sidecars. But since the sidecars execute with "localhost" target, I don't have a kubernetes_sd_config that will apply pod metadata to labels for me. So I'm stuck statically declaring my labels.

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: promtail
  name: sidecar-promtail
data:
  config.yml: |
    client:
      url: http://loki.loki.svc.cluster.local:3100/loki/api/v1/push
      backoff_config:
        max_period: 5m
        max_retries: 10
        min_period: 500ms
      batchsize: 1048576
      batchwait: 1s
      external_labels: {}
      timeout: 10s
    positions:
      filename: /tmp/promtail-positions.yaml
    server:
      http_listen_port: 3101
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: sidecar-logs
      static_configs:
        - targets:
          - localhost
          labels:
            job: sidecar-logs
            __path__: "/sidecar-logs/*.log"

----
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-logger
spec:
  selector:
    matchLabels:
      run: test-logger
  template:
    metadata:
      labels:
        run: test-logger
    spec:
      volumes:
        - name: nfs
          persistentVolumeClaim:
            claimName: nfs-claim
        - name: promtail-config
          configMap:
            name: sidecar-promtail
      containers:
        - name: sidecar-promtail
          image: grafana/promtail:2.1.0
          volumeMounts:
            - name: nfs
              mountPath: /sidecar-logs
            - mountPath: /etc/promtail
              name: promtail-config
        - name: simple-logger         
          image: foo/simple-logger
          volumeMounts:
            - name: nfs
              mountPath: /logs

What is the best way to label the collected logs based on the parent pod's metadata?

score 1 · Answer 1 · answered Jan 06 '22 at 19:55

You can do the following:

In the sidecar container, expose the pod name, node name and other information you need as environment variables, then add the flag '-config.expand-env' to enable environment expansion inside promtail config file, e.g.:

...
- name: sidecar-promtail
  image: grafana/promtail:2.1.0
  # image: grafana/promtail:2.4.1 # use this one if environment expansion is not available in 2.1.0
  args:
    # Enable environment expansion in promtail config file
    - '-config.expand-env'
  env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
...

Then in your configMap, add the environment variables in your static_config labels as such:

...
    scrape_configs:
    - job_name: sidecar-logs
      static_configs:
        - targets:
          - localhost
          labels:
            job: sidecar-logs
            pod: ${POD_NAME}
            node_name: ${NODE_NAME}
            __path__: "/sidecar-logs/*.log"
...

Kubernetes promtail sidecar: how to get labels from the parent pod metadata

1 Answers1