0

I have a Kubernetes cluster which I am trying to get to run fluentd to send logs to Loggly for viewing. I am having an issue where I know my token is correct since it was working on another cluster before.

Below is my manifet.yaml file, I have no logs to share as the pod isn't logging anything.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-es-v1.20
  namespace: kube-system
  labels:
    k8s-app: fluentd-loggly
    kubernetes.io/cluster-service: "true"
    version: v1.20
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-loggly
  template:
    metadata:
      labels:
        k8s-app: fluentd-loggly
        kubernetes.io/cluster-service: "true"
        version: v1.20
    spec:
      containers:
      - name: fluentd-loggly
        image: garland/kubernetes-fluentd-loggly:1.0
        command:
          - '/bin/sh'
          - '-c'
          - '/usr/sbin/td-agent 2>&1 >> /var/log/fluentd.log'
        env:
          - name: LOGGLY_URL
            value: "https://logs-01.loggly.com/inputs/<token>/tag/<tag>"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Edit 1

Logs requested

2021-06-21 16:33:43 +0000 [warn]: pattern not match: "2021-06-21T16:33:43.548145693Z stdout F 2021-06-21 16:33:43 UTC | CORE | ERROR | (pkg/autodiscovery/config_poller.go:123 in collect) | Unable to collect configurations from provider docker: temporary failure in dockerutil, will retry later: try delay not elapsed yet"

This repeats over and over with nothing else in the logs.

joshk132
  • 1,011
  • 12
  • 37
  • Well, the pod isn't logging anything to its stdout/stderr because you have redirected them into `/var/log/fluentd.log`; you'll want to either remove that redirection long enough to debug this issue, or `kubectl exec $POD -- cat /var/log/fluentd.log` to see what it is trying to tell you – mdaniel Jun 21 '21 at 15:48
  • @mdaniel I think that command is only redirecting the fluentd logs not my application logs. – joshk132 Jun 21 '21 at 15:55
  • It still makes things hard to debug, you'll have to exec into the container and look at that file. – coderanger Jun 21 '21 at 15:59
  • @coderanger have added an edit with the single logline that is repeating in that file – joshk132 Jun 21 '21 at 16:35
  • It sure is suspicious that your image is named `fluentd`, your env-var is `LOGGLY`, and the only output from the container [is from DataDog's agent](https://github.com/DataDog/datadog-agent/blob/7.28.1/pkg/autodiscovery/config_poller.go#L123). Since your cited yaml only includes the one container, and no application code, we are troubleshooting with the information provided – mdaniel Jun 21 '21 at 21:46
  • @mdaniel Datadog isn't even installed on the cluster so I have zero clue how it would be picking something up from it. Also fluentd can be used to send logs to loggly, it is a cloud hosted log service so you have to use something to send your logs to them – joshk132 Jun 21 '21 at 23:33
  • Also relevant to your interests: the repo for that image [even advises](https://github.com/sekka1/kubernetes-fluentd-loggly/issues/9#issue-447303655) to use [the fluentd images](https://github.com/fluent/fluentd-kubernetes-daemonset#readme) instead of some rando image, so you may have better luck trying those, too – mdaniel Jun 22 '21 at 03:34

0 Answers0