Fluent Bit cannot reach DNS for Azure Log Analytics

Question

I'm running Fluent Bit on Azure Kubernetes Service as DaemonSet, output goes to Azure Log Analytics. Once Fluent Bit tries to send logs it fails to connect DNS server:

getaddrinfo(host='XXX.ods.opinsights.azure.com', err=12): Timeout while contacting DNS servers

If I manually enter IP and XXX.ods.opinsights.azure.com into /etc/hosts, everything goes well.

If I ssh to fluent-bit pod and run wget XXX.ods.opinsights.azure.com host resolving works as well.

What can be wrong?

My configs:

ConfigMap

  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: fluent-bit-config
      namespace: airflow
      labels:
        k8s-app: fluent-bit
    data:
      fluent-bit.conf: |
        [SERVICE]
            Flush         1
            Log_Level     info
            Daemon        off
            HTTP_Server   On
            HTTP_Listen   0.0.0.0
            HTTP_Port     2020
        @INCLUDE input-kubernetes.conf
        @INCLUDE filter-kubernetes.conf
        @INCLUDE output-azure.conf
      input-kubernetes.conf: |
        [INPUT]
            Name              tail
            Tag               kube.*
            Path              /var/log/containers/*.log
            DB                /var/log/flb_kube.db
            Mem_Buf_Limit     5MB
            Skip_Long_Lines   On
            Refresh_Interval  10
      filter-kubernetes.conf: |
        [FILTER]
            Name                kubernetes
            Match               kube.*
            Kube_URL            https://kubernetes.default.svc:443
            Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
            Merge_Log           On
            K8S-Logging.Parser  Off
            K8S-Logging.Exclude Off
      output-azure.conf: |
        [OUTPUT]
            Name        azure
            Match       *
            Customer_ID xxx
            Shared_Key  yyy

DaemonSet

  - apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: fluentbit
      namespace: airflow
      labels:
        app.kubernetes.io/name: fluentbit
    spec:
      selector:
        matchLabels:
          name: fluentbit
      template:
        metadata:
          labels:
            name: fluentbit
        spec:
          serviceAccountName: fluent-bit
          containers:
            - name: fluent-bit
              imagePullPolicy: Always
              image: fluent/fluent-bit:1.8-debug
              volumeMounts:
              - name: varlog
                mountPath: /var/log
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
                readOnly: true
              - name: fluent-bit-config
                mountPath: /fluent-bit/etc/
              resources:
                limits:
                  memory: 1500Mi
                requests:
                  cpu: 500m
                  memory: 500Mi
          hostNetwork: true
          dnsPolicy: ClusterFirstWithHostNet
          volumes:
          - name: varlog
            hostPath:
              path: /var/log
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: fluent-bit-config
            configMap:
              name: fluent-bit-config

Answer 1

I ran against a similar issue today, downgrading the version of fluent bit to 1.8.4 resolved the issue. (https://github.com/fluent/fluent-bit/issues/4050)

In your case you would need to fix the docker container label to "1.8.4-debug" - hope this helps!

Answer 2

Can confirm findings above. same result with 1.8.5. had to go back to 1.8.4.