0

Fluentd switch is sending two of the same log. Configuration file below. I installed on kubernetes with helm chart I couldn't solve the problem. How do I edit?

I installed EFK with helm chart.

helm install elasticsearch elastic/elasticsearch --version 7.17.3 -n efk
helm install kibana elastic/kibana --version 7.17.3
helm install fluentd fluent/fluentd --version 0.3.9 -n efk

I'm pretty sure it's the same log. sometimes shows more than one log for the same service

Log record enter image description here

fluentd configmaps file fluentd configmaps

#apiVersion: v1
data:
  01_sources.conf: |-
    <source>
      @type tail
      @id in_tail_container_logs
      @label @KUBERNETES
      path /var/log/containers/*authentication*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_type string
          time_format "%Y-%m-%dT%H:%M:%S.%NZ"
          keep_time_key false
        </pattern>
        <pattern>
          format regexp
          expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
          time_format '%Y-%m-%dT%H:%M:%S.%NZ'
          keep_time_key false
        </pattern>
      </parse>
      emit_unmatched_lines true
    </source>
  02_filters.conf: |-
    <label @KUBERNETES>
      <match kubernetes.var.log.containers.fluentd**>
        @type relabel
        @label @FLUENT_LOG
      </match>

      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        skip_labels false
        skip_container_metadata false
        skip_namespace_metadata true
        skip_master_url true
      </filter>

      <match **>
        @type relabel
        @label @DISPATCH
      </match>
    </label>
  03_dispatch.conf: |-
    <label @DISPATCH>
      <filter **>
        @type prometheus
        <metric>
          name fluentd_input_status_num_records_total
          type counter
          desc The total number of incoming records
          <labels>
            tag ${tag}
            hostname ${hostname}
          </labels>
        </metric>
      </filter>

      <match **>
        @type relabel
        @label @OUTPUT
      </match>
    </label>
  04_outputs.conf: |-
    <label @OUTPUT>
      <match **>
        @type elasticsearch
        host "elasticsearch-master"
        port 9200
        path ""
        user elastic
        password changeme
      </match>
    </label>
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: fluentd
    meta.helm.sh/release-namespace: efk
  labels:
    app.kubernetes.io/instance: fluentd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluentd
    app.kubernetes.io/version: v1.14.6
    helm.sh/chart: fluentd-0.3.9
  name: fluentd-config
  namespace: efk

How can I fix

zoroglur
  • 395
  • 2
  • 18
  • I don't see any issues in your configuration. Would it be possible that you have two copies of fluentd running? Is there any errors in fluentd logs? – Neutralizer Jun 20 '23 at 17:56
  • @Neutralizer There are 4 fluentd pods in my kubernetes server – zoroglur Jun 20 '23 at 18:15
  • Can you check the `_index` of each document? Are they the same? – Val Jun 21 '23 at 06:08
  • @Val I installed EFK with helm chart. I did not make changes on the indexes, by default, the fluentd index came – zoroglur Jun 21 '23 at 07:05
  • Since you have 4 pods, is there a risk that two pods tail the same files? – Val Jun 21 '23 at 07:10
  • @Val I can't say no. When installed by default, 4 pods were created for fluentd. 3 pods were created for elasticsearch. How and in which configmap can I check the situation you are talking about? – zoroglur Jun 21 '23 at 07:13

0 Answers0