3

We're using FluentBit to ship microservice logs into ES and recently found an issue on one of the environments: some log entries are duplicated (up to several hundred times) while other entries are missing in ES/Kibana but can be found in the microservice's container (kubectl logs my-pod -c my-service).
Each duplicate log entry has a unique _id and _fluentBitTimestamp so it really looks like the problem is on FluentBit's side.

FluentBit version is 1.5.6, the configuration is:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Log_File     /fluent-bit/log/fluent-bit.log
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/parsers_java.conf

[INPUT]
    Name              tail
    Path              /home/xng/log/*.log
    Exclude_Path      /home/xng/log/*.zip
    Parser            json
    Buffer_Max_Size   128k

[FILTER]
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}

[OUTPUT]
    Name  es
    Match *
    Host es-logging-service
    Port 9210
    Type flink-logs
    Logstash_Format On
    Logstash_Prefix test-env-logstash
    Time_Key _fluentBitTimestamp

Any help would be much appreciated.

Yuri
  • 1,695
  • 1
  • 13
  • 23

2 Answers2

0

We had same problem Can you try in your configuration Write_operation upsert So if log has duplicate _id it will update instead of create Please note, Id_Key or Generate_ID is required in update, and upsert scenario.

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#write_operation

rasvi
  • 19
  • 6
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jun 30 '22 at 12:25
0

Continuing rasvi's answer, I was able to fix this by configuring the Generate_ID On option output on the configuration file. As describe on this doc In your case:

[OUTPUT]
    Name  es
    Match *
    Host es-logging-service
    Port 9210
    Type flink-logs
    Logstash_Format On
    Logstash_Prefix test-env-logstash
    Time_Key _fluentBitTimestamp
    Generate_ID On
licha
  • 31
  • 2