0

I'm running a logstash pipeline using kafka input plugin with following configuration

kafka {
    bootstrap_servers => "xxxx"
        topics => ['topic1', 'topic2', 'topic3' ... , 'topic50']
        auto_offset_reset => 'latest'
        group_id => 'xyz'
        decorate_events => 'basic'
    }

I want a field with kafka @metadata value so I added a new field - kafka_topic using

mutate {
        add_field => { 'kafka_topic' => '%{[@metadata]}' }
    }

but kafka_topic field contains some values of filebeat

{"beat":"filebeat","type":"_doc","version":"7.16.0"}
{“beat":"filebeat","topic":"a","type":"doc","version":"6.8.3"}
{“beat":"filebeat","topic":"b","type":"doc","version":"6.8.3"}
{“kafka”:{“topic":"c","consumer_group":"x","partition":11,"offset":5403823,"key":null,"timestamp":1690370271634}}
{“kafka":{"topic":"d","consumer_group":"x","partition":11,"offset":5403824,"key":null,"timestamp":1690370294548}}

I'm unable to understand why it is giving filebeat metadata for some topics

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Jinal Kothari
  • 1
  • 2
  • Sounds like you have logstash picking up more than one config file, or two versions running at the same time – OneCricketeer Jul 26 '23 at 14:48
  • @OneCricketeer thanks for your reply, I checked in the instance and there is only 1 logstash config file and we use systemctl to run logstash so only 1 version is running. – Jinal Kothari Jul 27 '23 at 07:04
  • Okay, then where would `version":"7.16.0"` and `version":"6.8.3"` come from? Also, not sure why would filebeat have a "topic" field? – OneCricketeer Jul 27 '23 at 16:56
  • 1
    today I got access to raw kafka messages and found that few topics were sending "@metadata" field which was overwriting logstash "@metadata" field and caused issue. I've put mutate filter before json filter (that parses kafka message into json fields) and now logstash pipeline is working fine. – Jinal Kothari Jul 31 '23 at 10:11

0 Answers0