1

I am injecting a file from the s3 bucket to logstash, My file name is containing some information, I want to split the file name into multiple fields, so I can use them as separate fields. Please help me I am new with elk.

input {
 s3 {
    bucket => "***********"
    access_key_id => "***********"
    secret_access_key => "*******"
    "region" => "*********"
    
    "prefix" => "Logs"
    "interval" => "1"
    "additional_settings" => {
           "force_path_style" => true
           "follow_redirects" => false
           }
    }
}

filter {
  mutate {
    add_field => {
      "file" => "%{[@metadata][s3][key]}"              //This file name have to split
    }
   
  }
}

output {
 elasticsearch {
  hosts => ["localhost:9200"]
  index => "indexforlogstash"
     
 }
}
ramkrishna kushwaha
  • 380
  • 1
  • 6
  • 17
  • Can you provide a sample file name (even redacted), it's just to see what kind of pattern we're talking about – Val Jan 13 '21 at 08:23
  • @Val file Logs/1232131-custombuildv12-log.txt this is my file where Logs is prefix and remaining is the filename, numeric value represent some device id and custombuildv12 represent build id – ramkrishna kushwaha Jan 13 '21 at 09:04

1 Answers1

1

In the filter section you can leverage the dissect filter in order to achieve what you want:

filter {
    ...

    dissect {
      mapping => {
        "file" => "Logs/%{deviceId}-%{buildId}-log.txt"
      }
    }
}

After going through this filter, your document is going to get two new fields, namely:

  • deviceId (1232131)
  • buildId (custombuildv12)
Val
  • 207,596
  • 13
  • 358
  • 360