0

I have two files named access_log and http_access_2015-03-06_log and I want to set access and http_access_2015-03-06 parts of the file as the indices.

I read some answers for similar questions but I couldn't get how I can filter the file path using grok filter and use it as a reference for indexing.

Below is my configuration file:

input {
  file {
    path => ["G:/logstash-1.5.0/bin/tmp/*_log"]
    start_position => "beginning"

  }
}

filter {
  if [path] =~ "access" {
    mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  } else if [path] =~ "error" {
    mutate { replace => { type => "apache_error" } }
  } else {
    mutate { replace => { type => "random_logs" } }
  }
}

output {
  elasticsearch { 
     action => "index"
     host => localhost
     index => "test" 
  }
  stdout { codec => rubydebug }
}

How it can be done?

Val
  • 207,596
  • 13
  • 358
  • 360
Asma Zinneera Jabir
  • 801
  • 3
  • 13
  • 31
  • Possible duplicate of [Logstash: how to add file name as a field?](http://stackoverflow.com/questions/22916200/logstash-how-to-add-file-name-as-a-field) – Evaldas Buinauskas Nov 08 '15 at 10:39

1 Answers1

0

You're setting a 'type' field. If that's good enough, you can refer to it in your output{} stanza:

index => "%{type}-%{+YYYY.MM.dd}"

If you wanted to use some information from your filters that you didn't also want stored in elasticsearch, you can put it in a metadata field, e.g.:

[@metadata][myField]

and then refer to that in the same way.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • I do not prefer referring the type field. I need to extract the file name and set the index. How exactly can I use the file name in the index? – Asma Zinneera Jabir Nov 11 '15 at 14:07
  • Do you get a 'file' or 'path' field in your documents? If you only want part of it, you can run grok{} on it to create the value that you want to use. – Alain Collins Nov 11 '15 at 16:40