Create a new index in elasticsearch for each log file by date

Question

Currently
I have completed the above task by using one log file and passes data with logstash to one index in elasticsearch :

yellow open logstash-2016.10.19 5 1 1000807 0 364.8mb 364.8mb

What I actually want to do

If i have the following logs files which are named according to Year,Month and Date

MyLog-2016-10-16.log
MyLog-2016-10-17.log
MyLog-2016-10-18.log
MyLog-2016-11-05.log
MyLog-2016-11-02.log
MyLog-2016-11-03.log

I would like to tell logstash to read by Year,Month and Date and create the following indexes :

yellow open MyLog-2016-10-16.log
yellow open MyLog-2016-10-17.log
yellow open MyLog-2016-10-18.log
yellow open MyLog-2016-11-05.log
yellow open MyLog-2016-11-02.log
yellow open MyLog-2016-11-03.log

Please could I have some guidance as to how do i need to go about doing this ?

Thanks You

If you are using the file input, see http://stackoverflow.com/questions/22916200/logstash-how-to-add-file-name-as-a-field — baudsp, Oct 21 '16 at 12:18

score 0 · Accepted Answer · answered Oct 21 '16 at 12:05

0

It is also simple as that :

 output {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "MyLog-%{+YYYY-MM-DD}.log"
   }
 }

answered Oct 21 '16 at 12:05

Renaud Michotte

389
4
17

score 0 · Answer 2 · answered Oct 21 '16 at 16:26

If the lines in the file contain datetime info, you should be using the date{} filter to set @timestamp from that value. If you do this, you can use the output format that @Renaud provided, "MyLog-%{+YYYY.MM.dd}".

If the lines don't contain the datetime info, you can use the input's path for your index name, e.g. "%{path}". To get just the basename of the path:

mutate {
    gsub => [ "path", ".*/", "" ]
}

score 0 · Answer 3 · answered Dec 17 '16 at 16:01

wont this configuration in output section be sufficient for your purpose ??

output {
    elasticsearch {
        embedded => false
        host => localhost
        port => 9200
        protocol => http
        cluster => 'elasticsearch'
        index => "syslog-%{+YYYY.MM.dd}"
    }
}

Create a new index in elasticsearch for each log file by date

3 Answers3