1

I am using the ELK stack for analyzing logs. So as per default configuration a new index by "logsatash-YYYY-MM-DD" is created by ES. So if I have configured logstash to read like this:

/var/log/rsyslog/**/2014-12-0[1-7]/auditd.log

So it is reading old logs and the index name created will be "logstash-2015-03-20", so this index will have documents (logs) of previous dates.

My problem occurs when I have to delete indexes. If I have to keep only last one weeks data and purge the older indices. When I will delete index names except the last 7 days, I have no track which days logs are kept in which index name. Eg: 2014-12-07 date's logs may be kept in any of index named logstash-2015-03-19 or logstash-2015-03-20.

So how shall I delete indexes??

Siddharth Trikha
  • 2,648
  • 8
  • 57
  • 101

1 Answers1

2

Log messages are stored into indexes based on the value of the @timestamp field (which uses UTC time). If your 2014-12-07 logs end up in 2015-03-19 this timestamp parsing isn't done correctly.

Correct the problem by adding a grok and/or date filter and your 2014-12-07 logs will end up in the logstash-2014.12.07 index and it'll be trivial to clean up old logs.

Magnus Bäck
  • 11,381
  • 3
  • 47
  • 59
  • I am not using the date filter at present. It's like index name is created by the date the ELK setup is made up not by logdate. Eg: I start ELK setup today to read logs of 2014, the index name will be `logstash-TODAY'S DATE`. So date filter will solve this? – Siddharth Trikha Mar 20 '15 at 06:52
  • Yes. The whole purpose of that filter is to set the `@timestamp` field, and that field controls in which index a log message ends up. – Magnus Bäck Mar 20 '15 at 08:47
  • I created a new question for the edit: http://stackoverflow.com/questions/29384754/syslog-timestamp-without-year – Siddharth Trikha Apr 01 '15 at 07:02