Issue in reading log file that contains date in it's name

Question

I have 2 linux boxes setup in which 1 box contains one component which generates log and logstash installed in it to transfer the logs. And in other box I have redis elasticsearch and logstash. here logstash will act as logstash indexer to grok the data.

Now my problem is that in 1st box component generate new log file everyday, but only difference in log file name varies as per date.

like

counters-20151120-0.log

counters-20151121-0.log

counters-20151122-0.log

and so on, I have included below type of code in my logstash shipper conf file:

file {
    path => "/opt/data/logs/counters-%{YEAR}%{MONTHNUM}%{MONTHDAY}*.log"
    type => "rg_counters"
  }

And in my logstash indexer, I have below type of code to catch those log files:

if [type] == "rg_counters" {
                grok{
                        match => ["message", "%{YEAR}%{MONTHNUM}%{MONTHDAY}\s*%{HOUR}:%{MINUTE}:%{SECOND}\s*(?<counters_raw_data>[0-9\-A-Z]*)\s*(?<counters_operation_type>[\-A-Z]*)\s*%{GREEDYDATA:counters_extradata}"]
        }
    }

output {
elasticsearch { host => ["elastichost1","elastichost1"  ] port => "9200" protocol => "http" }
stdout { codec => rubydebug }
}

Please note that this is working setup and other types log files are getting transfered and processed successfully, so there is no issue of setup.

The problem is how do I process this log file which contains date in it's file name.

Any help here?

Thanks in advance!!

You haven;t really described what the problem is with your config. On a quick glance, I'd say that your input{} path should be more like "counters-*.log" — Alain Collins, Nov 22 '15 at 21:59
The problem is , when I dump any counters type of log ifle into that path, its not getting parsed into elasticsearch, more likey its not coming into elasticsearch. — Amit Gawali, Nov 24 '15 at 05:58
Then it is getting parsed successfully.. but thing is my component is genarting logs everyday with difference of date in it's filename. so lets say I have to stop logstash shipper and indexer for some reason and if I start again then entire files may get process again and will cause duplication in elasticsearch. — Amit Gawali, Nov 24 '15 at 08:19
Logstash keeps information on what it has processed so far - which file (inode) and how many bytes (offset). Google for 'sincedb'. You'll be fine with a "*.log" pattern. — Alain Collins, Nov 24 '15 at 16:44

score 1 · Accepted Answer · answered Nov 24 '15 at 23:30

Based on the comments...

Instead of trying to use regexp patterns in your path:

path => "/opt/data/logs/counters-%{YEAR}%{MONTHNUM}%{MONTHDAY}*.log"

just use glob patterns:

path => "/opt/data/logs/counters-*.log"

logstash will remember which files (inodes) that it's seen before.

Issue in reading log file that contains date in it's name

1 Answers1