0

I read input from log file and write to kafka. even after log rotation, inode doesnt change. after log rotation, still reads rotated log file(xx.log.2020-xx-xx) instead of pointing to main file(xx.log).

Below is my config file setting for input file. Do I need to add any other config to ignore reading old files.

input {
  file {
    path => "C:/Users/xx.log"
  }
}
YLR
  • 1,503
  • 4
  • 21
  • 28
mouli
  • 11
  • 5
  • does it read `xx.log` after rotation at all ? – karan shah Oct 20 '20 at 14:19
  • @karanshah It reads the old log file(xx.log.2020-xx-xx) and finally after few hours(12 hr), it starts reading current file(xx.log). Adding sincedb_clean_after, will it help. – mouli Oct 20 '20 at 19:15
  • Yes I think setting `sincedb_clean_after` and `ignore_older` would help. Also change the path filter to something like `xx.log*` so logstash can track all old and new files and determine which ones to read. Check out this known issue regarding logrotation and file beats (https://www.elastic.co/guide/en/beats/filebeat/7.9/file-log-rotation.html). I think they are applicable to logstash as well. – karan shah Oct 21 '20 at 14:21

1 Answers1

0

it's the same issue as this one. Logstash handles pretty well file rotation by default.

All you need to do is to make sure to use a glob pattern (e.g. ...log*) that identifies all of your log files and Logstash will keep track of them:

input {
  file {
    path => "C:/Users/xx.log*"
  }
}
Val
  • 207,596
  • 13
  • 358
  • 360
  • I tried giving log*. That didnt help. It created new inode when rotation happens but still read from old rotated file and send data to kafka. I see and confirm that by running in debug mode. – mouli Oct 26 '20 at 14:56
  • with logstash 7.9.3, when I have input file path as xx.log* and when log rotation happens like xx.log.1/ xx.log.2 etc... it works. It read the file correctly pointing to xx.log. For rollover with dates like xx.log.2020-MM-DD etc... having input file path as xx.log* doesnt work. when rotation happens, it still read the old file in 5 min and dump it and then points to xx.log and read it. so still that reread happens. – mouli Oct 29 '20 at 13:25