0

I'm currently using logstash to import dozens of log files from different webapps into Graylog. It works great the files are tagged so I know from wich webapp they originate.

I can't change the webapp thus I can't add a GELF appender to the log4j conf of the webapp. The idea is to periodically retrieve the log files, parse them and import them with logstash into Graylog.

My problem is how do I make sure I don't import a log event I've already imported. For example, I have a log file that has a log pattern that increments: log.1, log.2, etc. So I'll have log events that could be in log.1 the first time and 2 weeks later when I reimport them they'll maybe be in log.3. I'm afraid I can't handle that with logstash's file input "sincedb_path" and "start_position".

So here are a few options I've gathered and I'd like your input about them, if anyone encountered the same issue:

  • Use a logstash filter dropping all events before a certain date, requires to keep an index of every last log date of every file imported (potentially 50+) and a lot of configuration writing
  • Use of a drool rule in GrayLog to refuse logs with timestamps prior to last log received for a given type
  • Ask to change the log pattern to be something like log.date instead of a log pattern that renames files (but I'd rather avoid this one)
  • Any other idea?
IppX
  • 305
  • 1
  • 13
  • Does your log rotation change the file's inode? If not, Logstash may ignore it, as the sincedb tracks files by inode rather than by name. – rutter Apr 07 '15 at 17:47
  • I didn't know about this but yeah it does change the file inode, when a new log file is created and .log is moved to .log.1, .log.1 keeps its inode index and the new .log has a new one. So if logstash uses the inode number to track files it should detect the changes properly ?! – IppX Apr 08 '15 at 08:30
  • I would test it to be safe, but since Logstash tracks files by inode it sounds like you'll be okay. – rutter Apr 09 '15 at 20:24
  • @lppX- please answer my question http://stackoverflow.com/questions/31003208/how-to-import-old-log-files-to-graylog-as-input – Alok Jun 23 '15 at 13:04

0 Answers0