filebeat marking the log file inactive even when there is unread content in it

Question

I am using filebeat version 5.6.16 on a centos server to push logs to logstash from path /opt/news-bff/logs/Icis.Genesis*.log

There are many matching log files

-rw-r--r--. 1 root root  5049 Sep 25 10:30 Icis.Genesis.News.Bff.Api-2019092510.log
-rw-r--r--. 1 root root  1551 Sep 25 12:15 Icis.Genesis.News.Bff.Api-2019092512.log
-rw-r--r--. 1 root root  2650 Sep 25 13:55 Icis.Genesis.News.Bff.Api-2019092513.log
-rw-r--r--. 1 root root 39447 Sep 25 14:50 Icis.Genesis.News.Bff.Api-2019092514.log
-rw-r--r--. 1 root root  6191 Sep 25 15:31 Icis.Genesis.News.Bff.Api-2019092515.log

But the issue is, filebeat opens harvester for every file, after five minutes marks it as inactive withiust shipping any logs. The /var/lib/filebeat/registry file is empty.

I prvision multiple centos server and insall filebeat on it using puppet. 5 out of 10 times , the filebeat picks the log files wothout any issues. otherwise i get this issue.

If delete the registry file and resart the service it works fine.

The filebeat prospector looks like below:

filebeat:
  prospectors:
    - input_type: log
      paths:
        - /opt/news-bff/logs/Icis.Genesis*.log
      encoding: plain
      fields_under_root: false
      document_type: log
      scan_frequency: 10s
      harvester_buffer_size: 16384
      max_bytes: 10485760`

logs:
2019-09-25T12:16:01+01:00 INFO Harvester started for file: /opt/news-bff/logs/Icis.Genesis.News.Bff.Api-2019092512.log
2019-09-25T12:16:10+01:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.publish.read_bytes=36
2019-09-25T12:16:40+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:17:10+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:17:40+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:18:10+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:18:40+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:19:10+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:19:40+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=30
2019-09-25T12:20:10+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:20:40+01:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36
2019-09-25T12:21:06+01:00 INFO File is inactive: /opt/news-bff/logs/Icis.Genesis.News.Bff.Api-2019092512.log. Closing because close_inactive of 5m0s reached.

filebeat marking the log file inactive even when there is unread content in it

0 Answers0