0

I am installing EFK stack to 100 days old cluster. Fluentd will pick up the logs of 100 days and will start sending to Elastic. Is there a provision that fluentd starts aggregating the log from today and not from beginning life cycle of the cluster?

Arpan Sharma
  • 395
  • 1
  • 10
  • 20

1 Answers1

1

If you look into Fluentd documentation, you can find limit_recently_modified flag, which allows limit the watching files that the modification time is within the specified time range.

Here's how the limit_recently_modified can be used in conf file:

...
<source>
    exclude_path ["/var/log/wedge/*/*/MattDaemon*.log"]
    path_key source
    format none
    read_from_head true
    tag foo.*
    path /var/log/wedge/*/*/*.log
    pos_file /var/log/td-agent/wedgelog
    limit_recently_modified 86400s
    @type tail
</source>
...

Another option is to use Filebeat (instead of Fluentd), where you can find ignore_older flag. Filebeat ignores any files that were modified before the specified timespan.

I hope it will helps you.

aga
  • 3,790
  • 3
  • 11
  • 18
  • Thanks, but in the YAML can we set this as an environment variable or is there another way? – Arpan Sharma Sep 30 '19 at 17:21
  • @Arpan Sharma I have edited answer and added conf file. `limit_recently_modified` parameter is useful when use * in the path. If target files are not updated within `limit_recently_modified`, such files are ignored from watching list. – aga Oct 06 '19 at 19:35