3

I'm having some issues getting filebeat to exclude lines from apache2's access log. I've got the apache2.yml config enabled and it does exclude log files but not lines. Here's an example of the line I'm looking to exclude:

example.site.com:80 192.168.0.1 - - [20/Dec/2017:10:18:37 -0500] "GET /server-status?auto= HTTP/1.1" 301 522 "-" "Go-http-client/1.1"

Here are the ways I've tried to use the regexp format with the exclude_lines: option in the apache2.yml file:

exclude_lines: ['.*server-status.*'] exclude_lines: ['(?i:/server-status\?auto=)'] exclude_lines: ['GET /server-status']

as well as a few other variations. I've also tried a processor in both the main filebeat config file and in the apache2 module config like this:

- module: apache2 # Access logs access: enabled: true processors: - drop_event: when: regexp: apache2.access.url: "server-status"

I tried it with contains: in place of regexp: but that doesn't seem to work either. But this:

exclude_files: [".gz$"] DOES work; from the log file:

017-12-21T10:15:27-05:00 DBG [prospector] Exclude file: /var/log/apache2/example.site.com/access.log.10.gz

It's all been making me a little nuts. Doesn't matter which way I try the expression or processor, the example log line above is still being fed to elasticsearch. I posted a similar question over at the discuss.elastic.co forums but no one has replied thus far.

Does anyone have experience excluding log lines? Filebeat, Elasticsearch, and Logstash are all at version 6.1.0.

Thanks in advance!

robscott27
  • 31
  • 3
  • Try this exclude_lines: ['.*server-status.*','(?i:/server-status\?auto=)','GET /server-status'] in filebeat.yml – Ali Ahmad Dec 22 '17 at 07:37
  • Thanks for the reply. Unfortunately, even after adding you suggesting into the filebeat.yml config file and restarting the service, those events are still being passed through into ES. – robscott27 Dec 22 '17 at 14:01
  • I almost get the feeling that it's not even reading/using the exclude_lines option in the first place. I decided to try to drop everything in the log file, regardless of the content using ['.'] but everything still goes into ES. – robscott27 Dec 22 '17 at 14:07
  • 2
    I figured out what the issue was. I noticed that even though I had commented out the exclude_files line, it was still excluding the .gz log files, even though it shouldn't. So I went digging. Turns out, there's an access.yml file located here: /usr/share/filebeat/module/apache2/access/config which still had the exclude_files option set to ignore the .gz files. Below that line, I added: exclude_lines: ['.*(?:server-status).*'] and restarted the service. Watched the log file, and bingo, lines are being dropped now. – robscott27 Dec 22 '17 at 14:36

2 Answers2

0

I figured out what the issue was. I noticed that even though I had commented out the exclude_files line, it was still excluding the .gz log files, even though it shouldn't. So I went digging.

Turns out, there's an access.yml file located here: /usr/share/filebeat/module/apache2/access/config which still had the exclude_files option set to ignore the .gz files. Below that line, I added: exclude_lines: ['.*(?:server-status).*'] and restarted the service. Watched the log file, and bingo, lines are being dropped now.

-- robscott27

Community
  • 1
  • 1
kenorb
  • 155,785
  • 88
  • 678
  • 743
0

The below is targeting Filebeat v7.17.10 so I'm making the assumption it worked the same on v6.* or that anyone else hitting this problem might be able to make use of it if they upgrade. Note that on v7 the filebeat references/paths to the Apache module changed from apache2 to apache.

Rather than amending what I assume is the default config file held under /usr/share/filebeat/module/apache/ (which will then likely mean changes being lost on any update of that package), this can be achieved with the /etc/filebeat/modules.d/apache.yml file which I assume was originally being targeted.

So a simple example for excluding server-status requests from the default Apache access log as was requested would be:

- module: apache
  access:
    enabled: true
    var.paths: ["/var/log/apache2/access.log"]
    input:
      exclude_lines: ['server-status']
Dave Jennings
  • 46
  • 2