Nginx module for filebeats doesn't parse access logs

Question

I am using nginx module for filebeats to send log data to elasticsearch. Here is my filebeats configuration:

output:
  logstash:
    enabled: true
    hosts:
      - logstash:5044
    timeout: 15

filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]

The problem is that logs are not parsed. This is what I see in Kibana:

{   "_index": "filebeat-2017.07.18",   "_type": "log",   "_id": "AV1VLXEbhj7uWd8Fgz6M",   "_version": 1,   "_score": null,   "_source": {
    "@timestamp": "2017-07-18T10:10:24.791Z",
    "offset": 65136,
    "@version": "1",
    "beat": {
      "hostname": "06d09033fb23",
      "name": "06d09033fb23",
      "version": "5.5.0"
    },
    "input_type": "log",
    "host": "06d09033fb23",
    "source": "/var/log/nginx/access.log",
    "message": "10.15.129.226 - - [18/Jul/2017:12:10:21 +0200] \"POST /orders-service/orders/v1/sessions/update/FUEL_DISPENSER?api_key=vgxt5u24uqyyyd9gmxzpu9n7 HTTP/1.1\" 200 5 \"-\" \"Mashery Proxy\"",
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied"
    ]   },   "fields": {
    "@timestamp": [
      1500372624791
    ]   },   "sort": [
    1500372624791   ] }

I am missing parsed fields, as specified in the documentation: https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-nginx.html

Why are log lines not parsed?

Probably this it the case why it is not working (https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html): At the moment, Filebeat modules require using the Elasticsearch Ingest Node. In the future, Filebeat Modules will be able to also configure Logstash as a more powerful alternative to Ingest Node. — Uros K, Jul 18 '17 at 10:30

score 7 · Accepted Answer · answered Oct 10 '17 at 14:41

7

When you run filebeat -v -modules=nginx -setup, it will essentially create 4 things:

mapping template
kibana dashboards
machineLearning job
filters in the ingest node

Here are the filters for parsing:
- nginx access log
- nginx error log

The filters are stored in the ingest node. You can access them on:
http://YourElasticHost:9200/_ingest/pipeline

So if you want your logs parsed, you need to send them via the ingest node.

answered Oct 10 '17 at 14:41

sumid

1,871
2
25
37

Thank you. I found those filters and used them in logstash, so the end result is the same. – Uros K Oct 11 '17 at 13:15
4

@UrosK Do you mind to share a file with those ingest filters converted into logstash filters? Thank you. – Xplouder Oct 18 '18 at 13:08
The given command works as `filebeat -v --modules=nginx setup` in version 7.x.x for filebeat. – Ashutosh Kumar Nov 17 '21 at 14:24

Nginx module for filebeats doesn't parse access logs

1 Answers1