Pushing structured log data directly to elastic search with filebeat

Question

I have configured filebeat to harvest my structured log output (greenfield project so each log entry is a JSON document in a pre-defined format) and publish it directly to ELS.

Example log file excerpt (note that additional is free form, all other properties are fixed. Pretty formatted for this post but each top level object is on a single line in the file):

{
    "TimeUtc": "2016-09-23T14:13:02.217520245Z",
    "ServiceKey": "MAAS_SVC",
    "Title": "Get All Campaigns - Start",
    "Additional": {
        "HTTPRequest": {
            "Method": "GET",
            "URL": {
                "Scheme": "",
                "Opaque": "",
                "User": null,
                "Host": "",
                "Path": "/admin/campaigns",
                "RawPath": "",
                "ForceQuery": false,
                "RawQuery": "",
                "Fragment": ""
            },
            "Proto": "HTTP/1.1",
            "ProtoMajor": 1,
            "ProtoMinor": 1,
            "Header": {
                "Accept": ["*/*"],
                "Accept-Encoding": ["gzip, deflate"],
                "Connection": ["keep-alive"],
                "Requestkey": ["78478050-47f0-4d0d-44e8-615d0599574a"],
                "User-Agent": ["python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"]
            },
            "Body": {
                "Closer": {
                    "Reader": null
                }
            },
            "ContentLength": 0,
            "TransferEncoding": null,
            "Close": false,
            "Host": "xxxxxxxxx",
            "Form": null,
            "PostForm": null,
            "MultipartForm": null,
            "Trailer": null,
            "RemoteAddr": "xxx.xxx.xxx.xxx",
            "RequestURI": "/admin/campaigns",
            "TLS": null,
            "Cancel": ,
            "Response": null
        }
    },
    "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
    "HostAddress": "xxxxxxxxx"
}

This results in filebeat making the following request to ELS:

{
    "@timestamp": "2016-10-12T13:53:21.597Z",
    "beat": {
        "hostname": "7bca0e28e69e",
        "name": "7bca0e28e69e"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "message": "{\"TimeUtc\":\"2016-09-23T14:13:02.217520245Z\",\"ServiceKey\":\"MAAS_SVC\",\"Title\":\"Get All Campaigns - Start\",\"Additional\":{\"HTTPRequest\":{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/admin/campaigns\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Connection\":[\"keep-alive\"],\"Requestkey\":[\"78478050-47f0-4d0d-44e8-615d0599574a\"],\"User-Agent\":[\"python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic\"]},\"Body\":{\"Closer\":{\"Reader\":null}},\"ContentLength\":0,\"TransferEncoding\":null,\"Close\":false,\"Host\":\"bistromath.marathon.mesos:40072\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.1.70:42854\",\"RequestURI\":\"/admin/campaigns\",\"TLS\":null,\"Cancel\":,\"Response\":null}},\"RequestKey\":\"78478050-47f0-4d0d-44e8-615d0599574a\",\"HostAddress\":\"ba47316c9c45\"}",
    "offset": 0,
    "source": "/filebeat/log-harvest/maas-service-single.log",
    "type": "log"
}

Can I prevent filebeat from escaping my log JSON so that it's becomes a nested object rather than a string, or do I need to patch filebeat?

Can you please post a sample log line and the Filebeat prospector config that you are using. — A J, Oct 11 '16 at 22:05

score 3 · Accepted Answer · answered Oct 12 '16 at 22:50

It is possible to parse the JSON messages in Filebeat 5.x, but not in Filebeat 1.x. A json option can be specified in the configuration file.

If you are limited to using Filebeat 1.x, then you would need to Logstash to parse the JSON data from the message field. You would configure Filebeat -> Logstash -> Elasticsearch.

Filebeat 5.x configuration:

filebeat:
  prospectors:
    - paths:
        - input.json
      json.message_key: Title
      json.keys_under_root: true
      json.add_error_key: true

output:
  console:
    pretty: true

Sample output:

{
  "@timestamp": "2016-10-12T22:40:16.338Z",
  "Additional": {
    "HTTPRequest": {
      "Body": {
        "Closer": {}
      },
      "Close": false,
      "ContentLength": 0,
      "Header": {
        "Accept": [
          "*/*"
        ],
        "Accept-Encoding": [
          "gzip, deflate"
        ],
        "Connection": [
          "keep-alive"
        ],
        "Requestkey": [
          "78478050-47f0-4d0d-44e8-615d0599574a"
        ],
        "User-Agent": [
          "python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"
        ]
      },
      "Host": "xxxxxxxxx",
      "Method": "GET",
      "Proto": "HTTP/1.1",
      "ProtoMajor": 1,
      "ProtoMinor": 1,
      "RemoteAddr": "xxx.xxx.xxx.xxx",
      "RequestURI": "/admin/campaigns",
      "URL": {
        "ForceQuery": false,
        "Fragment": "",
        "Host": "",
        "Opaque": "",
        "Path": "/admin/campaigns",
        "RawPath": "",
        "RawQuery": "",
        "Scheme": ""
      }
    }
  },
  "HostAddress": "xxxxxxxxx",
  "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
  "ServiceKey": "MAAS_SVC",
  "TimeUtc": "2016-09-23T14:13:02.217520245Z",
  "Title": "Get All Campaigns - Start",
  "beat": {
    "hostname": "host",
    "name": "host"
  },
  "input_type": "log",
  "offset": 919,
  "source": "input.json",
  "type": "log"
}

NOTE: The JSON data you posted isn't valid. The Cancel field was missing a value. I set it to null before running the data through Filebeat.

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

It looks like Kibana 7.2 (June 2019) does have RBAC now, with feature control

Want to hide Dev Tools from the left navigation? Show Stack Monitoring only to admins? Or, give certain users access to only Dashboard and Canvas? Feature controls allow you to hide and restrict applications and features in the Kibana UI.

You can configure Kibana applications and features based on your users’ needs, and when used with security, based on their privileges.

This means different roles can have access to different features in the same space. Power users might have privileges to create and edit visualizations and dashboards, while analysts or executives might have Dashboard and Canvas with read-only privileges.

Pushing structured log data directly to elastic search with filebeat

2 Answers2