How could I manage Graylog to parse my JSON logs correctly?

Question

I have an rails app and I'm trying to configure logging to graylog. Pipeline consists of next steps: 1) Logs are written to file in JSON format by SemanticLogger gem. Log message consist of header info (first level tags) and payload with several levels of hierarchy:

{
  "tag": "mortgage",
  "app": "sneakers",
  "pid": 3448,
  "env": "production",
  "host": "thesaurus-mortgage",
  "thread": "91090300",
  "level": "info",
  "name": "Sneakers",
  "payload": {
    "class": "EgrnListenerWorker",
    "method": "work",
    "json": {
      "resource": "kontur",
      "action": "request_egrn_done",
      "system_code": "thesaurus",
      "id": 35883717,
      "project_id": "mortgage",
      "bank_id": "ab",
      "params": {
        "egrn": {
          "zip": "rosreestr/kontur/kontur_4288_2018-10-11_021848.zip",
          "pdf": "rosreestr/kontur/kontur_4288_2018-10-11_021848.pdf",
          "xml": "rosreestr/kontur/kontur_4288_2018-10-11_021848.xml"
        },
        "code": "SUCCESS"
      }
    },
    "valid_json": true
  },
  "created_at": "2018-10-11T17:44:58.262+00:00"
}

2) File is being read by Filebeat service and sent to Graylog.

And graylog could not parse correctly payload contents:

As you can see - keys are concatenated with ":" in one string in such manner: key1=value1:key2=value2. This is not what I am expected. It would be perfect if I could manage graylog to parse contents of payload into different fields with names payload.key1, payload.key2 and so on (so I could perform search on these fields)

ps: my log data is heterogeneous, i.e. payload contents depend on functionality it was produced by, so I expect that there would be a huge amount of different fields of a kind "payload.xxxxx" - is it ok?

score 1 · Answer 1 · answered Oct 23 '18 at 14:25

1

This isn't exactly a filebeat question since filebeat only ships the logs in their original JSON format (zipped, if wanted).

From the Graylog Website: http://docs.graylog.org/en/2.4/pages/extractors.html

Using the JSON extractor Since version 1.2, Graylog also supports extracting data from messages sent in JSON format.

Using the JSON extractor is easy: once a Graylog input receives messages in JSON format, you can create an extractor by going to System -> Inputs and clicking on the Manage extractors button for that input. Next, you need to load a message to extract data from, and select the field containing the JSON document. The following page let you add some extra information to tell Graylog how it should extract the information.

This should get you going.

answered Oct 23 '18 at 14:25

Faulander

327
3
12

Nope, extractors do not work, I’ve tried them already and I think that reason is that json format of payload key is broken at the moment when extractor comes to place. So it could not extract data from it – Andrey Khataev Oct 23 '18 at 19:42
Wouldn't it be the correct approach to ask the greylog devs to fix their broken connectors? :) – Faulander Oct 24 '18 at 12:28
They need help to do it ) First, we need to find out broken unit of our chain - my json log file or filebeat or graylog itself? All votes are against latter for now – Andrey Khataev Oct 25 '18 at 03:44

How could I manage Graylog to parse my JSON logs correctly?

1 Answers1