Logstash Add field from grok filter

Question

Is it possible to match a message to a new field in logstash using grok and mutate?

Example log:

"<30>Dec 19 11:37:56 7f87c507df2a[20103]: [INFO] 2018-12-19 16:37:56 _internal (MainThread): 192.168.0.6 - - [19/Dec/2018 16:37:56] \"\u001b[37mGET / HTTP/1.1\u001b[0m\" 200 -\r"

I am trying to create a new key value where I match container_id to 7f87c507df2a.

filter {
  grok {
    match => [ "message", "%{SYSLOG5424PRI}%{NONNEGINT:ver} +(?:%{TIMESTAMP_ISO8601:ts}|-) +(?:%{HOSTNAME:service}|-) +(?:%{NOTSPACE:containerName}|-) +(?:%{NOTSPACE:proc}|-) +(?:%{WORD:msgid}|-) +(?:%{SYSLOG5424SD:sd}|-|) +%{GREEDYDATA:msg}" ]
  }
  mutate {
    add_field => { "container_id" => "%{containerName}"}
  }
}

The resulting logfile renders this, where the value of containerName isn't being referenced from grok, it is just a string literal:

"container_id": "%{containerName}"

I am trying to have the conf create:

"container_id": "7f87c507df2a"

Obviously the value of containerName isn't being linked from grok. Is what I want to do even possible?

I am not sure if I unterstand the question, but if so a simple `filter { mutate { copy => { "containerName" => "container_id" } } }` should do the job — Kali, Dec 19 '18 at 15:28
anther thing you can try is adding `add_field => { "container_id" => "%{containerName}"}` directly into the `grok` part - but if "containerName" is not matched in the grok pattern the key/value pair is not added to the result. — Kali, Dec 19 '18 at 15:35
@Quali neither have worked, is there an easier way to test that my grok matching is working properly? There seems to be no errors in the logs of logstash however. I updated the post with an example log file and what I am trying to do — Baily, Dec 19 '18 at 15:43
Your SYSLOG5424PRI matches "30", but %{NONNEGINT:ver} isn't going to match "Dec". Use the grok debugger to build your pattern up slowly from left to right: https://grokdebug.herokuapp.com/ — Alain Collins, Dec 19 '18 at 17:03
Or if you only want to grab `7f87c507df2a` and the container is always present at the same place in the logs, you can just use `%{NOTSPACE:containerName}\[`, using the `[` to find the containerName — baudsp, Dec 19 '18 at 17:21

score 4 · Accepted Answer · answered Dec 19 '18 at 17:56

As explained in the comments, my grok pattern was incorrect. For anyone that may wander towards this post that needs help with grok go here to make building your pattern less time consuming.

Here was the working snapshot:

filter {
  grok {
    match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP}%{SPACE}%{BASE16NUM:docker_id}%{SYSLOG5424SD}%{GREEDYDATA:python_log_message}" ]
    add_field => { "container_id" => "%{docker_id}" }    
  }  
}

Logstash Add field from grok filter

1 Answers1