Chaining grok filter patterns for logstash

Question

I am trying to configure logstash to manage my various log sources, one of which is Mongrel2. The format used by Mongrel2 is tnetstring, where a log message will take the form

86:9:localhost,12:192.168.33.1,5:57089#10:1411396297#3:GET,1:/,8:HTTP/1.1,3:200#6:145978#]

I want to write my own grok patterns to extract certain fields from the above format. I received help on this question trying to extract the host. So if in grok-patterns I define

M2HOST ^(?:[^:]*\:){2}(?<hostname>[^,]*)

and then in the logstash conf specify

filter {
  grok {
    match => [ "message", "%{M2HOST}" ]
  }
}

it works as expected. The problem I now have is I want to specify multiple patterns e.g. M2HOST, M2ADDR etc. I tried defining additional ones in the same grok-patterns file

M2HOST ^(?:[^:]*\:){2}(?<hostname>[^,]*)
M2ADDR ^(?:[^:]*\:){3}(?<address>[^,]*)

and changing the logstash conf

filter {
  grok {
    match => [ "message", "%{M2HOST} %{M2ADDR}" ]
  }
}

but now I just get the error _grokparsefailure.

your problem here is the concatenation, think replacing the M2HOST and M2ADDR by their regex, you'll get `^(?:[^:]*\:){2}(?[^,]*) ^(?:[^:]*\:){3}(?
[^,]*)` You have a start of line in the middle of your match which obviously can't work. The main idea with pultiples ALIASES is to split the regex part to reuse them but not to magically merge them. — Tensibai, Sep 24 '14 at 14:56
ah ok. What I want is to apply the M2HOST regex to the message, and then the M2ADDR to the message. Do you know what the correct syntax would be? — Philip O'Brien, Sep 24 '14 at 14:57
Well I would work with the grok tester you already know, trynig to match each field, once that done you may split the interesting parts. for this exmeple I would say (tested) `(?:[^:]*:){2}(?[^,]*)[^:]*:(?
[^,]*)` would do — Tensibai, Sep 24 '14 at 15:00
Excellent, thank you. That worked (missing a closing bracket) — Philip O'Brien, Sep 24 '14 at 15:04

score 1 · Accepted Answer · answered Sep 24 '14 at 15:07

with your sample input from other question and with some guessing about the values names the full match would be:

(?:[^:]*:){2}(?<hostname>[^,]*)[^:]*:(?<address>[^,]*)[^:]*:(?<pid>[^#]*)[^:]*:(?<time>[^#]*)[^:]*:(?<method>[^,]*)[^:]*:(?<query>[^,]*)[^:]*:(?<protocol>[^,]*)[^:]*:(?<code>[^#]*)[^:]*:(?<bytes>[^#]*).*

Producing:

{
  "hostname": [
    [
      "localhost"
    ]
  ],
  "address": [
    [
      "192.168.33.1"
    ]
  ],
  "pid": [
    [
      "57089"
    ]
  ],
  "time": [
    [
      "1411396297"
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "query": [
    [
      "/"
    ]
  ],
  "protocol": [
    [
      "HTTP/1.1"
    ]
  ],
  "code": [
    [
      "200"
    ]
  ],
  "bytes": [
    [
      "145978"
    ]
  ]
}

Chaining grok filter patterns for logstash

1 Answers1