Grabbing multiple lines using Grok

Question

The following is a log sample I need to parse using logstash and the logstash grok filter:

2018-02-12 15:17:39.216 [DEBUG] [    60] [CashTransactionReportCommand] [4564 456] - Xml of valid cash: <NewDataSet>
  <Table>
    <transaction_id>546464</transaction_id>
    <device_trans_id>24</device_trans_id>
    <value>3.5000</value>
    <product_code>40</product_code>
    <product_pa_code>E1</product_pa_code>
    <catalog_number />
    <decimal_place>2</decimal_place>
    <site_id>2</site_id>
    <machineSeTime>2018-02-12T17:17:39.273+00:00</machineSeTime>
    <payment_method_id>3</payment_method_id>
    <actor_id>4566</actor_id>
    <operator_id>55</operator_id>
  </Table>
</NewDataSet>

I almost have everything I need:

%{TIMESTAMP_ISO8601:log_timestamp} \[%{LOGLEVEL:loglevel}\] \[%{DATA:snId}\] \[%{WORD:snName}\] (?<test>\[\d+ \d+\]) %{GREEDYDATA:logmessage}

My only problem with the "logmessage". I need it to contain everything passed "[4564 456]" until the end of the example.

Do all the lines all your message in the same logstash event? Because if that's not the case, you'll have to first group them in the same event, in logstash with the multiline codec or on your log shipper. — baudsp, Feb 13 '18 at 09:21
Ok I think I got you now. Need to "stack" the log before it's being sent to Logstash (in my case I'm using Beats). https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html — JustAGuy, Feb 13 '18 at 10:32
Please excuse me, I should have linked to how to do it. Here's another way to it, with a logstash codec: https://stackoverflow.com/a/34896295/6113627 — baudsp, Feb 13 '18 at 10:45
Logstash codec isnt the right answer for me as I'm using Beats. The link I provided earlier is the right one and I already started using it :) — JustAGuy, Feb 13 '18 at 14:21
You can post it as an answer if you want and I'll tag it. It got me going down the right path. — JustAGuy, Feb 13 '18 at 14:27
I was trying to offer an alternative, but I agree that doing the multiline on the shipper (filebeat) is a better solution. — baudsp, Feb 13 '18 at 15:52

score 0 · Accepted Answer · answered Feb 13 '18 at 16:32

In order to be able to parse the message, including the XML, you'll have to group all the lines in the same logstash event, so that when using the grok filter, the message field contains the whole message. This can be done:

in logstash with the multiline codec

Multiline in logstash

Multiline codec documentation

in filebeat with the multiline option

Multiline in filebeat

Documentation of multiline option in filebeat configuration

Grabbing multiple lines using Grok

1 Answers1