Fields parsed from log path not added in logstash

Question

I'm parsing multiple log files with logstash - and want to add fields based on the path of the files to my output. Here are the relevant parts of the config file:

input {
  file {
    path => "/mnt/logs/**/console-20200108*.log"
    type => "tomcat"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  if [type] == "tomcat" {
    grok {
      patterns_dir => "/usr/share/logstash/patterns"
      match => {
        "message" => [ "%{TOMCAT_LOG_1}", "%{TOMCAT_LOG_2}" ]
        "path" => "\/mnt\/logs\/%{DATA:site}\/%{DATA:version}\/node%{NUMBER:node}\/store\/tomcat\/%{DATA:file}\.log"
      }
    }
  }
}

Here's a sample output:

{
         "level" => "INFO",
          "type" => "tomcat",
          "data" => "Finished indexer cronjob.\r",
     "timestamp" => "08-Jan-2020 11:00:05.860",
    "qualifier1" => "[update-backofficeIndex-CronJob::ServicelayerJob]",
    "@timestamp" => 2020-01-08T11:04:47.364Z,
          "path" => "/mnt/logs/protec/qa/node1/store/tomcat/console-20200108.log",
    "qualifier3" => "[SolrIndexerJob]",
          "host" => "elk",
      "@version" => "1",
       "message" => "INFO   | jvm 1    | srvmain | 08-Jan-2020 11:00:05.860 INFO  [update-backofficeIndex-CronJob::ServicelayerJob] (update-backofficeIndex-CronJob) [SolrIndexerJob] Finished indexer cronjob.\r",
    "qualifier2" => "(update-backofficeIndex-CronJob)"
}

Based on this, I was expecting to get relevant fields from parsing the message and a few more fields from parsing the path. Yet, I none of the fields from "path" parsing are added.

What am I missing? How do I add site, version, node and file fields?

I don't see why it doesn't work, but you can split the grok filter in two, one for message and one for path — baudsp, Jan 08 '20 at 13:11
Apparently it might be coming from the `break_on_match` option, which defaults to true. From the [doc](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-break_on_match): `The first successful match by grok will result in the filter being finished.` So if `%{TOMCAT_LOG_1}` or `%{TOMCAT_LOG_2}` match, it won't try to match the path field — baudsp, Jan 08 '20 at 13:16
@baudsp Ah! That makes sense. When I separated grok filter into two separate ones - one for message, the other for path - it did work as expected! — Aleks G, Jan 08 '20 at 13:22
@baudsp If you add your comment as an answer, I'll accept it. — Aleks G, Jan 24 '20 at 12:18

score 0 · Accepted Answer · answered Apr 07 '20 at 16:21

Creating an answer from a comment that solved the problem (hence community wiki).

Apparently it might be coming from the break_on_match option, which defaults to true. From the doc:

The first successful match by grok will result in the filter being finished. So if %{TOMCAT_LOG_1} or %{TOMCAT_LOG_2} match, it won't try to match the path field

Fields parsed from log path not added in logstash

1 Answers1