Logstash stopping when metadata is in output

Question

I am trying to set up ELK Stack following this tutorial: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

However, there is a problem with Logstash: the service is stopping if there is a pattern in the output section, for example index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

With constant strings, however, it works just fine: index => "nginx_web"

Is there a way to trace data incoming from filebeat in order to check a problematic portion?

logstash 2.3.2, filebeat 1.2.3

here is a full logstash.conf:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/path/to/certs/logstash.crt"
    ssl_key => "/path/to/private/logstash.key"
  }
}

filter {
  grok {
    match => {
        "message" => "%{IPORHOST:hostname} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{DATA:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{INT:response} (?:%{INT:bytes}|-) \"%{NOTSPACE:referrer}\" %{QS:useragent} %{NUMBER:resptime}"
    }
    remove_field => [ "message", "fields", "@timestamp", "input_type", "host", "request" ]
  }
  mutate {
    gsub => [ "useragent", "\"{1}", "" ]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

thanks in advance!

Have you tried with only a single pattern, like `index => "beat-%{+YYYY.MM.dd}"` ? Does it work? — Val, Jun 02 '16 at 03:00
i did, but that didn't work, too. as for now only constants work — funkifunki, Jun 02 '16 at 10:03
Try not to remove the `@timestamp` field since it's used for the `%{+YYYY.MM.dd}` part — Val, Jun 02 '16 at 10:04
that worked! could you make it an answer, not a comment, so that i could accept it? thanks a lot! — funkifunki, Jun 02 '16 at 10:34

score 1 · Accepted Answer · answered Jun 02 '16 at 10:35

You should not remove the @timestamp field since it's used for the %{+YYYY.MM.dd} part in the index name.

If you absolutely want to remove the @timestamp field, another way would be to add a new field for the index name before removing the @timestamp field.

Add this before remove_field:

add_field => { "index" => "beat-%{+YYYY.MM.dd}"}

And then use %{index} in your elasticsearch output.

Logstash stopping when metadata is in output

1 Answers1