How to set an Elasticsearch output template in Logstash

Question

I’m relatively new to Kibana and the ELK (Elasticsearch, Logstash and Kibana) stack and I’ve been doing pretty well setting one up, but I have run into what I see as an odd issue and need some help understanding what’s happening.

I’m using the ELK stack to crunch some Apache logs but I have my own custom type settings. So I need to explicitly specify field types and such instead of having Logstash (or is it Kibana?) guess what the data mapping would be.

From reading the Logstash documentation, it seems pretty clear that I can set the template value in the output.elasticsearch chunk of config shown here:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-apache"
    document_id => "%{[@metadata][fingerprint]}"
    manage_template => false
    template => "/path/to/logstash/logstash-apache.json"
    template_name => "logstash-apache"
    template_overwrite => true
  }
  stdout {
    codec => rubydebug
  }
}

100% sure I have the correct path set. But for some reason, if I use this, launch Logstash and let it do it’s things, the mappings I have specified in logstash-apache.json don’t show up. The index in Kibana is logstash-apache as well so this should work right?

So what I do now is preload the mappings template directly into Elasticsearch like this:

curl -ss -XPUT "http://localhost:9200/_template/logstash-apache/" -H 'Content-Type: application/json' -d @"/path/to/logstash/logstash-apache.json";

And it clearly works well and the data gets proper mapping… But doing something like this is fairly clunky. It would be cleaner to just have it all come from the logstash-apache.conf file I have setup.

So what am I doing wrong? What can I do to have my custom mappings template be used via that logstash-apache.conf without having to jump through the extra hoop of a curl command?

score 22 · Accepted Answer · answered Apr 13 '18 at 04:00

22

The problem is that you have set manage_template to false, which completely disables this template creation feature and requires you to create the template manually like you're doing right now.

So your output section should look like this instead and you should be good to go:

  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-apache"
    document_id => "%{[@metadata][fingerprint]}"
    manage_template => true                              <-- change this line
    template => "/path/to/logstash/logstash-apache.json"
    template_name => "logstash-apache"
    template_overwrite => true
  }

answered Apr 13 '18 at 04:00

Val

207,596
13
358
360

1

Thanks! I knew it would be something simple. But not that simple. But appreciate it! Just one small thing: As a relative newbie to the ELK stack world, reading the description for [`manage_template`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-manage_template) compared to simply [`template`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template) was a bit confusing. Was not aware one value affected the other. – Giacomo1968 Apr 13 '18 at 05:08
4

Basically, `manage_template` toggles on/off the template management. `template` gives the path to the template in case template management is toggle on. `template_name` is the name under which the template is saved in ES. and `template_overwrite` tells Logstash whether to overwrite the template if one exists already under the same name. – Val Apr 13 '18 at 05:18
2

Thanks for the further clarification. I understand that stuff at this point. But all I am saying now—in the context of this question and this answer—is how the official documentation seems a bit confusing in this context. [`manage_template`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-manage_template)’s and [`template`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template)’s stated purpose in official documentation doesn’t really explain their connection. – Giacomo1968 Apr 13 '18 at 14:27
1

Fair enough, you're free to suggest improvements to the documentation if you desire. – Val Apr 13 '18 at 14:38

How to set an Elasticsearch output template in Logstash

1 Answers1