Fluentd Elasticsearch target index

Question

I'm using Fluentd to transfer the data into Elasticsearch.

td-agent.conf

## ElasticSearch
<match es.**>
  type elasticsearch
  target_index_key @target_index  
  logstash_format true
  flush_interval 5s
</match>

Elasticsearch index :

"logstash-2016.02.24" : {
    "aliases" : { },
    "mappings" : {
      "fluentd" : {
        "dynamic" : "strict",
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "dummy" : {
            "type" : "string"
          }

        }
      }
    },

Transmit json data :

$ curl -X POST -d 'json={"@target_index": "logstash-2016.02.24","dummy":"test"}' http://localhost:8888/es.test

It should write the data to the given index instead of that It creates new index - logstash-2016.02.25 and it will write data into that. I want to write data to the given index.

Here is the Fluentd elasticsearch github link : https://github.com/uken/fluent-plugin-elasticsearch

Please correct me if I'm missing something.

I think you have a typo, `type elasticsearch` should read `@type elasticsearch`. — Val, Feb 29 '16 at 07:06
I'm using td-agent v2 on centOS, it's working properly without @. — Nimit, Feb 29 '16 at 14:14

score 4 · Answer 1 · edited Jun 17 '19 at 05:23

4

Maybe this is old but actually I run into the same problem and solved by

logstash_format false
index_name fluentd

This create only fluentd as index. From official fluentd tutorial https://docs.fluentd.org/output/elasticsearch

logstash_format (optional): With this option set true, Fluentd uses the conventional index name format logstash-%Y.%m.%d (default: false). This option supersedes the index_name option.

For clean up of old indices please consider using Curator: https://github.com/elastic/curator

I hope it helps someone.

edited Jun 17 '19 at 05:23

Obsidian

3,719
8
17
30

answered Jun 16 '19 at 21:44

Miroslav Spasovski

41
2

1

Thanks, this little sliver of information was exactly what I needed to get unblocked. – joar Apr 17 '20 at 20:00
Hi, I am still running into the same problem. When I change the index_name the new indices dont appear on Kibana but when I set it back to the default index logstash it appears. Any suggestions are welcomed! – Poh Peng Ric Oct 11 '20 at 08:46

Bilal Usean · Answer 2 · 2016-02-26T22:31:58.800

3

try this, its due to logstash_format true, please enter your index name in below index_name field (default value is fluentd)

<match es.**>
@type elasticsearch
host localhost
port 9200
index_name <.....your_index_name_here.....>
type_name fluentd
flush_interval 5s
</match>

after run this, please check index created or not by load below url in your browser

http://localhost:9200/_plugin/head/

have a good luck

edited Feb 26 '16 at 22:31

answered Feb 26 '16 at 22:20

Bilal Usean

2,322
3
22
45

Thanks for the answer. I have tried to change the logstash_format to false, but I'm getting same issue., now it creates duplicate index type. – Nimit Feb 29 '16 at 04:59
did you declare index_name? – Bilal Usean Mar 02 '16 at 03:51
Yes. in my case it's test_index – Nimit Mar 02 '16 at 04:39
Does it always take index_name in lowercase? I tried giving it in uppercase but it converted index_name into lowercase values. – NightOwl19 Jul 11 '19 at 09:36

score 2 · Answer 3 · edited Feb 26 '21 at 09:02

2

Its because you set logstash_format true, so you have to set the logstash_prefix.

Apache httpd example:

  logstash_prefix fluentd.httpd # defaults to "logstash"
  logstash_prefix_separator _   # defaults to "-"

edited Feb 26 '21 at 09:02

Fopa Léon Constantin

11,863
8
48
82

answered Mar 24 '19 at 11:10

NOZUONOHIGH

1,892
1
20
20

Fluentd Elasticsearch target index

3 Answers3