How to add timestamp & key for elasticsearch using fluent

Question

I have configured fluentd and elasticsearch and they both are working fine. I am tailing a file and then reading its data and publishing it to elasticsearch. Below is the json data:

{"time": "2018-05-14T11:37:30.339593", "Data count": 78, "Data status": "Sent", "DataId": "332"}

and below is the fluentd configuration file:

<source>
  @type tail
  time_key time
  path /home/user/file.json
  format json
  tag first
</source>

<match *first*>
  @type elasticsearch
  hosts 192.168.196.118:9200
  user <username>
  password <password>
  index_name myindex
  type_name mytype
  id_key 100
  time_key time
</match>

In the above configuration file, I have added time_key as time is the time which I want to use from json data. But the data which I receive in the elasticsearch do not contain any time data.

Also, I am using id_key but in the elasticsearch, id_key is some random values.

Please help. Thanks

score 4 · Answer 1 · answered May 14 '18 at 19:34

4

Try to inject the value you want:

<inject>
   time_key          @log_time
   time_format       %Y%m%dT%H%M%S%z
</inject>

Follow instructions at: https://docs.fluentd.org/v1.0/articles/inject-section

answered May 14 '18 at 19:34

Nicola Ben

10,615
8
41
65

4

Did you double check to make sure that elasticsearch plugin supports this? – James Forbes Feb 20 '21 at 23:20

score 3 · Accepted Answer · answered May 21 '18 at 08:44

3

I resolved this issue. I didnt have to add anything extra in the config file. I simply selected time as time field in elasticsearch.

answered May 21 '18 at 08:44

S Andrew

5,592
27
115
237

The other way around would be to use record_transformer or something like https://github.com/shunwen/fluent-plugin-rename-key to change `time` to whatever the default ES expects (I think @timestamp) – nijave Aug 30 '21 at 22:51

How to add timestamp & key for elasticsearch using fluent

2 Answers2

Linked