Fluentd not able to upload logs to Elasticsearch

Question

I have downloaded the elasticsearch & kibana on my Ubuntu18.04 machine. Both of them are running fine and I can access them. Below is the elasticsearch details:

{
  "name" : "TX-G1-000",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AobC_iiNSyyNftYl3pUJ7w",
  "version" : {
    "number" : "7.14.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I have also installed Fluentd and its service td-agent is running fine.

● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
   Loaded: loaded (/lib/systemd/system/td-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-09-20 21:18:12 IST; 7min ago
     Docs: https://docs.treasuredata.com/display/public/PD/About+Treasure+Data%27s+Server-Side+Agent
  Process: 5486 ExecStop=/bin/kill -TERM ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 5491 ExecStart=/opt/td-agent/bin/fluentd --log $TD_AGENT_LOG_FILE --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 5498 (fluentd)
    Tasks: 12 (limit: 4915)
   CGroup: /system.slice/td-agent.service
       ├─5498 /opt/td-agent/bin/ruby /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
       └─5501 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid --under-supervisor

Sep 20 21:18:11 TX-G1-000 systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Sep 20 21:18:12 TX-G1-000 systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.

Below is my td-agent.conf file:

<source>
  @type tail
  path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
  pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos
  format json
  time_format %Y-%m-%d %H:%M:%S
  tag health01
</source>

<source>
  @type tail
  path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
  pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos
  format json
  time_format %Y-%m-%d %H:%M:%S
  tag cycle01
</source>

<match health*>
  @type elasticsearch
  hosts http://localhost:9200/
  index_name health_skl_device
  type_name health
</match>

<match cycle*>
  @type elasticsearch
  hosts http://localhost:9200/
  index_name cycle_skl_device
  type_name cycle
</match>

When running the td-agent, below are its logs:

2021-09-20 21:18:12 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluentd' version '1.13.3'
2021-09-20 21:18:12 +0530 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json"
    pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos"
    format json
    time_format %Y-%m-%d %H:%M:%S
    tag "health01"
    <parse>
      time_format %Y-%m-%d %H:%M:%S
      @type json
      unmatched_lines 
      time_type string
    </parse>
  </source>
  <source>
    @type tail
    path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json"
    pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos"
    format json
    time_format %Y-%m-%d %H:%M:%S
    tag "cycle01"
    <parse>
      time_format %Y-%m-%d %H:%M:%S
      @type json
      unmatched_lines 
      time_type string
    </parse>
  </source>
  <match health*>
    @type elasticsearch
    hosts "http://localhost:9200/"
    index_name "health_skl_device"
    type_name "health"
  </match>
  <match cycle*>
    @type elasticsearch
    hosts "http://localhost:9200/"
    index_name "cycle_skl_device"
    type_name "cycle"
  </match>
</ROOT>
2021-09-20 21:18:12 +0530 [info]: starting fluentd-1.13.3 pid=5491 ruby="2.7.4"
2021-09-20 21:18:12 +0530 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2021-09-20 21:18:13 +0530 [info]: adding match pattern="health*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding match pattern="cycle*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: #0 starting fluentd worker pid=5501 ppid=5498 worker=0
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
2021-09-20 21:18:13 +0530 [info]: #0 fluentd worker is now running worker=0

I do not see any error logs but not sure why its not able to upload data. I try to create index pattern, it doesnt matches on Kibana. Can anyone please help me in debugging this issue. Thanks

Logs after adding debug

2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name health_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: health_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name cycle_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: cycle_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: No fluent logger for internal event

Since you're running ES 7.14, you need to remove the `type_name` property in the output, as it's not needed anymore (see the two warnings). It's probably not the issue, but it will be more correct. As mentioned by Yuri you should increase the log level so we can get more insights — Val, Sep 22 '21 at 18:44

Yuri Sh · Answer 1 · 2021-09-23T05:48:26.583

0

I think you have incorrect match tags. Nowhere in documentation does it mention that asterisks can be used that way, they should either take a place of a whole tag part or be used inside a regular expression. According to this section, Fluentd accepts all non-period characters as a part of a tag. So in fact health* is a valid name for a tag, fluentd expects exact matches of that string.

You should try using /health.*/ and /cycle.*/ instead.

Better yet, you can go the intended way, change the tag names to health.01 and cycle.01 and use health.** and cycle.** for matching.

edited Sep 23 '21 at 05:48

answered Sep 23 '21 at 05:07

Yuri Sh

899
1
7

That also explains why it hasn't reported any errors. – Yuri Sh Sep 23 '21 at 05:23
Why does the info log state `adding match pattern="health*" type="elasticsearch"` then? – Val Sep 23 '21 at 05:49
@Val `health*` is a valid matching "pattern", but only one tag can satisfy it - `health*` letter for letter. It will match `health*` and won't match `health01`. – Yuri Sh Sep 23 '21 at 05:52
Thank you for the explanation, I'm not a Fluentd expert, but glad I learned something new ;-) – Val Sep 23 '21 at 05:54
What if I have tag as `health.EAB2`, `health.ECB2`, `health.7CDE` so as per your suggestion, I should use match tag has `health.****` so that it matches data from all the 3 input health.? – S Andrew Sep 23 '21 at 08:17
1

@SAndrew no, you still need to use `health.**`. `.**` at the end matches any amount of *tag parts* (zero or more). Tag parts are everything in-between dots. `health.EAB2` has tag parts `health` and `EAB2` – Yuri Sh Sep 23 '21 at 13:58
@YuriSh Well I don't exactly agree on this, if the tag is `health01` I can use `health*` in match pattern. `health*` means anything after health will be matched and pushed to elasticsearch. But anyways thanks for your help. I have resolved the problem. Issue was not in fluentd configuration file. It was actually on how I was saving my `TX-S2-SKL-001_health.json`. I will answer to update – S Andrew Sep 23 '21 at 17:06

Fluentd not able to upload logs to Elasticsearch

1 Answers1