Stack Overflow
Stack Overflow

Questions tagged [td-agent]

Please use this tag for question regarding td-agent, a data collection daemon. https://support.treasuredata.com/hc/en-us/articles/360000687108-Overview-of-Server-Side-Agent-td-agent-

Arm Treasure Data provides Server-Side Agent called Treasure Agent (td-agent), to collect server-side logs and events. You can continuously import data using td-agent.

71 questions
0
votes
0 answers

td-agent configuration for postgresql Logs

I have installed the fluentd agent in my VM where postgresql is installed. Now I am looking for the td-agent.conf file which could help me to ship the pg logs and /var/log/messages to kibana. I am very new to ELF and hence looking for a sample file…
Bo_2000
  • 7
  • 2
0
votes
1 answer

fluentd(td-agent) output file, Time is missing from the filename

I want to get help. There is a message as below. The message is being received and saved to a file. I want to display the time in the file name, but it's failing. Env td-agent 4.3.1 fluentd 1.14.6 Input message { ... "kubernetes": { "host":…
ddakker
  • 55
  • 7
0
votes
0 answers

Fluentd Throwing error and not uploading data to Elastic

unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not a supported distribution of Elasticsearch."
Muhammad Ansab
  • 1
0
votes
1 answer

Fluentd does'nt store data on influxdbv2

I'm trying to store syslog on influxdbv2 using fluentd (td-agent). The configuration of td-agent is: @type influxdb2 url http://influxdb:8086 token my_token bucket …
C1ngh10
  • 13
  • 1
  • 7
0
votes
1 answer

fluentd: ignore_repeated_log_interval and ignore_same_log_interval

Trying to suppress same logs in fluentd by using fluentd system directives ignore_repeated_log_interval and ignore_same_log_interval, but no working action has been observed MY fluentd.conf ## conf file @type…
Nikhil Nayak
  • 1
0
votes
1 answer

error = undefined method `gsub' for nil:NilClass> td-agent

I am trying to mask password using @type record_transformer some of my application log statement don't have message due to this am getting below Error from td-agent log file is there any to over come this issue. How to ignore this if message field…
Aswath Murugan
  • 505
  • 2
  • 4
  • 12
0
votes
1 answer

Fluentd not able to upload logs to Elasticsearch

I have downloaded the elasticsearch & kibana on my Ubuntu18.04 machine. Both of them are running fine and I can access them. Below is the elasticsearch details: { "name" : "TX-G1-000", "cluster_name" : "elasticsearch", "cluster_uuid" :…
S Andrew
  • 5,592
  • 27
  • 115
  • 237
0
votes
0 answers

Fluentd change the source field value

Using fluentd I wanted to change the "source" field value but not sure how to go about it. Currently it is picking up the container IP address as "source" but I need that to be a self-defined name. Any ideas? EDIT: Things I tried: This is a Alpine…
SnazzyBootMan
  • 669
  • 2
  • 15
  • 30
0
votes
1 answer

td-agent does not validate google cloud service account credentials

Trying to configure fluentd output with td-agent and the fluent-google-cloud plugin. The plugin and all dependencies are loaded but fluentd is not outputting to google cloud logging and the td-agent log states error="Unable to read the credential…
Frosty
  • 1
  • 1
0
votes
1 answer

how to use fields from record transformer in S3 path fluentd-output-s3 plugin

we have below record transformer config in our fluentd pipeline: @type record_transformer enable_ruby true servername as1 hostname "#{Socket.gethostname}" project xyz env prod service…
chitender kumar
  • 394
  • 4
  • 21
0
votes
1 answer

data loss while sending from fluentd to aws kinesis firehose

We are using fluentd to send logs to aws kinesis firehose. We can see few records not sent to aws kinesis firehose every now and then. Here is our settings in fluentd. log_level info …
Dhirendra Rawal
  • 1
  • 1
0
votes
0 answers

Fluentd hide password or encrypt in configuration

Because of security reasons, we can't keep SQL authentication in plain text, is there a way to hide or encrypt passwords? @type sql @id output_sql host "sqlserverhost.aws_region.rds.amazonaws.com" database db_name adapter…
Mysterious288
  • 365
  • 5
  • 24
0
votes
1 answer

Fluentd sql output plugin configuration for auto incremented column

I have a fluentd configuration that pulls data from the file and pushes to the SQL server however there is a primary key with the auto-incremented column, so, in my fluentd configuration if I don't mention that column it throws an error saying that…
Mysterious288
  • 365
  • 5
  • 24
0
votes
0 answers

How to show missing index pattern data in EFK

I have 15 log files which are connecting to some services and saving data. I am uploading the data for these log files in elastic search using td-agent. I have created index pattern for each log files in td-agent-config file. I have log files with…
S Andrew
  • 5,592
  • 27
  • 115
  • 237
0
votes
0 answers

parse_audit_log not working with td-agent

I have installed the td-agent with version td-agent-4.1.0-1.el7.x86_64 on CentOS7. I'm not able to load the gem. # td-agent-gem install fluent-plugin-filter-parse-audit-log Fetching fluent-plugin-filter-parse-audit-log-0.1.2.gem Fetching…
Nix
  • 1
1 2
3
4 5