Stack Overflow
Stack Overflow

Questions tagged [td-agent]

Please use this tag for question regarding td-agent, a data collection daemon. https://support.treasuredata.com/hc/en-us/articles/360000687108-Overview-of-Server-Side-Agent-td-agent-

Arm Treasure Data provides Server-Side Agent called Treasure Agent (td-agent), to collect server-side logs and events. You can continuously import data using td-agent.

71 questions
0
votes
0 answers

log4j.xml sends same logs to file and td-agent in different way

How come same logs processed in two ways? The logs which have which are saved as json in a file are stopped by td-agent saying pattern (which is json) not match This is my log4j.xml configuration for logs which saves (see below first BOLD comment)…
arun kushwaha
  • 36
  • 1
  • 6
0
votes
1 answer

Error using FluentD fluent-plugin-sql plugin

I am trying to use FluentD to get data from SQL Server via fluent-plugin-sql. I have installed tiny_tds and activerecord-sqlserver-adapter gems and my config looks like this: @type sql @id output_sql host…
rh979
  • 657
  • 1
  • 5
  • 13
0
votes
0 answers

Fluentd element emitts message record

We have a td_agent.conf file with the following tag: #this filter is used for C API which remove "[stdout]" from log #if CLOG Unified Logging C API won't be used, this filter can be removed @type parser format…
Omer
  • 15
  • 4
0
votes
0 answers

Convert logstash filter to fluentd

I'm really new to fluentd configurations and need help to convert this logstash config to fluentd to get started filter { if [syslog5424_host] =~ /apilog/ { if [syslog5424_msg] =~ /\"ApplicationType\"\:\"API\"/ { json { source =>…
Sam
  • 471
  • 7
  • 24
0
votes
0 answers

Fluentd - how to access logs in user home directory

I have been install fluentd on my company servers to transfer the logs to Newrelic. I'm using Fluentd because the disto on the server is Amazon Linux 1 and newrelic agent doesnt support log fowarding in that version(if you wondered). The logs is…
Tamir Ohana
  • 133
  • 1
  • 1
  • 6
0
votes
0 answers

Logs are not sent to elasticsearch using fluentd. The log is written in one line

The Java module writes a single-line log: {"timestamp":"2020-09-29 10:46:18.761","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}{"timestamp":"2020-09-29…
Aleksandr Bryanskiy
  • 1
  • 1
0
votes
1 answer

EFK (Elasticsearch+Fluentd-(td-agent)+Kibana): Kibana not showing correct logs

I have installed EFK on redhat linux system(version 7.6). Stable distribution of fluentd, that is td-agent is used instead of fluentd. Path of log file(/mnt/Log/Startup.log) is configured in td-agent.conf file. But on kibana dashboard it displays…
Samra1810
  • 1
  • 1
  • 1
0
votes
1 answer

Convert Number date to a String Date Field in Fluentd For Storing into Elasticsearch

I'm new to fluentd and using it to ingest data to elasticsearch. I have a field called request_time which has value : 110820120501 as a number of the format [ddmmyyHHMMSS]. How can I convert this as a date field. This is what I've written but it…
Preyas
  • 244
  • 2
  • 7
0
votes
1 answer

Not able to Start td-agent as root user

I am new to td-agent conf .I want to start td-agent as root user,I modified user and group in the /etc/init.d/td-agent as root .Ultimately systemctl want to do start and stop the service .But still ps -ef |grep td-agent shows as td-agent…
Hariram K
  • 1
  • 1
0
votes
1 answer

Fluentd exclude filter works and doesn't work

I'm looking into fluentd to send Apache logs to an http output. I'm partly successful but I don't understand the grep filter it seems. Some things I put in there work and others don't, I don't really see a difference so I'm stumped.. Example…
user3675572
  • 23
  • 3
0
votes
0 answers

convert system time to utc and utc to system fails

I want to change time to system and utc of given log {"host":"sp-1","level":"INFO","log":{"classname":"common.server.hacluster.CSFHATopologyChangeHandlerMBean:93","message":"Finished processing CSF HA 'become standby'…
arun kushwaha
  • 36
  • 1
  • 6
0
votes
1 answer

How to prevent td-agent / fluentd from inserting undesirable metadata?

Please assist me in understanding how to prevent td-agent from inserting undesirable metadata. It transforms a record of the form JSONBLOB to TIMESTAMP LOGNAME JSONBLOB. I Only want the json, not the timestamp and logname. for example -- td-agent…
alansigudo
  • 1
  • 1
0
votes
1 answer

How to send haproxy logs to fluentd by td-agent?

I wanna send haproxy logs to fluentd/elasticsearch/kibana using td-agent, but I can't do it correctly I have installed EFK by dockers and it rules correctly. I have a haproxy with log type haproxy.tcp like this: haproxy[27508]: info 127.0.0.1:45111…
Alex Losa
  • 1
  • 3
0
votes
1 answer

td-agent does not work when fluent-plugin-google-cloud on GCP

I installed td-agent on GCP VM, checked that the service ran (through "systemctl status td-agent"). Then I ran "/usr/sbin/td-agent-gem install fluent-plugin-google-cloud" to install stackdriver output plugin. The installation succeeded but td-agent…
leonidnei
  • 223
  • 2
  • 10
0
votes
1 answer

How do I enrich my event stream with an environment variable in fluentd?

I have a source event stream which is coming from a log tail. I'm trying to enrich this by using the filter directive. Some searching mentioned using #{ENV['MYVAR']} would work but I am getting an empty string as the value. The filter is as…
Raghuveer
  • 1,737
  • 20
  • 27
1 2 3
4
5