Fluentd is open-source and distributed data collector, which receives logs in JSON format, buffers them, and sends them to other systems like Amazon S3, MongoDB, Hadoop, Loki(Grafana system) or other Fluentds.
I'm using Fluentd with Elasticsearch for logs from Kubernetes but I noticed that some JSON logs cannot be correctly indexed because JSON is stored as string.
Logs from kubectl logs look…
I'd like to parse ingress nginx logs using fluentd in Kubernetes. That was quite easy in Logstash, but I'm confused regarding fluentd syntax.
Right now I have the following rules:
type tail
path /var/log/containers/*.log
pos_file…
I want to monitor docker containers log from non-root user(td-agent) and on host server,
sudo chmod o+rx /var/lib/docker
sudo find /var/lib/docker/containers/ -type d -exec chmod o+rx {} \;
sudo find /var/lib/docker/containers/ -type f -exec chmod…
On this system I have calyptia-fluentd, and I'm trying to develop a custom filter plugin.
To start, I've created a gem with some initial code, and I have installed this gem, and have created a config file with uses the plugin.
However, starting…
Running Fluentd 3.5, which seems to give up after failing to flush the buffer. I can see there is a retry_forever parameter which is currently set to false, however I rather find out what is causing it and set a retry threshold to something…
Remove time and tag from fluentd output plugin stdout with json
Fluentd's output plugin produces output like:
2017-11-28 11:43:13.814351757 +0900 tag: {"field1":"value1","field2":"value2"}
So timestamp and tag are before the json. How can I remove…
I did install fluentd-kubernetes-daemonset on my kube workers on one did work without any error but on the other is throwing the following errors:
2018-12-07 03:48:33 +0000 [warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or…
I have a java application deployed in GCP .
Pointed that log file to Stackdriver logging using fluentd .
Java exception log was formatted in a separate line. So Stackdriver logging unable to capture it as error/ warning .
I need to format my…
I have a Fluentd instance, and I need it to send my logs matching the fv-back-* tags to Elasticsearch and Amazon S3.
Is there a way to configure Fluentd to send data to both of these outputs? Right now I can only send logs to one source using the…
The current setup I am working with is a Docker compose stack with multiple containers. These containers send their logging information to a logging container (inside the compose stack) running the Fluentd daemon. The configuration for Fluentd…
With Kibana I have managed to visualize continious requests in a linear chart by:
using a line chart
Y-axis: Aggregation: Count
X-axis: Date Histogram with field @timestamp
Split Lines: Sub Aggregation: Terms Field IP address
Now I would like to…
Here's my td-agent.conf file
@type http
port 8888
@type file
path /var/log/what.txt
But when I try to start the server using
sudo /etc/init.d/td-agent start
it gives the following error:…
I need some help for the following problem.
I have a spring boot application and I would like to configure a fluentd appender using logback.
I've created a file called logback.xml in my src/main/resources with the following content:
Im looking for a way to measure the elapsed time between to events using fluentd. Also it would be awesome to be able to merge two or more events.
Until now I was using logstash. There are two plugins for my usecase in logstash:
elapsed and…
To log to fluentd in java API, we need first to create a hashmap and then convert it to map, and then pass the map to the log function like:
Map data = new HashMap();
data.put("from", "userA");
…
