Fluentd is open-source and distributed data collector, which receives logs in JSON format, buffers them, and sends them to other systems like Amazon S3, MongoDB, Hadoop, Loki(Grafana system) or other Fluentds.
Questions tagged [fluentd]
1336 questions
8
votes
1 answer
Log entries lost while using fluent-bit with kubernetes filter and elasticsearch output
From time to time we find that some logs are missing in the ES, while we are able to see them in Kubernetes.
Only problems in logs I was able to find, point out to a problem with the kubernetes parser with things like these in the fluent-bit…
carrotcakeslayer
- 809
- 2
- 9
- 33
8
votes
3 answers
How can I debug why Fluentd is not sending data to Elasticsearch?
There are 0 error messages when bringing up the Fluentd docker container, so it makes it hard to debug.
curl http://elasticsearch:9200/_cat/indices from the fluentd-container shows indices, but however doesn't show the fluentd-index.
docker logs…
Dennis
- 2,866
- 7
- 32
- 49
8
votes
2 answers
How to change fluentd config for GKE-managed logging agent?
I have a container cluster in Google Container Engine with Stackdriver logging agent enabled. It is correctly pulling stdout logs from my containers. Now I would like to change the fluentd config to specify a log parser so that the logs shown in the…
edwinbs
- 525
- 3
- 14
8
votes
1 answer
Fluentd - "incoming chunk is broken"
I run fluentd image on docker container. When I open TCP connection with telnet (or netcat) and send "message" string, following message occurs:
2017-01-24 10:22:00 +0000 [warn]: incoming chunk is broken: source="host: 192.168.99.1,
addr:…
Görkem Özer
- 504
- 5
- 13
8
votes
1 answer
docker compose oci runtime error, executable file not found in $PATH
I'm following this post:
http://eric-price.net/blog/centralized-logging-docker-aws-elasticsearch
This is what my docker-compose.yml looks like :
version: "2"
services:
fluentd:
image: fluent/fluentd:latest
ports:
- "24224:24224"
…
tyrell_c
- 503
- 3
- 10
- 24
8
votes
1 answer
Log level in Google Stackdriver Logging
Configured Google Stackdriver Logging in one of the GCE VM and everything works except Log Level . Have used the parameter log_level in the file
/etc/google-fluentd/config.d/tomcat.conf
as per provided in…
arvindwill
- 1,960
- 1
- 23
- 39
8
votes
1 answer
Graylog vs Fluentd
Out of available open source log management tools, I have come across these two but couldn't figure out which one to use. I tried searching articles for Graylog vs Fluentd but couldn't find.
Could someone suggest which one would be good for the…
baig62
- 169
- 1
- 7
8
votes
2 answers
fluentd loses milliseconds and now log messages are stored out of order in elasticsearch
I am using fluentd to centralize log messages in elasticsearch and view them with kibana. When I view log messages, messages that occured in the same second are out of order and the milliseconds in @timestamp is all…
David Wartell
- 726
- 1
- 8
- 13
7
votes
3 answers
EFK system is build on docker but fluentd can't start up
I want to build the efk logger system by docker compose.
Everything is setup, only fluentd has problem.
fluentd docker container logs
2022-02-15 02:06:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-02-15…
Henry Kao
- 88
- 1
- 6
7
votes
3 answers
Can fluent-bit parse multiple types of log lines from one file?
I have a fairly simple Apache deployment in k8s using fluent-bit v1.5 as the log forwarder. My setup is nearly identical to the one in the repo below. I'm running AWS EKS and outputting the logs to AWS ElasticSearch…
chakatz
- 312
- 1
- 2
- 10
7
votes
1 answer
multiline fluentd logs in kubernetes
I am new to fluentd. I have configured the basic fluentd setup I need and deployed this to my kubernetes cluster as a daemon set. I'm seeing logs shipped to my 3rd party logging solution. However I now want to deal with some logs that are coming in…
Matthew The Terrible
- 1,589
- 5
- 31
- 53
7
votes
3 answers
How to access from one container to another container stdout and stderr inside Kubernetes pod
I have a Pod with two containers.
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- name: my-container
image: google/my-container:v1
- name: third-party
image: google/third-party:v1
One container is my image and the…
nassi.harel
- 81
- 1
- 5
7
votes
1 answer
fluentd nested json parsing
I have logs like the following:
{
"log": {
"header": {
"key": "value",
"nested": "{\"key1\":\"value\",\"key2\":\"value\"}",
"dateTime": "2019-05-08T20:58:06+00:00"
},
"body": {
"path": "/request/path/",
…
shadfc
- 6,104
- 3
- 25
- 19
7
votes
2 answers
How to Send On Premises Kubernetes Logs to Stackdriver
Objective: Get some logging/monitoring on Googles
Stackdriver from a Kuberntes HA cluster
that is on premises, version 1.11.2.
I have been able to send logs to Elasticsearch using Fluentd Daemonset for
Kubernetes, but the
project is not supporting…
sidious
- 81
- 5
7
votes
2 answers
Fluentd pods running via daemonset getting terminated with warnings on google container engine
I am trying to run fluentd as a daemonset on kubernetes cluster (GKE). The config is getting parsed successfully, then the plugins are receiving shutdown signal with few warn messages. There are no error messages. I tried increasing the verbosity…
drishti ahuja
- 1,379
- 2
- 15
- 26