Stack Overflow
Stack Overflow

Questions tagged [fluent-bit]

Fluent-Bit (not to be confused with Fluentd), is an open-source, light weight data collector which can act as both an aggregator and forwarder. It allows you to collect data/logs from different sources, unify and send them to multiple destinations.

Fluent bit links :

379 questions
2
votes
2 answers

Fluent-bit is not pushing data to amazon es service

I installed fluentbit using helm , Fluent-bit Verison is 1.13.11, fluentbit pods are running fine,still it is not able to send data to Amazon ES , below are the errors and yamls files. please provide any URL which can help me to install this…
Gaurav Agnihotri
  • 163
  • 1
  • 2
  • 7
2
votes
0 answers

How to get offset field or line number in tail plugin?

I deployed a fluent-bit to ouput the pod's logs to elasticsearch, then displayed with kibana. However, we have multiple logs at the same time. enter image description here The logs we searched from kibana may not be in the right order. So, can we…
julie
  • 43
  • 3
2
votes
2 answers

Is it possible to use a fluent-bit record's timestamp?

I'm trying to create a fluent-bit config which uses a record's timestamp to a custom key using a filter. Something like: [INPUT] Name tail Path /some/path ... [FILTER] Name record_modifier Match * Record fluenbit_orig_ts…
BugoK
  • 135
  • 1
  • 10
2
votes
2 answers

Fluent Bit Filter to convert Unix Epoch timestamp to human readable time format

I have a Java application in which I am using Log4j2 to print Logs in JSONLayout, here is a sample of the logs format: { "thread": "TopicStatusThreadPool-thread-1", "level": "INFO", "loggerName": "My Kafka Logger", "message": "Topic…
iamabhishek
  • 437
  • 6
  • 17
2
votes
2 answers

Fluentbit with Tomcat logs

I am using fluent-bit version 1.4.6 and I am trying to collect logs from a tomcat/logs folder, but I receive: [error] [input:tail:tail.0] read error, check permissions These files inside the logs folder are all "rw-r-----" (640). I tried to confirm…
voidcraft
  • 63
  • 1
  • 9
2
votes
2 answers

log key is not parsed by fluent-bit (both Docker containerized)

My regex parser doesn't seem to work. I'm guessing it has something to do with the logs coming from Docker and not being escaped. But I can't get it to work even if I include the Docker parser first. I've checked it in rubular:…
Quinten Scheppermans
  • 954
  • 1
  • 10
  • 29
2
votes
1 answer

Fluent-bit inside Spark Container

I am trying to run fluent-bit inside spark container so that spark driver container which is writing the logs in a file /var/log/sparkDriver.log controlled by spark log4j properties, can be read or consumed by fluent-bit. I know that running…
devnull
  • 161
  • 15
2
votes
1 answer

Activate kubernetes logs only for selected pods

I would like to annotate some kubernetes pods with an annotation like please_log_with_parser: myparser and have something like fluentbit to only process the logs of those pods with the given parser to our elastic search cluster. For fluentbit I'm…
michas
  • 25,361
  • 15
  • 76
  • 121
2
votes
1 answer

configuring fluentbit with docker

I am trying to configure the docker-compose file to utilize fluent-bit. In my docker-compose file I have the following config for fluentbit fluentbit: image: fluent/fluent-bit:1.0.4 command: /fluent-bit/bin/fluent-bit -c…
M.Holmes
  • 403
  • 1
  • 7
  • 22
2
votes
1 answer

FluentD: Forward vs TCP input

I am new to fluentd/fluentbit but could not understand the difference between forward and tcp inputs. In the examples, they emit the logs from the app using a forward input What exactly is the difference between logging via forward or tcp? They…
Cemre Mengü
  • 18,062
  • 27
  • 111
  • 169
2
votes
1 answer

Unable to collect all kubernetes container/pod logs via fluentd/elasticsearch

I'm new with fluentd/elasticsearch stack and I'm trying to deploy it on kubernetes. While I've managed to do that, I'm having a problem that not all pod/container logs are showing up on elasticsearch (I'm using Kibana for data visualisation). In…
Bakir Jusufbegovic
  • 2,806
  • 4
  • 32
  • 48
1
vote
1 answer

Shipping GKE container logs to memorystore

I have a GKE cluster which has some applications running. Those application log to stdout for performance reasons. I know GKE comes with fluentbit for logging. I was wondering if there is a way to get the application logs to a memorystore for redis…
saeed ektesabi
  • 13
  • 1
  • 6
1
vote
0 answers

Migrating from fluentd to fluent-bit for stackdriver logs on Container-Optimized OS

I'm attempting to upgrade from the legacy fluentd logging agent to fluent-bit on Container-Optimized OS, as recommended here. These are Java logs in JSON format, produced by logback, and configured using…
marthursson
  • 3,242
  • 1
  • 18
  • 28
1
vote
0 answers

How can I route ECS log to S3 with using AWS Firelens

I am making ECS Service withe Terraform. Now I have to route my ECS Service log to S3 my bucket. I looked it up and they told me to use AWS Firelens. But It dosen't work. Here is my setting. resource "aws_ecs_task_definition"…
Juzzn
  • 41
  • 2
1
vote
0 answers

How to convert JSON log to non JSON flat log using fluent bit?

I have an application log file, where each line is a JSON. A line looks like below. {"timestamp":"2023-04-25T03:40:03.180Z","level":"DEBUG","message":"\"some log…
Lahiru Chandima
  • 22,324
  • 22
  • 103
  • 179
1 2 3
…
25 26