Stack Overflow
Stack Overflow

Questions tagged [fluent-bit]

Fluent-Bit (not to be confused with Fluentd), is an open-source, light weight data collector which can act as both an aggregator and forwarder. It allows you to collect data/logs from different sources, unify and send them to multiple destinations.

Fluent bit links :

379 questions
1
vote
1 answer

Ruby regex for fluentbit

I need to create a regex expression for fluent bit This is what I tried ^(?[0-9-]+ [:0-9,]+)\s+\s+(?\w+)\s+-\s+(?.*) Input is 2022-07-20 15:21:31,994 - INFO - Moving to Dashboard Desired output: log_time: …
Ramzan Mahmood
  • 1,881
  • 2
  • 21
  • 46
1
vote
1 answer

how to enrich logs with cluster name fluentbit

I have been using fluentbit across multiple kubernetes clusters and sending data to a centralized elasticsearch cluster. Logs have a lot of metadata on them that helps up to pin point the origin of a given log. Its missing the one crucial…
Ojas Kale
  • 2,067
  • 2
  • 24
  • 39
1
vote
1 answer

Getting data of pod using binary

I'm trying to figure a way of building a configuration file/script that can help me retrieve the logs of the Kubernetes pod into elastic but using Binary fluent-bit only. I was managed to retrieve the logs from the pod into elastic using the…
DevFromI
  • 191
  • 1
  • 20
1
vote
1 answer

Opensearch throws 429 error when Fluentbit outputs log under heavy load

With the below fluentbit configuration we are getting errors from opensearch under heavy load. Http bulk requests to opensearch by fluentbit(respresenting 429 errors as spike) Fluentbit config: [INPUT] Name tail Tag …
shubham
  • 21
  • 3
1
vote
0 answers

ErrImagePull when helm installing fluentbit

I was trying to reinstall fluentbit on a minikube cluster running on my local machine by following the steps in https://docs.fluentbit.io/manual/installation/kubernetes#installing-with-helm-chart. I've been able to get fluentbit deployed in the past…
skarsky
  • 63
  • 1
  • 6
1
vote
1 answer

How to store fluent-bit logs into MongoDB

I am using the fluent-bit docker image in order to collect logs from my NGINX app. I've been doing some testing and it seems to be working ok. I run the fluent-bit image like this: docker run -p 127.0.0.1:24224:24224 fluent/fluent-bit:1.5…
Danubio
  • 93
  • 8
1
vote
0 answers

Any way to automatically generate documentation/schema of fluent-bit output messages?

I have multiple log messages from different services (different codebase/languages) that are sent to different outputs (syslog & http/json) using fluent-bit. I am trying to find ways to automatically generate documentation for format of those log…
ThatChrisGuy
  • 559
  • 2
  • 7
  • 23
1
vote
0 answers

Fluentbit pods stops after few seconds - need help resolving the issue

Fluentbit daemonset on GCP GKE stops after few seconds. There are no errors or warning in the debug logs. Any help would be appreciated. I have pasted the compressed logs below. Additionally the configuration has been added at the end. [2022/03/14…
Ali
  • 11
  • 2
1
vote
0 answers

Fluentbit - Sending one message to two outputs based on label

I've been trying to write new config for my fluentbit for a few days and I can't figure out how to write it with best performance result. Is there a better way to send many logs (multiline, cca 20 000/s-40 000/s,only memory conf) to two outputs…
Jane
  • 63
  • 1
  • 6
1
vote
1 answer

Fluent Bit Multiline logs issue

I'm trying to implement EFK stack (with Fluent Bit) in my k8s cluster. My log file I would like to parse sometimes is oneline and sometimes multiline: 2022-03-13 13:27:04 [-][-][-][error][craft\db\Connection::open] SQLSTATE[HY000] [2002]…
Murakami
  • 3,474
  • 7
  • 35
  • 89
1
vote
0 answers

Fluent Bit Java Multiline parser on docker container logs in Kubernetes cluster (via New Relic integration)

We're using New Relic Fluent Bit integration to send Kubernetes pod logs to New Relic. Some pods are running Java apps so we'd like to apply java multiline parsing. Unfortunately this fluent-bit conf catch logs but multiline java parsing added in a…
Stefano Lazzaro
  • 387
  • 1
  • 4
  • 22
1
vote
1 answer

How do I connect AWS ECS to ElasticSearch with Fluentbit?

I want to connect my AWS ECS with Elasticsearch. I have written a Pulumi script where I > shuold use the fluentbit docker component as a sidecar to my frontend and backend application components frontend and backend. The Pulumi script looks like…
stein korsveien
  • 1,047
  • 5
  • 13
  • 35
1
vote
2 answers

AWS EKS logging to CloudWatch - how to send logs only, without metrics?

I would like to forward the logs of select services running on my EKS cluster to CloudWatch for cluster-independent storage and better observability. Following the quickstart outlined at…
Toms Mikoss
  • 9,097
  • 10
  • 29
  • 41
1
vote
2 answers

How to create a lua script for fluentbit throttle by specific key

I would like to throttle logs per kubernetes service - in other other words apply throttle filter for specific tags or keys. Fluentd had a group_key attribute that enables throttling at a service level but Fluent-bit does not. This is the…
fledgling
  • 991
  • 4
  • 25
  • 48
1
vote
1 answer

fluentbit get container_name from log file name to use as a custom field

I need to extract a part related to the container_name from the log file name and use it as a field in the fluentbit output. For example given a log file…
Alex Konkin
  • 618
  • 1
  • 7
  • 15
1 2 3
…
25 26