Stack Overflow
Stack Overflow

Questions tagged [fluentd]

Fluentd is open-source and distributed data collector, which receives logs in JSON format, buffers them, and sends them to other systems like Amazon S3, MongoDB, Hadoop, Loki(Grafana system) or other Fluentds.

1336 questions
5
votes
1 answer

Fluentd log source format RegEX

I have logs of this format: 2015-02-25 18:33:06,975 INFO c.a.p.c.b.s.Monitor akka://application/user/daemons/monitor : 91 active threads, 4175691776 bytes used I came up to this regex: (?
kmkswamy
  • 55
  • 1
  • 5
5
votes
1 answer

Fluentd wildcard out_file with tag in file path

I'm just getting started with fluentd, but I would like to be able to set up a single output match rule, like so: type file path logs/ time_slice_format %Y%m%dT%H time_slice_wait 5m time_format…
taxilian
  • 14,229
  • 4
  • 34
  • 73
5
votes
2 answers

Python logging - overview of currently installed loggers/handlers

I am trying to set up a Fluentd log handler from a Django project using a logger from code: def get_fluentd_logger(name): import logging from fluent import handler logging.basicConfig(level=logging.DEBUG) logger =…
RickyA
  • 15,465
  • 5
  • 71
  • 95
4
votes
2 answers

fluentd elasticsearch plugin - The client is unable to verify that the server is Elasticsearch

I want to send some nginx logs from fluentd to elasticsearch , however, fluentd is unable to start due to following error message: The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the…
rugby2312
  • 1,056
  • 1
  • 10
  • 15
4
votes
1 answer

Fluent Bit 1.8+ and MULTILINE_PARSER

My goal is to collect logs from Java (Spring Boot) applications running on Bare Kubernetes. These logs are then translated into ES and visualized in Kibana. For these purposes I deployed Fleunt Bit 1.8.9 via Kubernetes 1.22. Since I use Containerd…
Maksim
  • 197
  • 2
  • 12
4
votes
1 answer

Fluentd - How to parse logs whose messages are JSON formatted parsed AND whose messages are in text; as is without getting lost due to parse error

I have certain log messages from certain services that are in JSON format; and then this fluentd filter is able to parse that properly. However with this; it discards all other logs from other components whose message field is not proper JSON. …
Alex Punnen
  • 5,287
  • 3
  • 59
  • 71
4
votes
0 answers

multi_format with multiple lines (logs)

Yoo! I'm new to fluentd and I've been messing around with it to work with GKE, and stepped upon one issue. Problem I'm using a filter to parse the containers log and I need different regex expressions so I added multi_format and it worked perfectly.…
rickerp
  • 103
  • 8
4
votes
1 answer

object tools is not a member of package scala

In my spark application, I'm trying to use fluentd-scala-logger for which I had to include an additional dependency in my build.sbt These are the 2 lines I added in my build.sbt: resolvers += "Apache Maven Central Repository" at…
Akshat Chaturvedi
  • 678
  • 1
  • 7
  • 15
4
votes
0 answers

Data not showing in Kibana Discovery Tab

I'm using Kibana 7.5.2 and Elastic search 7. My First approach: I'm sending log data and system data using fluentd and metricbeat respectively to my Kibana server. Everything working fine. I'm able to see data on the discovery page. But I had a…
Abhishek Kumar
  • 417
  • 7
  • 17
4
votes
0 answers

JSON parsing error in fluentd, throws pattern not matched always

I am trying to add a new source to my fluentd to parse a JSON file. The JSON file looks like this { "local": { "cmd_|-start_cmd_2_|-echo 'Starting to run second time'_|-run": { "comment": "Command \"echo…
Magic
  • 505
  • 2
  • 6
  • 19
4
votes
1 answer

Out of order logs between FluentD and FluentBit

I'm currently using JournalD + JournalBeat + Logstash as logging stack but I want to switch to using JournalD + FluentD. I tried using https://github.com/fluent-plugin-systemd/fluent-plugin-systemd as a input for FluentD but it results in a low…
Jonathan Erlich
  • 41
  • 3
4
votes
4 answers

Get the second match by regex

I want to get the second occurrence of the matching pattern (inside the brackets) by using a regex. Here is the text [2019-07-29 09:48:11,928] @hr.com [2] [AM] WARN I want to extract 2 from this text.I tried using (?
Charith_32
  • 145
  • 2
  • 10
4
votes
2 answers

Fluentd logs is full of backslash and kibana doesn't show k8s pods logs

I set up an EFK stack for gathering my different k8s pods logs based on this tutorial on a Microk8s single node cluster. Everything is up and working and I can connect kibanna to elasticsearch and see the indexes but in the discovery section of…
AVarf
  • 4,481
  • 9
  • 47
  • 74
4
votes
2 answers

Enabling Stackdriver Structured Logging on GKE

My k8s cluster on GKE has a fluentd pod setup by default (currently named fluentd-gcp-v3.1.0-....), but all the logs come out unstructured jammed into a single textPayload field. It would be much nicer to have my nginx (and all other) logs come out…
xref
  • 1,707
  • 5
  • 19
  • 41
4
votes
1 answer

How to replace a key value string with specific string in fluentd

I am new to the fluentd, I want to use record_modifier to replace the string, when specific string occurs in the key value. Example 1)input : {"message":"how are you"} output : {"message":"who are you"} when input key value having "how", i want…
user1514423
  • 41
  • 1
  • 2
1 2 3
…
88 89