Stack Overflow
Stack Overflow

Questions tagged [efk]

121 questions
1
vote
0 answers

Fluentd unable to send logs to Elasticsearch

Images being used here are: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch docker.elastic.co/elasticsearch/elasticsearch:7.5.1 Fluentd is able to place the logs in file-test.log file successfully but unable to place it in…
Shushruth Ad
  • 11
  • 1
1
vote
1 answer

Disable mapping for a specific field using an Index Template Elasticsearch 6.8

I have an EFK pipeline set up. Everyday a new index is created using the logstash-* prefix. Every time a new field is sent by Fluentd, the field is added to the index pattern logstash-*. I'm trying to create an index template that will disable…
Saad Malik
  • 99
  • 2
  • 7
1
vote
1 answer

Not able to see any data in discover tab when configuring index level permissions in Opensearch Dashboard

I have configured my Opensearch with some indices and able to see the logs in Discover tab using admin user. Now, the next step is to provide index level permissions to each user. Current Configuration: I have created one role with full cluster…
vivek
  • 187
  • 14
1
vote
0 answers

How to remove the unwanted characters from fluentd logs

Currently I am sending my Kubernetes logs to cloud watch using Fluentd, but when I check the logs in cloudwatch, the logs are having extra unicode characters. I tried different ways to and regexp to solve but no luck. Here is the sample how my log…
balu
  • 11
  • 2
1
vote
1 answer

Give multiple container names in logs pattern to scrape data from

I have setup EFK stack in K8s cluster. Currently fluentd is scraping logs from all the containers. I want it to only scrape logs from containers A, B, C and D. If I had some prefix with as A-app I could do something like…
confusedWarrior
  • 938
  • 2
  • 14
  • 30
1
vote
1 answer

OpenShift Logging: How do I make Kibana group lines of a stack trace into one record?

Context I have the Red Hat OpenShift Logging Operator installed in a cluster, explained on this page. This is basically an EFK stack (ElasticSearch, Fluentd, Kibana). Normally with Fluentd you can configure many plugins and hundreds of parameters,…
M455
  • 15
  • 5
1
vote
1 answer

How to parse a fluent-bit json $log. key?

I have the following log to be parsed: TID: [-1234] [] [2021-05-31 09:53:26,680] - Unique ID: Evento_Teste, Event: {"event":{"metaData":"blue"}} And below the configuration files that I created to deal with it. #td-agent-bit.conf [SERVICE] …
Kleyson Rios
  • 2,597
  • 5
  • 40
  • 65
1
vote
0 answers

Fluentd JSON logs truncate/splitting after 16385 characters- How to concate?

I have deployed Bitnami EFK stack on K8s environment: repository: bitnami/fluentd tag: 1.12.1-debian-10-r0 Currently, one of the modules/applications inside my namespaces are configured to generate JSON logs. I see logs in Kibana as JSON…
kishorK
  • 453
  • 2
  • 7
  • 16
1
vote
1 answer

error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster

I am getting this error from the fluentd pods and they keep restarting. I am running this on kuberentes v1.17.9-eks-4c6976. Not sure of what the cause is. Any help would be appreciated. …
David Essien
  • 1,463
  • 4
  • 22
  • 36
1
vote
1 answer

buffer flush took longer time than slow_flush_log_threshold

I am getting following Error in kibana- buffer flush took longer time than slow_flush_log_threshold. Don't know how to solve this. does anyone know how to solve this ? Things done so far- ES data nodes were filled 95%, so freed up some space, now…
Yogesh Jilhawar
  • 5,605
  • 8
  • 44
  • 59
1
vote
1 answer

How to setup Kibana (EFK stack) on Azure Kubernetes cluster using reverse proxy ingress?

Firstly wants to thank you for an amazing article above. I have a scenario I want to get some help with, So I have installed Elastic search, Filebeat and Kibana on AKS cluster. Now, since I am using Nginx Ingress controller to expose application on…
abh.bhardwaj
  • 11
  • 3
1
vote
1 answer

How i filter fluentD logs on kubernetes?

My kubernetes have liveness enable, and it log on application, like this: kubectl logs -n example-namespace example-app node-app ::ffff:127.0.0.1 - - [17/Sep/2020:14:12:19 +0000] "GET /docs HTTP/1.1" 301 175 ::ffff:127.0.0.1 - -…
Rafael Santos
  • 11
  • 4
1
vote
1 answer

How to configure long term retention of logs for EFK stack using S3?

What is the best way to configure a long term retention of logs in S3 for a kubernetes cluster with ElasticSearch, FluentD, and Kibana installed?
Josh Woodcock
  • 2,683
  • 1
  • 22
  • 29
1
vote
1 answer

How to configure discovery.type in elasticsearch?

I have a Kubernetes cluster with 2 nodes on which I'm running EFK stack (es + fluentd + kibana), flunetd is running on a daemonset, which means it runs two pods, one on each node. However, I want to know how to configure discovery.type for es which…
touati ahmed
  • 311
  • 8
  • 21
1
vote
1 answer

Getting AWS logs from S3 on kibana using EFK stack

I have an EFK stack running on EKS. How can I collect logs from an S3 bucket and see them through Kibana?
Capobar
  • 35
  • 6
1 2
3
…
8 9