Stack Overflow
Stack Overflow

Questions tagged [filebeat]

Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis.

Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis. See more details

1457 questions
2
votes
1 answer

elastic-agent is not collecting data

I have two elastic agents. One is running on my elasticsearch server and the other one on my other server. The agent on my elastic host got the fleet server intergration and the system intergration. The agent on my other hosts just has the system…
woldgrep
  • 97
  • 2
  • 12
2
votes
0 answers

Filebeat : Split the multiline json(array of json object) from message field to separate fields

I need to use filebeat to push my json data into elastic search, but I'm having trouble decoding my json fields into separate fields extracted from the message field. Filebeat version : 7.16.2 Filebeat.yml filebeat.inputs: - type: log enabled:…
max
  • 255
  • 2
  • 7
2
votes
0 answers

How to filter json data in filebeat yml file

While using kafka input, I want to output only when json data contains a specific string. I tried setting "include_lines" in filebeat.yml, but it was not filtered properly. When the filebit.yml setting is as follows and data-set1 and 2 are input,…
박배선
  • 37
  • 5
2
votes
1 answer

How to extract nested JSON object from kafka in flink table?

Here is a json format message in kafka. { "@timestamp": "2021-08-14T11:11:00.301Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "7.14.0" }, "message": { "k1": "v1", "k2":…
king
  • 300
  • 1
  • 13
2
votes
0 answers

How to parse log with json using Filebeat to store in Elasticsearch

The log file that I am using has logs in the following format: 2021-07-29 14:51:00,668 INFO audit [http-nio-8080-exec-4] {"user": "John", "country": "USA"} 2021-07-29 14:51:00,668 INFO audit [http-nio-8080-exec-4] {"user": "Jack", "country":…
Galet
  • 5,853
  • 21
  • 82
  • 148
2
votes
2 answers

How to properly monitor all ELK components with Prometheus?

I would like to monitor all ELK service running in our kubernetes clusters to be sure, that is still running properly. I am able to monitor Kibana portal via URL. ElasticSearch via Prometheus and his metrics (ES have some interested metrics to be…
Adavan
  • 63
  • 2
  • 6
2
votes
1 answer

Filebeat: How to export logs of specific pods

This is my filebeat config map. apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config namespace: kube-system labels: k8s-app: filebeat data: filebeat.yml: |- filebeat.inputs: - type: container paths: -…
Vikas Rathore
  • 8,242
  • 8
  • 35
  • 54
2
votes
0 answers

cannot load the dashboard to kibana

I tried various ways to fix this issue without any success. Step - edit the metricbeat YAML and edit the kibana host Index setup finished. Loading dashboards (Kibana must be running and reachable) Exiting: error connecting to Kibana: fail to…
IOT Test
  • 75
  • 1
  • 4
2
votes
2 answers

Can't disable field indexing on ElasticSearch AWS

I'm using AWS ElasticSearch 7.9. I'm pushing logs to it with Filebeat 7.12 installed on an Elastic Beanstalk AMI2. My log structure is as follow: { "timestamp": "2021-04-07T22:58:08.012Z", "label": "My API", "level": "info", "module":…
Tnargib
  • 73
  • 12
2
votes
1 answer

How to identify the bottleneck in slow Filebeat ingestion

I'm using Filebeat to monitor a single rapidly-rolling log file. At busy system times, the file rolls every few seconds. The logging configuration is set to roll once it gets to 10mb (Between 15k and 35k lines). I have a close_timeout of 5m set to…
beirtipol
  • 823
  • 5
  • 21
2
votes
1 answer

How to show mulitple servers logs in kibana separatly from each other

I have installed ELK on my Ubuntu server and install filebeat on remote server-A and server-B. I have configured Log-stash to receive data from filebeat and forward them to Elasticsearch. Both servers logs are showing in…
huzaifa224
  • 51
  • 2
2
votes
0 answers

How to send logs from multiple servers to ELK server

I have a server in which ELK installed, On other end i have 2 source servers which sending logs to ELK server through filebeat. But the issue is both server's logs showing on same page on kibana. which is too complicated to identify which log is…
huzaifa224
  • 51
  • 2
2
votes
1 answer

Sending filebeat outputs via http

I have a filebeat instance reading a log file, and there is a remote http server that needs to receive the log outputs via rest api calls. For now I'm sending filebeat outputs to logstash, and make logstash do some filtering and passing the log the…
ssssssleeeeee
  • 41
  • 4
2
votes
2 answers

Docker Filebeat Nginx Logs

I m using filebeat as docker and when ı point my nginx logs in filebeat.yml ı m not able to see nginx logs in kibana here is my filebeat.yml. I have elastichsearch and kibana containers ready to go. When I start filebeat container in the logs it…
kazasker
  • 21
  • 2
2
votes
1 answer

How can I add fields dynamically in Filebeats via the command line tool?

I am trying to add two dynamic fields in Filebeats by calling the command via Python. The fields themselves are populated after some processing is done so I cannot pre-populate it in a .yml file. Otherwise I know I can have the following code in my…
George Alfaro
  • 127
  • 7
1 2 3
…
97 98