Stack Overflow
Stack Overflow

Questions tagged [filebeat]

Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis.

Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis. See more details

1457 questions
4
votes
2 answers

Grok/Oniguruma pattern to match first IP from X-Forwarded-For header

For this issue I'm trying to create a grok pattern, which matches the first IP from the X-Forwarded-For header in a nginx log. A log line typically looks like this: 68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET…
sepal
  • 43
  • 2
  • 5
4
votes
0 answers

filebeat custom processor to mutate the logevents

I was going through filebeat documentation where it has promising feature to process the input logevents- https://www.elastic.co/guide/en/beats/filebeat/current/configuration-processors.html with which I should be able to mutate my logevetns. The…
prayagupa
  • 30,204
  • 14
  • 155
  • 192
4
votes
1 answer

filebeat @timestamp not overwritten

I use filebeat to write logs to an elasticsearch server. My logs are in json format. Every line is a json string that looks like this {"@timestamp": "2017-04-11T07:52:480,230", "user_id": "1", "delay": 12} I want the @timestamp field from my logs…
LetsPlayYahtzee
  • 7,161
  • 12
  • 41
  • 65
4
votes
3 answers

Using filebeat with elasticsearch

I am not getting that how to run this filebeat in order to send output to elasticsearch. This is from the filebeat.yml file, - input_type: log # Paths that should be crawled and fetched. Glob based paths. paths: -…
Luv33preet
  • 1,686
  • 7
  • 33
  • 66
4
votes
3 answers

Filebeat with ELK stack running in Kubernetes does not capture pod name in logs

I am using the ELK stack (elasticsearch, logsash, kibana) for log processing and analysis in a Kubernetes (minikube) environment. To capture logs I am using filebeat. Logs are propagated successfully from filebeat through to elasticsearch and are…
Eric Broda
  • 6,701
  • 6
  • 48
  • 72
4
votes
1 answer

Filebeat > Logstash > ElasticSearch - Lumberjack Error

Trying to get Filebeat to work with logstash. Currently I am getting this error: 2016/11/14 04:54:27.721478 output.go:109: DBG output worker: publish 2047 events 2016/11/14 04:54:27.756650 sync.go:85: ERR Failed to publish events caused by:…
Seasonal_showers
  • 111
  • 2
  • 8
4
votes
2 answers

Pushing structured log data directly to elastic search with filebeat

I have configured filebeat to harvest my structured log output (greenfield project so each log entry is a JSON document in a pre-defined format) and publish it directly to ELS. Example log file excerpt (note that additional is free form, all other…
Myles McDonnell
  • 12,943
  • 17
  • 66
  • 116
4
votes
3 answers

ElasticSearch 5.0.0-aplha4 won't start without setting vm.max_map_count

I wish to update my ES version from 2.3 to 5.0.0-alpha4 to be able to use Ingest nodes and remove Logstash out of the question. But it seems ES 5.x version won't start without me setting vm.max_map_count to 262144. I don't want to set that value..I…
Mrunal Gosar
  • 4,595
  • 13
  • 48
  • 71
4
votes
1 answer

Log rotation in logstash

I am using file as input for logs in logstash . My log files are rotated daily so , I wanted to ask how can we configure file plugin of logstash so that it work with the files that are rotated daily. And adding to this, is log rotation available…
rresol
  • 323
  • 4
  • 20
4
votes
2 answers

NGINX log filter $upstream_response_time JSON ELK "-" parsefailure

I have my NGINX logs formated as JSON: log_format le_json '{ "@timestamp": "$time_iso8601", ' '"remote_addr": "$remote_addr", ' '"remote_user": "$remote_user", ' '"body_bytes_sent":…
Jesper Persson
  • 63
  • 1
  • 7
3
votes
1 answer

K8s - Metricbeat sending data but Filebeat doesn't to Elasticsearch

SOS I'm trying to deploy ELK stack on my Kubernetes**a ElasticSearch, Metricbeat, Filebeat and Kibana running on Kubernetes, but in Kibana there is no Filebeat index logs Kibana accessable: URL here Only MetricBeat index available I don't know…
IDIf Dsd
  • 33
  • 4
3
votes
1 answer

Correct way to use modules in Filebeat

I'm slightly confused about the correct way to use Filebeat's modules, whilst running Filebeat in a Docker container. It appears that the Developers prefer the modules.d method, however it's not clear to me of their exact intentions. Here is the…
Soggetto insubordinato
  • 41
  • 1
  • 7
3
votes
1 answer

Error while parsing the container logs using Filebeat

I have created a demonset of filebeat on azure kubernetes to collect the logs and ingest on Graylog but seems like there is an parsing error related to parsing which am not able to figure out. Kindly help me out. I got the code from…
Docgyan
  • 655
  • 2
  • 12
  • 29
3
votes
0 answers

multiple tokenizer using filebeat

I have multiple log files and I want to parse the message to get the correct timestamp. Here is the issue, I had logs that were ingested at later date because of which the service count hits are astronomical high around that date. But, since the…
harry123
  • 760
  • 1
  • 7
  • 22
3
votes
1 answer

elasticsearch filebeat mapper_parsing_exception when using decode_json_fields

I have ECK setup and im using filebeat to ship logs from Kubernetes to elasticsearch. Ive recently added decode_json_fields processor to my configuration, so that im able decode the json that is usually in the message field. -…
Kay
  • 17,906
  • 63
  • 162
  • 270
1 2 3
…
97 98