Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

logstash is a free and open source tool (apache 2.0 license) for managing events and logs. It can be used to collect and parse logs, and distribution to other indexing systems. It has a web interface for searching and drilling into the logs.

260 questions
4
votes
3 answers

logstash failing to parse syslog input

I've configured logstash (v1.5.0), with a simple syslog input, as follows: input { syslog { type => syslog port => 5514 } } filter { kv {} } output { elasticsearch { cluster => "logs" host => "0.0.0.0" protocol =>…
Olly
  • 449
  • 1
  • 4
  • 11
4
votes
2 answers

Elasticsearch performance tuning

In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge. Logstash conf input { file { } } filter { mutate { …
Devaraj
  • 41
  • 1
  • 4
4
votes
1 answer

Logstash, Kibana and email alerts

I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails. I'd like to create a conditional rule so that let's say "X logfile has…
Gabriel
  • 141
  • 1
  • 1
  • 2
4
votes
4 answers

How to parse audit.log using logstash

I want to use logstash to collect a log file, and the format of the file was like this: type=USER_START msg=audit(1404170401.294:157): user pid=29228 uid=0 auid=0 ses=7972 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open…
txworking
  • 479
  • 2
  • 6
  • 15
4
votes
2 answers

Postfix - searching emails (logstash, greylog or other solution)

We are currently having ~100 servers and all of them are using remote syslog, so we have aggregated all logs on one server. The most questioned problem from our support team is: Has an email from .... to ... been delivered? I'd like to give to our…
Yarik Dot
  • 1,583
  • 12
  • 26
4
votes
4 answers

Nothing appearing in kibana dashboard

I have installed Logstash ElasticSearch Kibana on an EC2 instance. I can visit http://example.com:9200 Here I get { "status" : 200, "name" : "Aleta Ogord", "version" : { "number" : "1.1.0", "build_hash" :…
user2099762
  • 153
  • 2
  • 5
  • 18
4
votes
3 answers

Foward slash in kibana 3 query

I'm trying to add a query that will match a request that ends with a slash, like this one: n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like…
G Mawr
  • 173
  • 1
  • 3
  • 6
4
votes
1 answer

Logstash Date Has the Wrong Year?

I'm parsing Nginx logs into logstash with the following config: input { stdin { type => "nginx"}} filter { grok { type => nginx pattern => "%{COMBINEDAPACHELOG}" } date { type => nginx match => [ …
Brian Hicks
  • 185
  • 1
  • 8
4
votes
1 answer

Adding TTL to elasticsearch with logstash

I can't figure out how to apply this documentation http://www.elasticsearch.org/guide/reference/mapping/ttl-field/ to logstash. Specifically, i don't know which keys to use (instead of "tweet" etc). The goal is to have all logs sent to elasticsearch…
Maciej Swic
  • 290
  • 5
  • 19
4
votes
4 answers

CentOS centralised logging, syslogd, rsyslog, syslog-ng, logstash sender?

I'm trying to figure out the best way to setup a central place to store and interrogate server logs. syslog, Apache, MySQL etc. I've found a few different options but I'm not sure what would be best. I'm looking for something that is easy to install…
batfastad
  • 456
  • 1
  • 11
  • 22
3
votes
0 answers

Logstash losing connection to Elasticsearch nodes

We're running Logstash on a server which is pushing logs into an Elasticsearch cluster. In the Logstash logs we see it losing connection to the Elasticsearch servers with a connection reset error. We see this between all of our logstash and…
Nick Peirson
  • 31
  • 4
3
votes
2 answers

Windows Event Forwarding large scale with multiple domains

Let me start off explaining what I'm trying to do: We have an RMM tool installed on lots of Windows servers. It is able to send Windows Event Logs to a central store, but not in an efficient or reliable way. I'd like to use the native WEF on Windows…
floppyraid
  • 33
  • 3
3
votes
1 answer

Why is this exclude_lines in filebeat excluding all logs?

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-access log. This happens every second, and I'd like…
trueCamelType
  • 1,086
  • 5
  • 20
  • 42
3
votes
1 answer

Logstash S3 input plugin re-scanning all bucket objects

I am using the Logstash S3 Input plugin to process S3 access logs. The access logs are all stored in a single bucket, and there are thousands of them. I have set up the plugin to only include S3 objects with a certain prefix (based on date eg…
Garreth McDaid
  • 3,449
  • 1
  • 27
  • 42
3
votes
1 answer

elk stack error "unable to fetch mapping do you have indices matching the pattern"

i am trying to setup ELK stack with collectd on Ubuntu 16.04 LTS (so pretty much latest version of stack available) kibana is behind nginx proxy (followed this guide…
uberrebu
  • 503
  • 6
  • 17
  • 36
1 2
3
…
17 18