Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

logstash is a free and open source tool (apache 2.0 license) for managing events and logs. It can be used to collect and parse logs, and distribution to other indexing systems. It has a web interface for searching and drilling into the logs.

260 questions
1
vote
0 answers

Redis creates many events in logstash

I'm doing some testing for a POC with logstash. When using redis to buffer messages it appears they stay in redis and continually get added to elasticsearch as new events. For example, if I restart a service I'll see the same event many times when…
tweeks200
  • 351
  • 1
  • 3
  • 11
1
vote
0 answers

Visualize multiline ruby exceptions in kibana 4

I have setup the latest version of Kibana4 ElasticSearch stack. The logs are being pooled from remote app sources which are running on Ruby. I want to search for Multi Line exceptions created by ruby. Is there a way in kibana where we can search…
Swapnil jaiswal
  • 111
  • 5
1
vote
1 answer

rsyslog cannot send logs to logstash

Here is my rsyslog.conf (nothing in /etc/rsyslog.d/ folder): #### MODULES #### $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # provides kernel logging support (previously done by…
Iokanaan Iokan
  • 185
  • 1
  • 1
  • 8
1
vote
1 answer

Why can't the logstash syslog_pri filter see the priority in syslog messages

I'm running LS 2.0.0, and I've noticed an issue where the syslog_pri filter doesn't detect the priority at the start of my syslogs. My filter config is as follows: filter { if [type] == "relp" { syslog_pri { } grok { match => {…
Rumbles
  • 994
  • 1
  • 12
  • 28
1
vote
0 answers

Experiences with logfile monitoring

In an IT-integrator, I'm currently trying to evangelize the use of log file monitoring of hardware installations. The idea is to use a Elasticsearch/Logstash/Kibana stack (most flexibility, open source). However to convince management I'm looking…
stefan schbr
1
vote
1 answer

How to provision dashboards into kibana?

I am giving my developers the gift of kibana using the elasticsearch that is bundled with logstash. They will have kibana in their vagrant boxes. Goal: Install and configure kibana dashboards via scripts on startup. Problem: How to pre-load exported…
Dan Garthwaite
  • 2,962
  • 1
  • 19
  • 31
1
vote
1 answer

What is the scope of directives in logstash pipelines?

I'm setting up a generic Elasticsearch-Logstash-Kibana stack to deploy to a few of my clients. I'm trying to template some of the pipelines, so that we only need to deploy configs/pipelines as needed for each client. Logstash refers to input{...},…
Drew
  • 263
  • 3
  • 11
1
vote
0 answers

What is wrong with my logstash configuration?

Background I have two types of log files: output from an ETL process, and then output from a downstream processor. We call these "ETL" and "Processor" logs. The ETL logs are in their own folder under our logging directory, while the processor logs…
SeanKilleen
  • 1,083
  • 8
  • 25
  • 38
1
vote
1 answer

logstash mutate remove API Key from URL

I'm putting nginx logs into logstash and the api information is sent via get unfortunately. So there's 2 parts in logstash where API creditianals are stored. Here are examples message: 10.120.40.105 - - [29/Jul/2015:16:41:09 +0000] "PUT…
Mike
  • 22,310
  • 7
  • 56
  • 79
1
vote
0 answers

LogStash: iterate array using exec input plugin

I'm using the following logstash config file to request and parse xml response from a server: input { exec { command => "/usr/bin/curl -s http://%{host}/api/messages" interval => 10 type => "mytype" } } filter { if [type] ==…
Alex
  • 11
  • 3
1
vote
0 answers

Logstash sometimes duplicately populating fields when using multiline filter?

The issue I have seems to be similar to https://logstash.jira.com/browse/LOGSTASH-1892, but I'm running on Logstash-shipper 1.4.2 and this page claims the issue was fixed in 1.4.1. See a screenshot here: .
ujjain
  • 3,983
  • 16
  • 53
  • 91
1
vote
1 answer

How can I plot Apache HTTPd status counts in Graphite without Statsd using Logstash?

I'd like to send Apache HTTPd log statistics, such as 200 status counts to Graphite/Carbon. Logstash looks ideal but all the examples I've seen use Statsd to act as the state counter. This means spinning up a Statsd server (or enabling Statsd in…
Alastair McCormack
  • 2,184
  • 1
  • 15
  • 22
1
vote
0 answers

Unable to get structured data in logged messages even after adding mmpstrucdata and mmrfc5424addhmac?(RFC5424)

I am trying to log messages with structured data . But it is showing null value for structured data. I am working with rsyslog 8.9.0.Can someone tell me either i need to load some module or modify source to get structure data SD-IDs in logged…
StackUser
  • 803
  • 1
  • 7
  • 7
1
vote
2 answers

How can I parse an nginx msec field in logstash?

I'm trying to handle an nginx access log in logstash. In order to get millisecond accuracy for my timestamps, I'm using the $msec variable. This means that timestamps will be of the form 1430832725.814, where the integer part is a UNIX timestamp,…
leedm777
  • 305
  • 5
  • 9
1
vote
2 answers

Passing JSON application log to remote LogStash via NXLog on Windows

I have been trying to pass logs from a windows application which are already formatted in JSON to logstash via NXlog. When I have NXLOG send the file to Logstash, I get blasted with errors in the logstash.log: :message=>"An error occurred. Closing…
Noobixide
  • 126
  • 1
  • 13
1 2 3
…
17 18