Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

logstash is a free and open source tool (apache 2.0 license) for managing events and logs. It can be used to collect and parse logs, and distribution to other indexing systems. It has a web interface for searching and drilling into the logs.

260 questions
1
vote
1 answer

Logstash continued grokking after _grokparsefailure

Goodday, (using logstash 1.4.2) I'm trying to do something like: filter { if type == "feed" { grok { match => [ "message", "%{COMBINEDAPACHELOGS}" ] add_tag => [ "grokked", "web" ] tag_on_failure => [ "notweb" ] } } if type…
Hevisko
  • 11
  • 3
1
vote
2 answers

How can I parse a human-readable byte count in Logstash?

I'm dealing with log files containing parts such as: 538,486K of 1,048,576K These represent memory use (Java heap space) rendered in a human-readable format. I would like to track those numbers in charts in Kibana. To do this I would like to…
Peter Becker
  • 180
  • 10
1
vote
1 answer

Logstash Grok Pattern

First of all I apologize for this,I am pretty bad in regular-expression and try to wrote custom pattern(as I am unable to find something in existing grok pattern or may be I am missing something) for parsing svn logs which is in the format of…
Prashant Lakhera
  • 713
  • 2
  • 10
  • 25
1
vote
0 answers

How parse a multiple objects serialized in log with logstash and grok

I'm parsing some log files with grok, but I have the following trace: [es.org.some.Object[ attr1=53212 searchTags=[love, rock] ]] [es.org.some.Object[ attr1=54612 searchTags=[love, peace, rock] ]] I use the following expression for…
Ganchix
  • 111
  • 2
1
vote
1 answer

Syslog connection to Logstash not resetting

We have a setup with one Logstash server (1.4.2, using embedded Elasticsearch) to receive logs from a number of other clients (via TCP and from their rsyslogd instances). One issue that now happens repeatedly is that one by one messages from hosts…
ferada
  • 111
  • 3
1
vote
2 answers

Elasticsearch index much larger than the actual size of the logs it indexed?

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like 2.5-3GB. Is there any reason for this and is there…
Christopher Bruce
  • 339
  • 2
  • 4
  • 14
1
vote
1 answer

Is there a well-known port for logstash-forwarder?

I'm setting up a few logstash-forwarder agents which will send logs to a central Logstash server, and need to choose a port for Logstash to listen on. After doing some searching, there doesn't seem to be a consensus about what this port number…
smitelli
  • 1,214
  • 1
  • 10
  • 16
1
vote
0 answers

Install logstash using the Deb file

I have installed Java 7 and Java 8 on my machine and java -version returns the correct version when I change the environment variables. When I go to install Logstash using the deb I receive the follow message: dpkg: dependency problems prevent…
Gabriel Matteson
  • 11
  • 1
1
vote
1 answer

How does Logstash internally work with messages/events?

I'm new to Logstash, and I'm trying to understand how inputs, filters and outputs work together. I get that there's a variety of inputs, filters and outputs, but what I don't get is how Logstash works with messages internally. Say I configure a…
sbrattla
  • 1,578
  • 4
  • 28
  • 52
1
vote
0 answers

logstash / kibana can't connect to instance

I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries. However.. today when I went to my logstash page, I see only thing message come up: Connection…
user99201
  • 287
  • 2
  • 8
  • 22
1
vote
1 answer

Logstash gradually losing sockets

Using logstash 1.4.2 pushing to an elastic search 1.1.1 instance on a separate server. Pulling log items from AWS SQS. Everything works fine the first several hours or so, but then the following begins to appear in the logstash log. :message=>"Error…
Todd Rowan
  • 111
  • 2
1
vote
1 answer

Indexing individual cookies with Logstash

I'm working on some analysis of Apache logs. Our logs happen to capture the cookies the browser communicates it has and we log it with the request. Right now I can import that long string fine, however I'm wondering if there is a way to have an…
BDM
  • 141
  • 5
1
vote
1 answer

Statistics of logstash memory consumption

I want to install it on a really small server and I want to see if it will be capable of running it with some dignity. So, does anyone have any memory and CPU consumption statistics of running logstash? I understand it depends a lot on the usage,…
ipinak
  • 135
  • 6
1
vote
2 answers

Logstash not reading logs unless world-readable

I recently set up Logstash with Kibana and Elasticsearch, but am having a problem whereby it's not reading logs unless the directory permissions are world read/execute (and the files at least world read). I've installed version 1.4.2 from the RPM,…
RCross
  • 469
  • 2
  • 6
  • 19
1
vote
1 answer

What software can I use to retrieve log files from multiple remote servers via SSH?

I know I can do this using scripts, but I'd like to know if there is such a thing as a "ready made" software which lets me configure the host (and file paths) which I'd like to fetch (at regular intervals) from? I'm planning to set up LogStash to…
sbrattla
  • 1,578
  • 4
  • 28
  • 52
1 2 3
…
17 18