Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Its logo is a log with a moustache.

Logstash provides many APIs for collecting, parsing, and outputting logs. You can also create your personal API using the Ruby language.

Logstash is often mentioned as one of the components of the (, and ) and is maintained by Elastic.

Useful links:

8605 questions
37
votes
4 answers

Change default mapping of string to "not analyzed" in Elasticsearch

In my system, the insertion of data is always done through csv files via logstash. I never pre-define the mapping. But whenever I input a string it is always taken to be analyzed, as a result an entry like hello I am Sinha is split into…
37
votes
2 answers

Kibana returns "Connection Failed"

I am trying to run ElasticSearch with Kibana in Windows 2008 R2. I followed this article: Install-logstash-on-a-windows-server-with-kibana Step by step, but all I get is: Connection Failed Possibility #1: Your elasticsearch server is down or…
m0fo
  • 2,179
  • 6
  • 33
  • 43
37
votes
4 answers

How to to log JS errors from a client into kibana?

I have web application backed end in NodeJS and logstash/elasticsearch/kibana to handle system logs like (access_error.log, messages.log etc). Right now I need to record all JavaScript client side errors into kibana also. What is the best way to do…
Erik
  • 14,060
  • 49
  • 132
  • 218
36
votes
5 answers

What is the format of logstash config file

Does logstash use its own file syntax in config file? Is there any parser or validator for config file syntax? For anyone that does not use logstash but have idea about file formats here is a sample syntax: input { file { path =>…
kokeksibir
  • 1,725
  • 2
  • 18
  • 30
33
votes
1 answer

Logstash optional fields in logfile

I'm trying to parse a logfile using grok Each line of the logfile has fields separated by commas: 13,home,ABC,Get,,Private, Public,1.2.3 ecc... I'm using match like this: match => [ "message",…
alpa
  • 345
  • 1
  • 4
  • 6
32
votes
4 answers

Error: index_not_found_exception

I use ELK stack to analyze my log file. I have tested last week and everything works well. Today, I tested but I get this error when I typed "http://localhost:9200/iot_log/_count" (iot_log is my index…
Fariha
  • 497
  • 1
  • 5
  • 13
32
votes
3 answers

How do I match a newline in grok/logstash?

I have a remote machine that combines multiline events and sends them across the lumberjack protocol. What comes in is something that looks like this: { "message" => "2014-10-20T20:52:56.133+0000 host 2014-10-20 15:52:56,036 [ERROR …
Wayne Werner
  • 49,299
  • 29
  • 200
  • 290
30
votes
4 answers

Kibana Logstash ElasticSearch | Unindexed fields cannot be searched

I am exploring ELK stack and coming across an issue. I have generated logs, forwarded the logs to logstash, logs are in JSON format so they are pushed directly into ES with only JSON filter in Logstash config, connected and started Kibana pointing…
rohit12sh
  • 827
  • 2
  • 11
  • 24
29
votes
3 answers

Using Log4J with LogStash

I'm new to LogStash. I have some logs written from a Java application in Log4J. I'm in the process of trying to get those logs into ElasticSearch. For the life of me, I can't seem to get it to work consistently. Currently, I'm using the following…
user3469584
  • 613
  • 3
  • 10
  • 15
29
votes
2 answers

Logstash conditional to check if tag exists?

Is there any way in logstash to use a conditional to check if a specific tag exists? For example, grok { match => [ "message", "Some expression to match|%{GREEDYDATA:NOMATCHES}" ] if NOMATCHES exists Do something. How do I verify if NOMATCHES tag…
CodeRunner
  • 391
  • 2
  • 4
  • 9
28
votes
2 answers

Logstash date parsing as timestamp using the date filter

Well, after looking around quite a lot, I could not find a solution to my problem, as it "should" work, but obviously doesn't. I'm using on a Ubuntu 14.04 LTS machine Logstash 1.4.2-1-2-2c0f5a1, and I am receiving messages such as the following…
Josep
  • 563
  • 1
  • 5
  • 12
28
votes
2 answers

Logstash grok multiline message

My logs are formatted like this: 2014-06-19 02:26:05,556 INFO ok 2014-06-19 02:27:05,556 ERROR message:space exception at line 85 solution:increase space remove files There are 2 types of events: -log on one line like the…
user2443476
  • 1,935
  • 9
  • 37
  • 66
28
votes
2 answers

Logstash: how to add file name as a field?

I'm using Logstash + Elasticsearch + Kibana to have an overview of my Tomcat log files. For each log entry I need to know the name of the file from which it came. I'd like to add it as a field. Is there a way to do it? I've googled a little and I've…
machinery
  • 3,793
  • 4
  • 41
  • 52
28
votes
4 answers

How to process multiline log entry with logstash filter?

Background: I have a custom generated log file that has the following pattern : [2014-03-02 17:34:20] - 127.0.0.1|ERROR| E:\xampp\htdocs\test.php|123|subject|The error message goes here ; array ( 'create' => array ( 'key1' => 'value1', …
emonik
  • 1,540
  • 1
  • 17
  • 24
25
votes
3 answers

SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]

Recently our server was rebooted without correctly shutting down the Elastic Search / Kibana. After that reboot, both applications were running but no indices were getting created anymore. I checked logstash setup in debug mode and it is sending…
RayofCommand
  • 4,054
  • 17
  • 56
  • 92