Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Its logo is a log with a moustache.

Logstash provides many APIs for collecting, parsing, and outputting logs. You can also create your personal API using the Ruby language.

Logstash is often mentioned as one of the components of the (, and ) and is maintained by Elastic.

Useful links:

8605 questions
13
votes
1 answer

Logstash config, "if string contains..."

So, let's assume that I have a portion of a log line that looks something like this: GET /restAPI/callMethod1/8675309 The GET matches a http method, and get's extracted, the remainder matches a URI, and also gets extracted. Now in the logstash…
A_Elric
  • 3,508
  • 13
  • 52
  • 85
13
votes
1 answer

Delete documents older than 30 days in elasticsearch

I want to delete documents in my elasticsearch index which are older than 30 days. Any ideas? EDIT: I want this to happen automatically - no document in my index shoudl be older than 30 days. So, in my opinion there are 2 options: either using…
ACKflow
  • 173
  • 1
  • 1
  • 8
13
votes
2 answers

Logstash filter remove_field for all fields except a specified list of fields

I am parsing a set of data into an ELK stack for some non-tech folks to view. As part of this, I want to remove all fields except a specific known subset of fields from the events before sending into ElasticSearch. I can explicitly specify each…
redstonemercury
  • 354
  • 1
  • 2
  • 9
13
votes
1 answer

Logstash agent not indexing anymore

I have a Logstash instance running as a service that reads from Redis and outputs to Elasticsearch. I just noticed there was nothing new in Elasticsearch for the last few days, but the Redis lists were increasing. Logstash log was filled with 2…
Antoine
  • 5,055
  • 11
  • 54
  • 82
13
votes
2 answers

logstash tab separator not escaping

I have tab separated data which I want to input into logstash. Here is my configuration file: input { file { path => "/*.csv" type => "testSet" start_position => "beginning" } } filter { csv { separator…
Noah Santacruz
  • 460
  • 1
  • 3
  • 18
13
votes
2 answers

Logstash close file descriptors?

BACKGROUND: We have rsyslog creating log files directories like: /var/log/rsyslog/SERVER-NAME/LOG-DATE/LOG-FILE-NAME So multiple servers are spilling out their logs of different dates to a central location. Now to read these logs and store them in…
Siddharth Trikha
  • 2,648
  • 8
  • 57
  • 101
13
votes
1 answer

How to do a time range search in Kibana

We are using the ELK for log aggregation. Is it possible to search for events that occured during a particular time range. Lets say I want to see all exceptions that occurred between 10am and 11am in last month. Is it possible to extract the time…
markus
  • 1,631
  • 2
  • 17
  • 31
13
votes
1 answer

How to generate @timestamp in logstash by combining two fields / columns of input csv

We have data that is coming from external sources as below in csv file: orderid,OrderDate,BusinessMinute,Quantity,Price 31874,01-01-2013,00:06,2,17.9 The data has date in one column and time in another column - I need to generate a time-stamp by…
Gopalakrishna Palem
  • 1,705
  • 1
  • 20
  • 34
13
votes
3 answers

Import JSON Files into Logstash + Elasticsearch + Kibana

So, I have a web platform that prints a JSON file per request containing some log data about that request. I can configure several rules about when should it log stuff, only at certain levels, etc... Now, I've been toying with the Logstash +…
Pedro M. Silva
  • 1,298
  • 2
  • 12
  • 23
13
votes
6 answers

How to reference environment variables in logstash configuration file?

Is it possible to reference environment variables in logstash configuration? In my case, i want to make my elasticsearch address configurable that i have set in the environment.
Julio Faerman
  • 13,228
  • 9
  • 57
  • 75
13
votes
3 answers

Logstash indexing JSON arrays

Logstash is awesome. I can send it JSON like this (multi-lined for readability): { "a": "one" "b": { "alpha":"awesome" } } And then query for that line in kibana using the search term b.alpha:awesome. Nice. However I now have a JSON log…
JP.
  • 5,507
  • 15
  • 59
  • 100
12
votes
4 answers

Difference between using Filebeat and Logstash to push log file to Elasticsearch

I am trying out the ELK to visualise my log file. I have tried different setups: Logstash file input plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html Logstash Beats input plugin…
user1589188
  • 5,316
  • 17
  • 67
  • 130
12
votes
2 answers

How to install logstash plugin with docker-compose?

I try to install logstash with a docker-compose but docker exited with code 0 just after Installation successful when I try to install a logstash plugin. The part of docker-compose file for logstash is: logstash: image:…
B3n
  • 536
  • 1
  • 3
  • 16
12
votes
2 answers

Filebeat vs Rsyslog for forwarding logs

I am currently using filebeat to forward logs to logstash and then to elasticsearch. Now, I am thinking about forwarding logs by rsyslog to logstash. The benefit of this would be that, I would not need to install and configure filebeat on every…
Luv33preet
  • 1,686
  • 7
  • 33
  • 66
12
votes
1 answer

Elasticsearch monitoring indices

I am new to ELK stack and playing around with it in a development environment. That's why I end up deleting an index (DELETE /index_name) and recreating multiple times. Deleting an index that I created works fine, but I notice that there are few…
javakurious
  • 487
  • 6
  • 13