Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Its logo is a log with a moustache.

Logstash provides many APIs for collecting, parsing, and outputting logs. You can also create your personal API using the Ruby language.

Logstash is often mentioned as one of the components of the (, and ) and is maintained by Elastic.

Useful links:

8605 questions
25
votes
1 answer

Logstash date filter not updating @timestamp with Apache timestamp

I'm trying to backfill some past Apache access log data with logstash, therefore I need the event @timestamp to be set to the date appearing in the log message. This is my current logstash configuration: input { tcp { type =>…
nuqqsa
  • 4,511
  • 1
  • 25
  • 30
25
votes
4 answers

How to debug the logstash file plugin

Is it possible to log actions of the logstash file plugin? (i.e. what files it tries to send, what errors happen, etc)
user626528
  • 13,999
  • 30
  • 78
  • 146
24
votes
2 answers

import csv into elasticsearch

I'm doing "elastic search getting started" tutorial. Unfortunatelly this tutorial doesn't cover first step which is importing csv database into elasticsearch. I googled to find solution but it doesn't work unfortunatelly. Here is what I want to…
adelura
  • 556
  • 1
  • 5
  • 12
24
votes
3 answers

Can I delete the message field from Logstash?

I have a basic Logstash -> Elasticsearch setup, and it turns out the 'message' field is not required after the logstash filter done its job - storing this raw message field to elasticsearch is only adding unnecessary data to storage imo. Can I…
James Jiang
  • 2,073
  • 6
  • 19
  • 25
24
votes
4 answers

List of SYNTAX for logstash's grok

The syntax for a grok pattern is %{SYNTAX:SEMANTIC}. How do i generate a list of all available SYNTAX keywords ? I know that I can use the grok debugger to discover patterns from text. But is there a list which i can scan through?
Cola
  • 2,097
  • 4
  • 24
  • 30
22
votes
1 answer

How to create multiple indexes in logstash.conf file?

I used the following piece of code to create an index in logstash.conf output { stdout {codec => rubydebug} elasticsearch { host => "localhost" protocol => "http" index => "trial_indexer" } } To…
kavya
  • 759
  • 4
  • 14
  • 31
22
votes
6 answers

Docker apps logging with Filebeat and Logstash

I have a set of dockerized applications scattered across multiple servers and trying to setup production-level centralized logging with ELK. I'm ok with the ELK part itself, but I'm a little confused about how to forward the logs to my…
Gianluca
  • 805
  • 1
  • 10
  • 20
22
votes
5 answers

Is it possible to have Centralised Logging for ElasticBeanstalk Docker apps?

We have custom Docker web app running in Elastic Beanstalk Docker container environment. Would like to have application logs be available for viewing outside. Without downloading through instances or AWS console. So far neither of solutions been…
Roman
  • 1,920
  • 2
  • 17
  • 17
21
votes
7 answers

Logstash configtest

I ran service logstash configtest but error given was: logstash: unrecognized service I was able to run logstash service individually but not with "configtest". In etc/logstash/conf.d/ I created logstash.conf file where consist of code as present…
Shann
  • 311
  • 1
  • 2
  • 9
21
votes
1 answer

Regex query syntax examples in kibana

I am trying to find the different kinds of syntax I can give in regex type of query through kibana, but I was not able to find any information on this. I am running logstash and elasticsearch in the backend. Any answer or example will be helpful.
ashbondu
  • 225
  • 1
  • 3
  • 7
20
votes
2 answers

Send spring boot logs directly to logstash with no file

So, I'm building a full cloud solution using kubernetes and spring boot. My spring boot application is deployed to a container and logs directly on the console. As containers are ephemerals I'd like to send logs also to a remote logstash server, so…
Phate
  • 6,066
  • 15
  • 73
  • 138
19
votes
4 answers

check if a string starts with number using regular expression

I am writing a filebeat configuration when I am matching if a line starts with a number like 03:32:33 ( a timestamp). I am currently doing it by- \d But its not getting recognised, is there anything else which I should do. I am not particularly…
Y0gesh Gupta
  • 2,184
  • 5
  • 40
  • 56
19
votes
1 answer

Logstash - remove deep field from json file

I have JSON file that I'm sending to ES through logstash. I would like to remove 1 field ( It's deep field ) in the JSON - ONLY if the value is NULL. Part of the JSON is: "input": { "startDate": "2015-05-27", "numberOfGuests": 1, …
Amit Daniel
  • 297
  • 1
  • 5
  • 16
19
votes
3 answers

How to stop logstash from creating a default mapping in ElasticSearch

I am using logstash to feed logs into ElasticSearch. I am configuring logstash output as: input { file { path => "/tmp/foo.log" codec => plain { format => "%{message}" } …
Prakash Shankor
  • 437
  • 1
  • 7
  • 16
19
votes
3 answers

Importing and updating data in Elasticsearch

We have an existing search function that involves data across multiple tables in SQL Server. This causes a heavy load on our DB, so I'm trying to find a better way to search through this data (it doesn't change very often). I have been working with…
Adrian J. Moreno
  • 14,350
  • 1
  • 37
  • 44