Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Its logo is a log with a moustache.

Logstash provides many APIs for collecting, parsing, and outputting logs. You can also create your personal API using the Ruby language.

Logstash is often mentioned as one of the components of the (, and ) and is maintained by Elastic.

Useful links:

8605 questions
18
votes
3 answers

Parsing nested JSON string in Logstash

I am logging to logstash,in json format, my logs have the following fields, each field is a string and the atts field is a stringified json (note: atts sub fields are different each time) here is an example: {"name":"bob","last":"builder",…
dina
  • 4,039
  • 6
  • 39
  • 67
18
votes
6 answers

Get the number of fields on an index

For optimization purposes, I am trying to cut down my total field count. However before I am going to do that I want to get an idea of how many fields I actually have. There doesn't seem to be any Information in the _stats endpoint and I can't quite…
Fairy
  • 3,592
  • 2
  • 27
  • 36
18
votes
4 answers

Logging from Java app to ELK without need for parsing logs

I want to send logs from a Java app to ElasticSearch, and the conventional approach seems to be to set up Logstash on the server running the app, and have logstash parse the log files (with regex...!) and load them into ElasticSearch. Is there a…
Rob
  • 5,512
  • 10
  • 41
  • 45
18
votes
3 answers

Logstash configuration reload

Is the Logstash configuration reloaded every time the agent is restarted? It doesn't seem to pick up my changes immediately (e.g. changed type value) I'm running it with an embedded elasticsearch v.0.90.7 on Windows 7 and Kibana 3. Thank you very…
Paul
  • 588
  • 1
  • 4
  • 16
17
votes
5 answers

Logstash if statement with regex example

Can anyone show me what an if statement with a regex looks like in logstash? My attempts: if [fieldname] =~ /^[0-9]*$/ if [fieldname] =~ "^[0-9]*$" Neither of which work. What I intend to do is to check if the "fieldname" contains an integer
Shawn Sim
  • 545
  • 1
  • 5
  • 17
17
votes
3 answers

multiple inputs on logstash jdbc

I am using logstash jdbc to keep the things syncd between mysql and elasticsearch. Its working fine for one table. But now I want to do it for multiple tables. Do I need to open multiple in terminal logstash agent -f…
Asim Zaidi
  • 27,016
  • 49
  • 132
  • 221
17
votes
4 answers

Decompose Logstash json message into fields

It have a logfile that stores event with a timestamp and a json message. For example: timestamp {"foo": 12, "bar": 13} I would like to decompose the keys (foo and bar) in the json part into fields in the Logstash output. I'm aware that I can set…
Maurits Rijk
  • 9,789
  • 2
  • 36
  • 53
16
votes
4 answers

logstash with java10 get error : Unrecognized VM option 'UseParNewGC'

I'm running logstash with Java 10 as follows: ./logstash -f std_std.conf but I get this error: Unrecognized VM option 'UseParNewGC' Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Thank…
kylekong
  • 199
  • 1
  • 1
  • 5
16
votes
1 answer

Logstash make a copy a nested field with mutate.add_field

I wanted to make a copy of a nested field in a Logstash filter but I can't figure out the correct syntax. Here is what I try: incorrect syntax: mutate { add_field => { "received_from" => %{beat.hostname} } } beat.hostname is not replaced mutate…
jmcollin92
  • 2,896
  • 6
  • 27
  • 49
15
votes
1 answer

which jdbc (connector/j) to install for mac os

I'm trying to download jdbc connector, but I cannot find mac os from the selection options from the link below: https://dev.mysql.com/downloads/connector/j/ Where can I download mysql connector for mac os? Or is it the case that jdbc connector is…
Sam Seo
  • 171
  • 1
  • 1
  • 6
15
votes
1 answer

Logstash multiple inputs multiple outputs

I'm trying to sync data between MySQL and Elasticsearch with Logstash. I set multiple jdbc inputs and multiple outputs to different elasticsearch indexes ... and something I am doing wrong because everything is going to the else block. Here is my…
JPG
  • 750
  • 2
  • 8
  • 23
15
votes
2 answers

Simplest way to get data from AWS mysql RDS to AWS Elasticsearch?

I have data in an AWS RDS, and I would like to pipe it over to an AWS ES instance, preferably updating once an hour, or similar. On my local machine, with a local mysql database and Elasticsearch database, it was easy to set this up using…
15
votes
1 answer

How to customize Rails log messages to JSON format

I need to customize log messages to a JSON format in my Rails app. To illustrate, currently the log messages my app produces look like this: I, [2015-04-24T11:52:06.612993 #90159] INFO -- : Started GET "/time_entries" for ::1 at 2015-04-24…
Conor Livingston
  • 905
  • 1
  • 8
  • 17
15
votes
2 answers

Logstash: Received an event that has a different character encoding

when using logstash I see an error like this one : Received an event that has a different character encoding than you configured. {:text=>"2014-06-22T11:49:57.832631+02:00 10.17.22.37 date=2014-06-22 time=11:49:55 device_id=LM150D9L23000422…
user3711857
  • 205
  • 2
  • 4
  • 9
15
votes
1 answer

Using kibana and mongodb together without elasticsearch

Is it possible to use kibana front-end along with a mongodb back-end without using elastic search? I'm using logstash to parse logs and store in mongodb and want to use kibana to display data? If not, are there any alternatives to implement…
Akk
  • 406
  • 1
  • 5
  • 17