Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Its logo is a log with a moustache.

Logstash provides many APIs for collecting, parsing, and outputting logs. You can also create your personal API using the Ruby language.

Logstash is often mentioned as one of the components of the (, and ) and is maintained by Elastic.

Useful links:

8605 questions
15
votes
3 answers

logstash - Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

Log stash is 100% a disaster for me. I am using LS 1.4.1 and ES 1.02 in the same machine. Here is how I start logstash indexer: /usr/local/share/logstash-1.4.1/bin/logstash -f /usr/local/share/logstash.indexer.config input { redis { host =>…
Tampa
  • 75,446
  • 119
  • 278
  • 425
14
votes
1 answer

Append specific MDC field into logstash log

I am trying add custom field into logstash appender in logback-spring.xml like that:
Denis Stephanov
  • 4,563
  • 24
  • 78
  • 174
14
votes
4 answers

Django server not sending logs to Logstash

I am using ELK stack for centralised logging from my Django server. My ELK stack is on a remote server and logstash.conf looks like this: input { tcp { port => 5959 codec => json } } output { elasticsearch { hosts =>…
Arpit Solanki
  • 9,567
  • 3
  • 41
  • 57
14
votes
1 answer

Elasticsearch: No handler for type [keyword] declared on field [hostname]

I get above Mapper Parsing Error on Elasticsearch when indexing log from filebeat. I tried both Filebeat -> Elasticserach and Filebeat -> Logstash -> Elasticsearch approach. I have followed their own documentations, I installed filebeat template as…
rayhan
  • 636
  • 3
  • 9
  • 26
14
votes
1 answer

How to setup ElasticSearch index structure with multiple entity bindings

recently I started working on ElasticSearch (ES) implementation into legacy e-commerce app written in PHP using MySQL. I am completely new to all this stuff and reading the docs is fine, yet I really need somebody with experience to advise me. From…
Jan Richter
  • 1,976
  • 4
  • 29
  • 49
14
votes
1 answer

Change ID in elasticsearch

I'm having trouble with ElasticSearch, how can I change id to another field in log file ?
katy
  • 181
  • 1
  • 3
  • 13
14
votes
2 answers

best way to send docker container logs to logstash

Let say I have Nginx running inside a container (docker). The access log and error logs are sent through STDOUT, in the Dockerfile : # forward request and error logs to docker log collector RUN ln -sf /dev/stdout /var/log/nginx/access.log RUN ln -sf…
kondor
  • 783
  • 1
  • 8
  • 22
14
votes
2 answers

have a grok filter create nested fields as a result

I have a drupal watchdog syslog file that I want to parse into essentially two nested fields, the syslog part and the message part so that I get this result syslogpart: { timestamp: "", host: "", ... }, messagepart:{ parsedfield1: "", …
Killerpixler
  • 4,200
  • 11
  • 42
  • 82
14
votes
2 answers

Understanding sincedb files from Logstash file input

When using the file input with Logstash, a sincedb file is written in order to keep track of the current position of monitored log files. How to understand its contents? Example of a sincedb file: 286105 0 19 20678374
Aldian
  • 2,592
  • 2
  • 27
  • 39
14
votes
1 answer

logstash _grokparsefailure issues

I'm having issues with grok parsing. In ElasticSearch/Kibana the lines I match come up with the tag _grokparsefailure. Here is my logstash config : input { file { type => logfile path => ["/var/log/mylog.log"] } } filter {…
lepolac
  • 263
  • 1
  • 3
  • 9
14
votes
1 answer

logstash to receive log from android? or is this elasticsearch?

I'm still a bit confused after reading documentation provided by logstash. I'm planning on writing an Android app, and I want to log the activity of the app. Logs will be sent over the network. is logstash not the right solution? because it needs to…
user2773013
  • 3,102
  • 8
  • 38
  • 58
14
votes
4 answers

Parse Apache2 Error logs with Grok for Logstash

Im trying to parse my apache2 error log and im having a bit of trouble.. It doesnt seem to be matching the filter. Im pretty sure the timestamp piece is wrong, but im not sure, and i cant really find any documentation to figure it out. Also, is…
Ascherer
  • 8,223
  • 3
  • 42
  • 60
14
votes
8 answers

Logstash with Elasticsearch

I am trying to connect Logstash with Elasticsearch but cannot get it working. Here is my logstash conf: input { stdin { type => "stdin-type" } file { type => "syslog-ng" # Wildcards work, here :) path => [ "/var/log/*.log",…
fth
  • 2,478
  • 2
  • 30
  • 44
13
votes
2 answers

Sync MongoDB with ElasticSearch

I want to sync my MongoDB data to ElasticSearch, I read a lot of posts talking about elasticsearch river plugin and mongo connector, but all of them are deprecated for mongo 4 and elasticsearch 7! As logstash is a proprietary software I would like…
Marco Sousa
  • 185
  • 1
  • 1
  • 9
13
votes
1 answer

How to decrease Logstash Memory Usage

I am using Logstash-5.6.5 (in Windows) running in a standalone system (no cloud or cluster). Planning to watch some log files and post it to locally run elasticsearch. But when checked the Logstash's memory usage, without a configuration to watch…