Questions tagged [logstash]

logstash is a tool for collecting and distributing log events.

logstash is a free and open source tool (apache 2.0 license) for managing events and logs. It can be used to collect and parse logs, and distribution to other indexing systems. It has a web interface for searching and drilling into the logs.

260 questions
1
vote
1 answer

Documents from Elasticsearch Cluster appearing twice in Grafana

I set up an Elasticsearch cluster with one dedicated master node, two master-eligible data nodes and one coordinating node. The number of replicas is set to one. There are two pipelines in Logstash, each receiving syslog messages from a firewall,…
Henry S.
  • 141
  • 4
1
vote
2 answers

Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis

I am using nginx with PHP-FPM and ELK as log file analysis. When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file. Problem is: Sometimes those error logs contains…
n.r.
  • 289
  • 1
  • 2
  • 10
1
vote
1 answer

Logstash's ElasticSearch Input Plugin should be used to output to ElasticSearch

We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch. This feature has been successfully collected by MongoDB Input Plugin in real time. However, the ElasticSearch Input Plugin does not start with the data that you…
junsung kang
  • 15
  • 1
  • 6
1
vote
1 answer

ELK Logs from application disappearing when grok rules are enabled

We have two instances of an application (same application, different database), lets call them app1 and app2. The route the logs take is: appX runs filebeat, which picks up the applications log file tags it with appX, and sends each entry to an…
GeoSword
  • 1,657
  • 12
  • 16
1
vote
1 answer

Retrieve pfSense/freeBSD logs with elk

I am attempting to centralize logs from different systems. I installed the Elastick Stack (Elasticsearch, Logstash, Kibana) and WAZUH OSSEC on one server (named elk). I have installed the OSSEC agent on three ubuntu server and I am able to check…
eli0T
  • 110
  • 11
1
vote
1 answer

Local Logstash failing to forward /var/log/syslog to central Logstash

I want my local instance of Logstash to forward syslog and all /var/log/* files to a remote, central instance of Logstash. The pipeline in /etc/logstash/conf/logstash-local.conf looks as follows: input { file { type => "syslog" path => […
0xF2
  • 187
  • 9
1
vote
1 answer

logstash: Running update via RPM stepped on my logstash.yml

Using RPM to update from logstash 5.4.1 to 5.6.4 Everything went great, but the command: yum localinstall logstash-5.6.4.rpm resulted in my /etc/logstash/logstash.yml getting overwritten by the config in the rpm. I don't recall seeing an RPM do…
Jonesome Reinstate Monica
  • 5,445
  • 10
  • 56
  • 82
1
vote
1 answer

Auto delete elasticsearch data older than 30 days

I have setup a ELK stack to collect logs at central server. It is working perfectly. But by default it is holding elasticsearch index/data permanently. We just want to maintain the data for 30Days. Please anyone point me how to delete indexs/data…
Sunil Bhoi
  • 189
  • 1
  • 2
  • 9
1
vote
1 answer

Logstash input log stream via port

Is there any way to read logstash raw input data that is forwarded via certain port? input settings are as below input { tcp { port => 1234 type => "test" } } and from output config... output { stdout { codec =>…
1
vote
1 answer

Logtash grok / multiline confusion

My real patterns are more complex but I have tried to boil the problem down to the core issue. Something I don't understand. Please try this out on http://grokconstructor.appspot.com/do/match I'm trying to match the following lines: Start-Date:…
spoovy
  • 354
  • 4
  • 15
1
vote
3 answers

Rsyslog not forwarding when running as service

I have 2 local VM's running CentOS 7 to simulate our production environment. 1 proxy server and 1 centralized (ELK) server. On the ELK server is running logstash to collect al syslogs from running instances. The proxy server is provisioned with…
Fjarlaegur
  • 111
  • 1
  • 5
1
vote
1 answer

Aggregating stats from events old and new

We'd like to feed CDN logs into Graphite and aggregate numbers found in there (rate of different HTTP status-codes, average response sizes, average cache-hit ratio, etc.) However, the logs are only uploaded to us occasionally and some times even out…
Mikhail T.
  • 2,338
  • 1
  • 24
  • 55
1
vote
1 answer

Logstash / Elasticsearch - Balancing index count with performance

We have a 4 data node ElasticSearch cluster: each node has 4 cores, 16GB RAM, and 160GB storage (the cluster has separate dedicated master nodes). The cluster is responsible for storing and presenting (with Kibana) a swath of different logs across…
J. Doe
  • 11
  • 1
1
vote
1 answer

Logstash behind the proxy doesn't work

I'm trying to launch few Logstash instances behind the proxy. I'm using nginx, but HAProxy is an option as well. worker_processes 4; events { worker_connections 1024; } http { upstream streams { server logstash_ip:5044; } …
dr11
  • 105
  • 2
  • 7
1
vote
2 answers

Logstash output to Azure blobstorage

I am currently importing IIS-logs into Logstash using Filebeat, and the Logstash is configured to output the documents into Elasticsearch. Now I also want to output my IIS logs to Azure storage (blob) for longtime-backup purposes, but I cannot find…
Andreas
  • 309
  • 1
  • 5
  • 17