Logstash File Input stream events from Files
Questions tagged [logstash-file]
221 questions
2
votes
0 answers
logstash file input in read mode for gzip file is consuming very high memory
Currently i am processing gzip files in logstash using file input plugin. its consuming very high memory and keeps on restarting even after giving a high heap size. As of now on an avg we are processing 50 files per min and the planning to process…
Rakesh Patri
- 21
- 1
2
votes
1 answer
Elapsed Filter Plugin weird behaviour
I am using elapsed filter plugin for calculating time difference between multiple start/end events for a particular id.
if [StepName] == "Step1" and [StepStatus] == "start" {
mutate { add_tag => "Step1_start" }
} else if [StepName] == "Step1"…
Rohit Agrawal
- 51
- 7
2
votes
2 answers
How to add newline with gsub in logstash
I have a field 'message'.
When I do this : mutate {
update => {"message" => "%{message} \n"}
}
It converts message to: "2017-12-31 Error \\n" ["2017-12-31 Error" is my message value]
What I want is "2017-12-31…
S'chn T'gai Spock
- 1,203
- 18
- 16
2
votes
1 answer
logstash send log file to redis
I need to send my access log file to redis through logstash. My logstash is under /usr/share/logstash and the configure file is under /etc/logstash/conf.d/test.conf
test.conf:
input {
file {
path =>…
user1611237
- 123
- 1
- 5
- 14
2
votes
3 answers
Logstash 5.1.1 “bad URI(is not URI?)"
Error:
c: \ Program Files \ Logstash \ bin> logstash.bat -e 'input {stdin {}}
output {stdout {}}' An unexpected error occurred! : Error => bad URI
(is not URI?): File: // c: / Program Files / Logstash / confi g /
log4j2.properties,: backtrace…
Kalyani
- 31
- 1
- 2
2
votes
1 answer
Logstash Always Keeps One message in PipeLine
I am using Logstash to read and parse logs from a file and send them to a Rest based API. My shipper is working fine, but I am experiencing a strange behavior.
Version:
logstash-2.3.2
Problem:
When Logstash shipper parses the first log entry, it…
omer khalid
- 855
- 1
- 12
- 39
2
votes
1 answer
Logstash compare a field to a number
I'm searching a way to compare a Logstash field to a number in a conditional statement, but couldn't find anything in the documentation.
Something like this for example:
if [myfiels] => 1{
mutate {
add_field =>…
Vucomir Ianculov
- 23
- 1
- 3
2
votes
1 answer
How to print the logs of logstash execution
I have been trying to search this online, but couldn't get any lead.
Is there a way we can print logstash execution output to a log file? For example, I am using a jdbc plugin to read data as per sql_last_start. I want to know at what time the query…
Crickcoder
- 2,135
- 4
- 22
- 36
1
vote
1 answer
logstash with loki, grafana not picking all the kubernetes pod logs
I have setup running some log generator with loki and logstash. Grafana is able to identify the datasource and labels are picking, but the log generator logs are coming under grafana labels. What iam doing wrong here.
---
# Source:…
boycod3
- 5,033
- 11
- 58
- 87
1
vote
2 answers
Logstash how to give different index name to ElasticSearch based on file name
I have the following .conf file for Logstash:
input {
file {
path => "C:/elastic/logstash-8.3.2/config/*.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
separator => ";"
…
Yağız Can Aslan
- 102
- 12
1
vote
1 answer
1
vote
0 answers
Is Elastic search filter in log stash work in synchronous or asynchronous mode?
I am facing one issue in ELK when I continuous log and tried to update some fields in log stash configuration file using elastic search filter plugin its updating logs sometimes but sometimes not working, so is there and load issue or synchronous or…
Sandeep Thakur
- 13
- 4
1
vote
0 answers
Dump Elasticsearch documents into avro file using logstash
I want to store elasticsearch query results into avro files using logstash..As far as I know there file output logstash plugin which by default stores JSON documents.
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html
after…
Sagar Kadu
- 251
- 2
- 18
1
vote
0 answers
Kibana Visualize is not showing up all fields which are available on indexed data
I am new to ELK stack. I am trying to analyze and process java stack logs from my application to get useful information. In my case after starting the logstash, I can see my data under Discover tab on Kibana portal, but as soon I am trying to create…
Ankit Sachan
- 51
- 2
1
vote
0 answers
Logstash not reading log file but reads text file
The contents of my log file is same but it does not read the file in .log extension but is able to read it in .txt extension
Config for reading .txt file :-
input{
file{
path => ["/home/viggy/dummy/eppm.txt" ],
start_position =>…
Vighnesh Mahadik
- 23
- 5