Stack Overflow
Stack Overflow

Questions tagged [logstash-file]

Logstash File Input stream events from Files

221 questions
1
vote
2 answers

How to declare multiple output.logstash in single filebeat DaemonSet in kubernetes?

I have 2 applications (Application1, Application2) running on the Kubernetes cluster. I would like to collect the logs from my applications from outside of the Kubernetes cluster and save them in different directories(for eg:…
Kyaw Min Thu L
  • 537
  • 8
  • 26
1
vote
1 answer

How to split file name in logstash?

I am injecting a file from the s3 bucket to logstash, My file name is containing some information, I want to split the file name into multiple fields, so I can use them as separate fields. Please help me I am new with elk. input { s3 { bucket…
ramkrishna kushwaha
  • 380
  • 1
  • 6
  • 17
1
vote
1 answer

logstash 7.9.1 docker conatiner: file input is not working

I am trying to read a log file but is not working, it works when the logstash.conf is configure to listen in the port 5000 but from a file not works. I am using logstash version 7.9.1 from docker container and trying to sent the logs to Elastic…
user2994005
  • 57
  • 1
  • 9
1
vote
0 answers

Grokparsefailure and type problems in logstash configuration file

I have several problems with my configuration file. My goal is to parse three types of logs (for the moment). Here they are : [29/05/2020 07:41:51.354] - ih912865 - 10.107.119.121 - 93 - Transaction 7635 COMPLETED 318 ms wait time 3183…
Mathis
  • 25
  • 6
1
vote
0 answers

multiline field in csv (logstash)

I am trying to make the multiline field for csv file work in logstash. But the multiline for a field is not working. My log stash.conf content is: input { file { type => "normal" path => "/etc/logstash/*.csv" …
stack007 stack007
  • 51
  • 4
1
vote
0 answers

Logstash http input plugin accepting gzip file but how to detect only text format?

I am using the below conf for Logstash HTTP input plugin: I am trying to send .gz file and in header passing Content-Encoding: gzip. It is successfully doing its job but I am interested only in text format inside the gzip and not any other format.…
Agniv Gon
  • 11
  • 1
1
vote
0 answers

How to output logs to different Kafka topics under different keywords?

Here is my Logstash input log file: {"timestamp":"2019-10-18T16:37:53.137068+0800","flow_id":1072393101531249,"in_iface":"eno1","event_type":"event…
S4kur4
  • 11
  • 1
1
vote
0 answers

Cannot aggregate data from jdbc to output one single document

I'm trying to ingest data from jdbc into elastic search via logstash but i think the aggregate function is not working my pipeline code is input { jdbc { jdbc_driver_library => "/home/simrat/Downloads/mysql-connector-java-5.1.6.jar" …
Simrat Khanuja
  • 13
  • 4
1
vote
1 answer

After restart logstash sincedb is not cleaned (sincedb that generated in previous running)

I have an experience with the inode recycling problem, so I changed the sincedb_clean_after value like below. sincedb_clean_after => "6 hour" It is working fine. But 6 hours after restarting the logstash, the old record which is generated previous…
JERRY LEE
  • 11
  • 1
1
vote
2 answers

In java language, how to convert nano seconds value, for example 1568694302232954486 and 1568703521360049938 into a Java Date object?

I have many long numbers which look like 1568694302232954486 and 1568703521360049938, and I need to convert each of them into a Java Date object. How to implement the above requirement in Java 8? I also need transform the nanosecond value into…
fengnix
  • 85
  • 6
1
vote
1 answer

Json transformation in Logstash

I want to transform a json document from one structure to another. I've already tried to solve it with multiple workarounds but somehow I don't get my expected output. I have a json log file as following: { "consumer": { "created_at":…
SidD
  • 5,697
  • 4
  • 18
  • 30
1
vote
1 answer

Logstash "add_field" saves "%{...}" as value when key value pair missing in JSON

add_field => {"ExampleFieldName" => "%{[example][jsonNested1][jsonNested2]}"} My Logstash receives a JSON from Filebeat, which contains object example, which itself contains object jsonNested1, which contains a key value pair (with the key being…
ahjaokay
  • 11
  • 2
1
vote
0 answers

(Logstash) setting @timestamp value using timestamp from logs

I need to use my log's timestamp as @timestamp. I've tried different methods from the internet but none of the worked in my case input { s3 { bucket => "*" access_key_id => "*" secret_access_key => "*" prefix =>…
Artisan Ilustrado
  • 25
  • 3
1
vote
0 answers

How to get real-time timestamp?

I'm setting logstash for nginx error log, and output is ElasticSearch(ES)/Kibana. Reading log from files is good, but in ES, all of timestamp is 15:00:00 UTC. (I'm korean, and 15 UTC is midnight at korea.) It seems that logstash read log files at…
aibees
  • 11
  • 3
1
vote
1 answer

how to use elapsed filter- logstash

I am working in the Elapsed filter. I read the guide of Elapsed filter in logstash. then i made a sample config file and csv to test the working of Elapsed filter. But it seems to be not working. There is no change in uploading the data to ES. i…
Smack Alpha
  • 1,828
  • 1
  • 17
  • 37
1 2
3
…
14 15