Stack Overflow
Stack Overflow

Questions tagged [logstash-file]

Logstash File Input stream events from Files

221 questions
1
vote
0 answers

Enfore different order for reading files in logstash

I have multiple log files which are written in descending order. i.e contents of xyz.log.5 are written before contents of log xyz.log.4. The number of log files is unknown. So, I'm using a wildcard to read files input{ file{ path =>…
Yellowjacket
  • 548
  • 2
  • 7
  • 19
1
vote
1 answer

Generate random number/text in logstash

Is there any inbuilt way to generate random text (of certain length) or number (of certain range) in logstash input/filer plugins? or should we write one in ruby.
user3366706
  • 1,529
  • 3
  • 31
  • 54
1
vote
2 answers

Maximum line length supported by logstash?

What is the maximum character length does logstash can read as a single event from a file (single line input, NOT multiline input)? Also does logstash take specific number of spaces/tabs in between a line as newline?
user3366706
  • 1,529
  • 3
  • 31
  • 54
1
vote
0 answers

Logstash Simple File Input Configuration

I am new to Logstash, and I have been trying to make a simple .conf file to read logs from sample Log file. I have tried everything from making sincedb_path to $HOME/.sincedb to setting the start_path to "Beginning", but I can't seem to get the data…
CKHobbes
  • 11
  • 4
1
vote
0 answers

Read logs after a particular day?

BACKGROUND: We have rsyslog creating log files directories like: /var/log/rsyslog/SERVER-NAME/LOG-DATE/LOG-FILE-NAME So multiple servers are spilling out their logs of different dates to a central location. Now to read these logs and store them in…
Siddharth Trikha
  • 2,648
  • 8
  • 57
  • 101
1
vote
1 answer

Logstash: since_db not getting created

Was playing with the since_db option and it appears that the sincedb file isn't getting created. Below is my Logstash File configuration. I have verified that I can create the file manually so there is no permission issue. Would appreciate if anyone…
John C
  • 1,795
  • 4
  • 27
  • 42
0
votes
1 answer

Logstash config doesn't parse JSON file and not push data to index - log: {:count=>1,:running_pipelines=>[:main], :non_running_pipelines=>[]}

I generate a JSON file every 5 mins through a Python code and try to pushing data to Elastic, but the Logstash throw following message and doesn't push any data to Kibana. My pipeline: File --> Logstash --> Elastic --> Kibana JSON file output: { …
M.A.Murali
  • 9,988
  • 36
  • 105
  • 182
0
votes
1 answer

Logstash how to send data from a file in a loop?

As part of load testing, I have a couple of premade log files that simulate different scenarios. I'm trying to use Logstash to send the content of 1 file, and resend it again in a loop every time it reaches EOF. Is there a way for me to do that?
dorony
  • 1,008
  • 1
  • 14
  • 31
0
votes
0 answers

Why logstash gives filebeat metadata for kafka input plugin?

I'm running a logstash pipeline using kafka input plugin with following configuration kafka { bootstrap_servers => "xxxx" topics => ['topic1', 'topic2', 'topic3' ... , 'topic50'] auto_offset_reset => 'latest' group_id =>…
Jinal Kothari
  • 1
  • 2
0
votes
1 answer

What is the best way to extract request and response data from a log file using Logstash and regex?

I have application log file which contains the application requests and responses, the complete request and response looks like the below, I tried different patterns using RegEx but unfortunately without any luck, can some one suggest what should I…
Hamzeh Abu Ajamieh
  • 37
  • 4
0
votes
0 answers

Encountered a retryable error (will retry with exponential backoff) {:code=>400, :url=>"http://localhost:9200/_bulk", :content_length=>41360}

I need to create indexing of exiting postgres database in elasticsearch. For this purpose I have setup elasticsearch 7.17.4, kibana 7.17.4 and logstash 7.17.4 on my local machine. I have downloaded csv file of posts table from db that contain almost…
Manzoor.faisal
  • 51
  • 1
0
votes
0 answers

how to parse historic tomcat logs (txt) from logstash to elastic with specific pattern and historic timestamp index?

I have some historic tomcat access logs in this basic format - - [19/Dec/2022:00:00:05 +0100] "POST HTTP/1.1" 200 1321 I want to ship this log entries to ElasticSearch. My starting Logstash looks like this: input { file { path =>…
F.M.
  • 193
  • 2
  • 8
0
votes
1 answer

How to map array inside message in Logstash HTTP Output

I am using Logstash to update by query existing Elasticsearch documents with an additional field that contains aggregate values extracted from Potgresql table. I use elastichsearch output to load one index using document_id and http output to update…
Carlitoz
  • 1
  • 1
0
votes
0 answers

Remove complete event and replace with string in logstash

Working on a way to completely replace the event with the string in logstash filter. Input: { "a": "b", "c", "d" } Output "a:b-c:d" Tried using the ruby code. I'm able to form the pattern but how can i replace the output string with the original…
jeya kumar
  • 1
  • 1
0
votes
0 answers

Can not process text file locally using logstash input plugin

I want to process a text file locally and the output I want to save in a file as log/text. This is my code but this does not work. input { file { path => "C:/Users/USERNAME/Documents/Projects/test_data.txt" start_position =>…
AnonymousScientificUser
  • 146
  • 17
1 2 3
…
14 15