I'm trying to parse a log file into elasticsearch through logstash.
I want to send the following log as single event(i.e. as a single document) into elasticsearch.
Here is my log file looks like:
######################################################
ETL Wrapper Initializing - 09/27/2018 06:33:57
######################################################
------------------------------------------------------
Wrapper Information - 09/27/2018 06:33:57
------------------------------------------------------
------------------------------------------------------
Reading Component Log Port Files - 09/27/2018 06:34:53
------------------------------------------------------
-- > -- > Found 3 files and only merge non-zero byte files
------------------------------------------------------
Renaming Reject Files - 09/27/2018 06:34:56
------------------------------------------------------
######################################################
Sending Notifications - 09/27/2018 06:34:56
######################################################
------------------------------------------------------
Setting Exit Status - 09/27/2018 06:34:56
------------------------------------------------------
######################################################
ETL Wrapper Finalizing - 09/27/2018 06:34:56
######################################################
------------------------------------------------------
Here is my logstash configuration:
input {
file {
path => "D:/logs/file.log"
start_position => "beginning"
}
}
filter{
grok {
match => {"message" => "ETL Wrapper Initializing - %{DATESTAMP:JobStartTime}"}
match => {"message" => "ETL Wrapper Finalizing - %{DATESTAMP:JobEndTime}"}
}
if "_grokparsefailure" in [tags]{
drop{}
}
if [message] =~ /^$/ {
drop { }
}
mutate{
remove_field => ["@version","host","message" ]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200/"
index => "success_index"
}
stdout { codec => rubydebug }
}
Output of above configuration:
{
"JobStartTime" => "09/27/2018 09:33:41",
"@timestamp" => 2018-12-05T10:55:44.698Z,
"path" => "D:/logs/file.log"
}
{
"JobEndTime" => "09/27/2018 09:34:16",
"@timestamp" => 2018-12-05T10:55:44.784Z,
"path" => "D:/logs/file.log"
}
My expected output:
{
"JobStartTime" => "09/27/2018 09:33:41",
"@timestamp" => 2018-12-05T10:55:44.698Z,
"path" => "D:/logs/file.log"
"JobEndTime" => "09/27/2018 09:34:16"
}
How can I merge "JobStartTime" and "JobEndTime" into single document?
Any help is appreciable.. Thanks in advance!