I'm trying to parse timestamp and store it as date
type in elasticsearch
. Storing it as string type is no problem, but I would want to query the logs based on this timestamp. Its format is not standard and isn't found built in logstash patterns.
this is how the date appears: 12/24/15 16:37:14:921 CST
and sometimes 1/24/15 16:37:14:921 CST
MM/dd/YY HH:mm:ss.SSS z
and M/dd/YY HH:mm:ss.SSS z
are both not working.
Here's the example of my logs
[12/24/15 16:37:14:921 CST] 00000094 ApplicationMg A WSVR0204I: Application: <some_project_name> Application build level: Unknown
[12/24/15 16:37:15:436 CST] 00000094 SibMessage W [:] CWSII0269W: The runtime accessed the user repository for <some module name> to populate missing unique name data when loading the authorization model for the bus.
[12/24/15 16:37:15:452 CST] 00000094 SibMessage I [:] CWSII0210I: The authorization policy for the <some module name> has been updated as a result of an administrative update.
[12/24/15 16:37:15:468 CST] 00000094 SibMessage W [:] CWSII0269W: The runtime accessed the user repository for<some module name> to populate missing unique name data when loading the authorization model for the bus.
and here's my logstash config file
input {
file {
path => "<path_to_logs_dir>/applevac.log"
codec => multiline {
pattern => "^\[%{TS}\]"
negate => true
what => "previous"
patterns_dir => "..\patterns"
}
start_position => "beginning"
sincedb_path => "since_db"
}
}
filter {
grok {
#includes custom patterns
match => { "message" => "(\[)%{TS}(\])(\s)+%{HEX:thread}(\s)+%{WORD:module}(\s)+%{ULETTER:letter}(\s)+%{EVERYTHING:eventMsg}" }
}
date {
match => ["eventtime","M/dd/YY HH:mm:ss:SSS z"]
}
}
output {
stdout { }
}
How do I store the timestamp appearing in between [] as date in elasticsearch. I'm able to store it as string by mapping my the custom pattern TS
based on that timestamp