1

I'm trying to parse timestamp and store it as date type in elasticsearch. Storing it as string type is no problem, but I would want to query the logs based on this timestamp. Its format is not standard and isn't found built in logstash patterns.

this is how the date appears: 12/24/15 16:37:14:921 CST

and sometimes 1/24/15 16:37:14:921 CST

MM/dd/YY HH:mm:ss.SSS z and M/dd/YY HH:mm:ss.SSS z are both not working.

Here's the example of my logs

[12/24/15 16:37:14:921 CST] 00000094 ApplicationMg A   WSVR0204I: Application: <some_project_name> Application build level: Unknown
[12/24/15 16:37:15:436 CST] 00000094 SibMessage    W   [:] CWSII0269W: The runtime accessed the user repository for <some module name> to populate missing unique name data when loading the authorization model for the bus.
[12/24/15 16:37:15:452 CST] 00000094 SibMessage    I   [:] CWSII0210I: The authorization policy for the <some module name> has been updated as a result of an administrative update.
[12/24/15 16:37:15:468 CST] 00000094 SibMessage    W   [:] CWSII0269W: The runtime accessed the user repository for<some module name> to populate missing unique name data when loading the authorization model for the bus.

and here's my logstash config file

input {
    file {
     path => "<path_to_logs_dir>/applevac.log"
     codec => multiline {
        pattern => "^\[%{TS}\]"
        negate => true
        what => "previous"
        patterns_dir => "..\patterns"
     }
     start_position => "beginning"
     sincedb_path => "since_db"
    }
}

filter {
    grok {
        #includes custom patterns
        match => { "message" => "(\[)%{TS}(\])(\s)+%{HEX:thread}(\s)+%{WORD:module}(\s)+%{ULETTER:letter}(\s)+%{EVERYTHING:eventMsg}" }
    }
    date {
        match => ["eventtime","M/dd/YY HH:mm:ss:SSS z"]
    }
}

output {
    stdout { }
}

How do I store the timestamp appearing in between [] as date in elasticsearch. I'm able to store it as string by mapping my the custom pattern TS based on that timestamp

Mudasir Ali
  • 128
  • 1
  • 11
  • 1
    Where does your `eventtime` field come from? I've tried both date formats and they work fine. – Val Feb 29 '16 at 16:04
  • Perhaps in your grok{} you meant %{TS:eventtime} and then to use both patterns in your date{} ? – Alain Collins Feb 29 '16 at 17:52
  • %{TS:eventTime} stores it as string. Not date. Because TS is not a standard date format accepted by elasticsearch. More over the month and day will appear in one digit if they're below 9. These logs come for a log4j custom appender, I can't modify that part of code – Mudasir Ali Feb 29 '16 at 18:54
  • 1
    As Val pointed out, you're referring to 'eventtime' in the date{} stanza, but didn't show how it was created. Since you're parsing the date out of the message using %{TS} in grok, it seems like putting that value into 'eventtime' there would make sense, and perhaps that's what's missing. Or not. – Alain Collins Feb 29 '16 at 19:15
  • Yeah it kinda does. Let me try that – Mudasir Ali Feb 29 '16 at 19:17
  • I had tried that, but what I didn't notice is that it was creating date out of @timestamp field instead of eventTime. Thanks for the help – Mudasir Ali Feb 29 '16 at 20:01

0 Answers0