0

I have a bunch of log files that are named as 'XXXXXX_XX_yymmdd_hh:mm:ss.txt' - I need to include the date and time (separate fields) from the filename in fields that are added to Logstash.

Can anyone help?

Thanks

baudsp
  • 4,076
  • 1
  • 17
  • 35
AMC
  • 321
  • 1
  • 5
  • 11

1 Answers1

3

Use a grok filter to extract the date and time:

filter {
  grok {
    match => [
      "path",
      "^%{GREEDYDATA}/[^/]+_%{INT:date}_%{TIME:time}\.txt$"
    ]
  }
}

Depending on what goes instead of XXXXXX_XX you might prefer a stricter expression. Also, GREEDYDATA isn't very efficient. This might yield better performance:

filter {
  grok {
    match => [
      "path", "^(?:/[^/]+)+/[^/]+_%{INT:date}_%{TIME:time}\.txt$"
    ]
  }
}
Magnus Bäck
  • 11,381
  • 3
  • 47
  • 59
  • Thanks. Is there any way I can replace the @timestamp value with the time contained within each line of the log file? Each line starts "08:55:43.23" (no quotes). I'd like to take the date from the file name + time in the log file – AMC Feb 02 '15 at 15:40
  • Create a field (possibly with the [mutate filter](http://logstash.net/docs/1.4.2/filters/date)) that concatenates the date field picked up from the filename with the time from the log message and use the [date filter](http://logstash.net/docs/1.4.2/filters/date) to populate the `@timestamp` field. – Magnus Bäck Feb 02 '15 at 15:50
  • Can what you posted above be used in conjunction with what I currently have? i.e. can I put the match statement below this? grok { match => [ "message", "%{TIME:timestamp},%{WORD:agent},%{NUMBER:agentid},%{NUMBER:campaignid},%{CISCO_REASON:campaign_name},%{NUMBER:unknown1},%{NUMBER:unknown2},%{NUMBER:unknown3},%{NUMBER:unknown4},%{NUMBER:unknown5},%{NUMBER:unknown6},%{NUMBER:unknown7},%{NUMBER:unknown8},%{CISCO_REASON:Status}" ] } – AMC Feb 02 '15 at 15:53
  • Sure. Your two grok filters parse different fields so they're not in conflict and they don't have an order dependency. – Magnus Bäck Feb 02 '15 at 19:39
  • Should your suggestion work in Grok Debugger? I'm seeing 'No Matches' when using the following input "C:/inetpub/tslogs/ACTrace_ACTAT_150202_100034.txt" (no quotes) – AMC Feb 03 '15 at 12:38
  • The first example works with Windows paths. The second one doesn't (but could be made to work with a minor change). – Magnus Bäck Feb 03 '15 at 13:56
  • Sorry to sound like I need to be spoon fed, but are you able to help me with the syntax? I've tried #logstash to no avail. I need the date from the filename and time from each event in the log... – AMC Feb 03 '15 at 14:02
  • Just prepend the second expression with "C:" and it'll match Windows paths too. – Magnus Bäck Feb 04 '15 at 07:05