2

I have been trying to ingest data into influx from a log file. The structure is as follows 

20171130-22:02:21.832 : something data work now
20171230-22:02:22.843 : something data2
20171231-22:02:23.232 : something data3

I have used this pattern 

[inputs.logparser.grok]
        #patterns = ['%{ts-"20060102-15:04:05.999":timestamp}%{GREEDYDATA:random_data}']
        patterns = ['(?<timestamp>\d{8}-\d{2}:\d{2}:\d{2}\.\d+)\s*:\s*%{GREEDYDATA:random_data}']

Can I get help regarding formulating the pattern ?

phaigeim
  • 729
  • 13
  • 34

1 Answers1

2

You may use

(?<timestamp>\d{8}-\d{2}:\d{2}:\d{2}\.\d+)\s*:\s*%{GREEDYDATA:random_data}

Details

  • (?<timestamp>\d{8}-\d{2}:\d{2}:\d{2}\.\d+) - timestamp field pattern:
    • \d{8} - 9 digits
    • - - a hyphen
    • \d{2}:\d{2}:\d{2} - 2 digits, :, 2 digits, : and 2 digits
    • \.\d+ - a dot and 1+ digits
  • \s*:\s* - : enclosed with 0+ whitespace chars
  • %{GREEDYDATA:random_data} - a .*, anything up to the end of the line
Wiktor Stribiżew
  • 607,720
  • 39
  • 448
  • 563
  • Thanks a lot.. It says Error parsing log line: error parsing regexp: invalid or unsupported Perl syntax: `(?<` ... Can you help me with the syntax in order to write to influx successfully ? I am very new at this .. – phaigeim Dec 04 '17 at 12:22
  • Sorry, are you sure you are using the regex with Grok? Because Grok regex flavor supports named capturing groups, they are used for setting extracted field names. – Wiktor Stribiżew Dec 04 '17 at 12:23
  • I think yes. Here is the syntax that I am using [inputs.logparser.grok] #patterns = ['%{ts-"20060102-15:04:05.999":timestamp}%{GREEDYDATA:random_data}'] patterns = ['(?\d{8}-\d{2}:\d{2}:\d{2}\.\d+)\s*:\s*%{GREEDYDATA:random_data}'] – phaigeim Dec 04 '17 at 12:26
  • 1
    @phaigeim Please check [the docs](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html). I am sure there is some problem with how the pattern is used. I tested it at http://grokdebug.herokuapp.com/ and it works well. – Wiktor Stribiżew Dec 04 '17 at 12:30
  • Thanks again @Wiktor. I have been using this https://github.com/influxdata/telegraf/tree/master/plugins/inputs/logparser in order to construct the patterns. Somethings didn't work with telegraf while they worked at grokdebug.herokuapp.com .. Is it different ? – phaigeim Dec 04 '17 at 12:32
  • The document says -- The Telegraf grok parser uses a slightly modified version of logstash "grok" patterns, with the format %{[:][:]} – phaigeim Dec 04 '17 at 12:45
  • Try with `custom_patterns` – Wiktor Stribiżew Dec 04 '17 at 12:50
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/160426/discussion-between-phaigeim-and-wiktor-stribizew). – phaigeim Dec 04 '17 at 12:52
  • Did you solve the problem? in that case can you post how? – Óscar Andreu Aug 02 '18 at 12:34
  • @ÓscarAndreu Doesn't the above work? See https://regex101.com/r/4dDCcd/1 – Wiktor Stribiżew Aug 02 '18 at 12:37
  • @WiktorStribiżew I have a slighly different problem, I posted the question here: https://stackoverflow.com/questions/51655530/telegraf-tail-with-grok-pattern-error – Óscar Andreu Aug 02 '18 at 14:00