Extract timestamp from log message

Question

I am trying to index log files to Elastic search. All the log entries are being indexed into a field named message. @timestamp field shows the time the entry was indexed and not the timestamp from log entry.

I created a ingest pipeline with grok processor to define the pattern of the log entry. I have tried several patterns and am unable to get this working, particularly because i am new to grok.

Log sample

2019-08-05 00:04:06 info [index.js]: Request: HTTP GET /
2019-08-05 00:04:06 error [error.js]: No authorization token was found

Ingest pipeline with grok & date processor

"description" : "Extracting date from log line"
, "processors": [
{
"grok": {
"field": "message",
"patterns": ["%{yyyy-mm-dd HH:mm:ss:logtime} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"]
},
"date": {
"field": "logtime",
"target_field": "@timestamp",
"formats": ["yyyy-mm-dd HH:mm:ss"]
}
}
]
}

All i want is the ability to extract the timestamp from the log message and everything else can be ignored or wildcarded or stored in just one variable like message. So essentially indexing the log file should index the timestamp from the log message and rest of the message can stay as text or string in one field, no need to parse rest of the message.

Any help would be appreciated.

Have a look at this https://stackoverflow.com/questions/48745506/logstash-grok-filter-config-for-php-monolog-multi-linestacktrace-logs — Sathishkumar Rakkiyasamy, Aug 15 '19 at 09:07
Thanks @SathishkumarRakkiasamy, i should have mentioned that Logstash is not being used. filebeat to ES is what i am trying to do. — rocky, Aug 15 '19 at 22:43

score 2 · Answer 1 · answered Aug 15 '19 at 08:49

2

use this as grok pattern:

%{TIMESTAMP_ISO8601:logtime} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}

use thes to set the timestamps :

date{
      match => ["logtime", "yyyy-MM-dd HH:mm:ss", "ISO8601"]
      timezone => "Europe/Berlin"
      target => "@timestamp"
    }

you may change the timezone to yours

answered Aug 15 '19 at 08:49

LinPy

16,987
4
43
57

Works with _simulate API but filebeat is unable to index the log with below error message. `[08/15/2019 22:31:27 > 7f74b1: ERR ] 2019-08-15T22:31:27.167Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure` In the filebeat elasticsearch output configuration, i have the following config `output.elasticsearch: hosts: ["host"] index: "index-name-%{+yyyy.MM}" pipeline: "redate"` If i remove the pipeline from the config then the logs are getting indexed, if pipeline entry is removed from filebeat config then i get those errors. Any thoughts? – rocky Aug 15 '19 at 22:37
meant "If i remove the pipeline from the config then the logs are getting indexed, if pipeline entry is r̶e̶m̶o̶v̶e̶d̶ ̶f̶r̶o̶m̶ **added to** filebeat config then i get those errors. Any thoughts? – rocky Aug 15 '19 at 22:51

score 1 · Answer 2 · answered Aug 15 '19 at 23:18

I made this below change and the log messages are getting indexed now. Although i do not understand how, appreciate if someone can shed some light on it

I had the pipeline: "pipelinename" setting in Elasticsearch output section of the filebeat config file. I moved that line to filebeat inputs section right under file path section, like so

filebeat.inputs: -type: log paths: - D:\home\site\wwwroot\logs*.log pipeline: "redate"

And the log messages are getting indexed now.

Extract timestamp from log message

2 Answers2