13

I am using Filebeat to ship log data from my local txt files into Elasticsearch, and I want to add some fields from the message line to the event - like timestamp and log level. For example here is one of my log lines:

2016-09-22 13:51:02,877 INFO 'start myservice service'

My question is: Can I do that by Filebeat -> Elasticsearch or must I go through Logstash?

A J
  • 2,508
  • 21
  • 26
Hadash
  • 228
  • 1
  • 2
  • 7
  • You could conceivably use the dissect processor within filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html – ndtreviv Sep 08 '20 at 13:54

1 Answers1

14

You can use Filebeat -> Elasticsearch if you make use of the Ingest Node feature in Elasticsearch 5.0. Otherwise, yes, you need to use Logstash.

In both cases you would use a grok filter to parse the message line into structured data. Also you'll want to use a date to parse and normalize the date.

A J
  • 2,508
  • 21
  • 26