1

We're ingesting data to Elasticsearch through filebeat and hit a configuration problem.
I'm trying to specify a date format for a particular field (standard @timestamp field holds indexing time and we need an actual event time). So far, I was unable to do so - I tried fields.yml, separate json template file, specifying it inline in filebeat.yml. That last option is just a guess, I haven't found any example of this particular configuration combo.
What am I missing here? I was sure this should work:

filebeat.yml

 #rest of the file
 template:

      # Template name. By default the template name is filebeat.
      #name: "filebeat"

      # Path to template file
      path: "custom-template.json"

and in custom-template.json

{
  "mappings": {
    "doc": {
      "properties": {
        "eventTime": {
          "type": "date",
          "format": "YYYY-MM-dd HH:mm:ss.SSSS"
        }
      }
    }
  }
}

but it didn't.
We're using Filebeat version is 6.2.4 and Elasticsearch 6.x

chester89
  • 8,328
  • 17
  • 68
  • 113
  • Can you show a sample document that gets indexed and the current mapping that gets generated? – Val Aug 02 '18 at 04:40
  • sure, I'll do so in a bit – chester89 Aug 02 '18 at 06:00
  • we're indexing a string field that looks like `2018-08-01 09:03:35.4540`, and the mapping generated is https://gist.github.com/chester89/eed2f656242bd783ef4b27cb91aff142 (time field is the relevant part) – chester89 Aug 02 '18 at 10:04
  • why not using a recognized date format, such as `2018-08-01T09:03:35.4540` so that you don't have to care about this and let ES modify the mapping for you? – Val Aug 02 '18 at 10:05
  • @Val it's an option, sure, but that's more complicated for us to do then changing mapping. seems we'll need to change the log format – chester89 Aug 02 '18 at 10:08
  • The correct way to do it is described here: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html – Val Aug 02 '18 at 10:15
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/177280/discussion-between-chester89-and-val). – chester89 Aug 02 '18 at 11:57

1 Answers1

1

I couldn't get the Filebeat configuration to work. So in the end changed the time field format in our service and it worked instantly.
I found official Filebeat documentation to be lacking complete examples. May be that's just my problem

EDIT actually, it turns out you can specify a list of allowed formats in your mapping

chester89
  • 8,328
  • 17
  • 68
  • 113