Getting timestamp from WAF/Kinesis to Elasticsearch

Question

We're trying to find how to convert the timestamp coming from AWS WAF/Kinesis Firehose into Elasticsearch so that it's type is a date field. When the index mapping gets created it has the timestamp field but it's a type long, even though there seems to be an option for type epoch_millis (which is what the data is).

The Kibana interface says using the mapping api to change the field type but I can't seem to figure this out. The example here shows how to do this with creating new indexes but kinesis is creating/rotating the indexes so we seem to need a way to modify the default.

The field looks like this

  "timestamp": {
    "type": "long"
  },

and the full index definition looks like this but again these get created on a regular basis so we are trying to figure out how to change the default

  "waf-prod-2018-10-05": {
    "mappings": {
      "waf-prod": {
        "properties": {
          "action": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "formatVersion": {
            "type": "long"
          },
          "httpRequest": {
            "properties": {
              "args": { 
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "clientIp": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "country": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "headers": {
                "properties": {
                  "name": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "value": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              },
              "httpMethod": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "httpVersion": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "uri": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "httpSourceId": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "httpSourceName": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "nonTerminatingMatchingRules": {
            "properties": {
              "action": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ruleId": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "rateBasedRuleList": {
            "properties": {
              "limitKey": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "maxRateAllowed": {
                "type": "long"
              },
              "rateBasedRuleId": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "terminatingRuleId": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "terminatingRuleType": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "timestamp": {
            "type": "long"
          },
          "webaclId": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  },

Answer 1

The code of the template depends of your ES version. If you are using version 7.x. You need to remove the field (the mapping type field, in your case called by "waf-prod") before the properties and after the mappings. You can try to do this (for example, this is my configuration of ES 7.x):

PUT _template/template_waf-logs
{
  "order": 0,
  "index_patterns": [
    "aws-waf-logs-detected-requests-*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "0",
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "properties": {
      "httpRequest": {
        "properties": {
          "clientIp": {
            "type": "keyword",
            "fields": {
              "keyword": {
                "type": "ip"
              }
            }
          }
        }
      },
      "timestamp": {
        "type": "date",
        "format": "epoch_millis"
      }
    }
  }
}

1. Check de documentation of AWS here: https://aws.amazon.com/blogs/security/how-to-analyze-aws-waf-logs-using-amazon-elasticsearch-service/
2. Update your knowledge with the answer of the ES community here: https://discuss.elastic.co/t/root-mapping-definition-has-unsupported-parameters-when-creating-custom-index/240690

• Explanation to add the mapping type: https://discuss.elastic.co/t/common-error-root-mapping-definition-has-unsupported-parameters/63300/2
• Explained by ES version: https://stackoverflow.com/a/39290979/10746857

Answer 2

Just add the timestamp format to the mapping:

"timestamp": {
    "type": "date",
    "format": "epoch_millis"
}