0

I have an Elasticsearch index which uses the @timestamp field to store the date in a date field.

There are many records which are missing the @timestamp field, but have a timestamp field containing a unix timestamp. (Generated from PHP, so seconds, not milliseconds)

Note, the timestamp field is of date type, but numeric data seems to be stored there.

How can I use Painless script in a reindex and set @timestamp where it is missing, IF there is a numeric timestamp field with a unix timestamp?

Here's an example record that I would want to transform.

  {
    "_index": "my_log",
    "_type": "doc",
    "_id": "AWjEkbynNsX24NVXXmna",
    "_score": 1,
    "_source": {
      "name": null,
      "pid": "148651",
      "timestamp": 1549486104
    }
  },
Coder1
  • 13,139
  • 15
  • 59
  • 89

1 Answers1

0

Did you have a look at the ingest module of Elasticsearch??

https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html

Parses dates from fields, and then uses the date or timestamp as the timestamp for the document. By default, the date processor adds the parsed date as a new field called @timestamp. You can specify a different field by setting the target_field configuration parameter. Multiple date formats are supported as part of the same date processor definition. They will be used sequentially to attempt parsing the date field, in the same order they were defined as part of the processor definition.

It does exactly what you want :) In your reindex statement you can direct documents through this ingest processor.

If you need more help let me know, then I can jump behind a computer and help out :D

Byron Voorbach
  • 4,365
  • 5
  • 27
  • 35