0

I am using the reindex API in ES 5.4.1, and I need to convert a long field(which represents a date) to a date field. So the source index looks like 

"hits": {
      "total": 1,
      "max_score": 1,
      "hits": [
         {
            "_index": "twitter",
            "_type": "tweet",
            "_id": "1",
            "_score": 1,
            "_source": {
               "temp": 1496938873065,
               "message": "hello",
               "user": "joan"
            }
         }
      ]
   }

temp has to be converted to a date object.

I want to use a processor,

PUT _ingest/pipeline/p1
{
  "processors": [
      {
        "date" : {
        "field" : "temp",
        "target_field" : "updatedOn",
        "formats":["epoch_millis"],
        "timezone" : "Europe/Amsterdam"
      }
      }
    ]
}

But on trying to create this processor, I get an error 

{
   "error": {
      "root_cause": [
         {
            "type": "exception",
            "reason": "java.lang.IllegalArgumentException: Illegal pattern component: p",
            "header": {
               "processor_type": "date"
            }
         }
      ],
      "type": "exception",
      "reason": "java.lang.IllegalArgumentException: Illegal pattern component: p",
      "caused_by": {
         "type": "illegal_argument_exception",
         "reason": "Illegal pattern component: p"
      },
      "header": {
         "processor_type": "date"
      }
   },
   "status": 500
}

Any ideas?

user2689782
  • 747
  • 14
  • 31

1 Answers1

0

The formats parameter is wrong, you need to use UNIX_MS instead of epoch_millis, like this:

PUT _ingest/pipeline/p1
{
  "processors": [
      {
        "date" : {
        "field" : "temp",
        "target_field" : "updatedOn",
        "formats":["UNIX_MS"],
        "timezone" : "Europe/Amsterdam"
      }
      }
    ]
}
Val
  • 207,596
  • 13
  • 358
  • 360
  • Nice, now creating the pipeline worked! Thank you so much Val, you are awesome. However there is a new problem now. When I try to apply the pipeline using the reindex API, using - POST _reindex { "source": { "index": "twitter" }, "dest": { "index": "twitter", "pipeline": "p1" } } I now get an exception "field [temp] of type [java.lang.Long] cannot be cast to [java.lang.String]" – user2689782 Jun 12 '17 at 13:56
  • You cannot use the same index as source and destination. Your destination must be another index with the new mapping. – Val Jun 12 '17 at 14:08
  • I am using a different destination index – user2689782 Jun 12 '17 at 14:27
  • In your previous comment, they are both labelled `twitter` – Val Jun 12 '17 at 14:29
  • Sorry, that was a typo, I did use 2 differnet indices actually – user2689782 Jun 12 '17 at 15:18
  • I am actually trying to do it using java now, and script_fields. Still trying to get that working. – user2689782 Jun 12 '17 at 15:19