elastic re-index date format is too short

Question

Trying to re-index an index because the format of date field changed. The format changed from

 ...
 "start_date": {
      "type":   "date",
      "format": "yyyy-MM-dd HH:mm",
       "fields": {
        "keyword": { 
          "type": "keyword"
        }
      }
    }
  ...

to

 ...
 "start_date": {
      "type":   "date",
      "format": "yyyy-MM-dd HH:mm:ss",
       "fields": {
        "keyword": { 
          "type": "keyword"
        }
      }
    }
 ...

I try to re-index my index into a tmp index but it throws the following error:

"cause": {
    "type": "mapper_parsing_exception",
    "reason": "failed to parse [start_date]",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Invalid format: \"2019-01-30 13:03\" is too short"
    }
  },

So, now i have a big problem. How to change the format of my date field? Is there another option not to re-index?

score 1 · Accepted Answer · answered Mar 12 '19 at 14:57

Since the format changed, what you need to do is to append :00 to your date field in order to match the new format:

POST _reindex
{
  "source": {
    "index": "oldindex"
  },
  "dest": {
    "index": "newindex"
  },
  "script": {
    "source": "ctx._source.start_date = ctx._source.start_date + ':00';"
  }
}

glenacota · Answer 2 · 2019-03-12T16:29:12.753

I'd like to contribute to @Val's answer, in case you're looking for a more flexible solution. Please check the code below:

POST _reindex
{
  "source": { "index": "old-index" },
  "dest": { "index": "new-index" },
  "script": {
    "source": """
      def old_sf = new SimpleDateFormat("yyyy-MM-dd HH:mm");
      def old_dt = old_sf.parse(ctx._source.start_date);
      ctx._source.start_date = new SimpleDateFormat('yyyy-MM-dd HH:mm:ss').format(old_dt);
    """
  }
}

This way, it'll be easier to change your date format dramatically. E.g.,

POST _reindex
{
  "source": { "index": "old-index" },
  "dest": { "index": "new-index" },
  "script": {
    "source": """
      def old_sf = new SimpleDateFormat("yyyy-MM-dd HH:mm");
      def old_dt = old_sf.parse(ctx._source.start_date);
      ctx._source.start_date = new SimpleDateFormat('HH:mm:ss dd MMM yyyy').format(old_dt);
    """
  }
}

elastic re-index date format is too short

2 Answers2