2

I'm attempting to index data in elasticsearch that includes dates with time zones.

My date mapping is:

"date": {
    "type": "date",
    "format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
}

I use the following code to index the data:

client.prepareIndex(INDEX, TYPE, id))
        .setSource(gson.toJson(object), XContentType.JSON)
        .setRefreshPolicy(RefreshPolicy.IMMEDIATE)
        .get();

I've created my own ZonedDateTime adapter as follows:

public class ZonedDateTimeAdapter implements JsonSerializer<ZonedDateTime> {

    public JsonElement serialize(ZonedDateTime date, Type typeOfSrc, JsonSerializationContext context) {
        DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
        return new JsonPrimitive(date.format(formatter));
    }

}

The resulting date is like 2005-01-01T13:35:50.596-0500. Which to me seems to match my format of yyyy-MM-dd'T'HH:mm:ss.SSSZ. However I receive an error stating:

Exception in thread "main" MapperParsingException[failed to parse [startDate]]; nested: IllegalArgumentException[Invalid format: "2005-01-01T13:35:50.596-0500" is malformed at ".596-0500"]

What's interesting is that if I change format in ZonedDateTimeAdapter to read yyyy-MM-dd'T'HH:mm:ss.000Z (forcing the second fraction to always be 000) I get a resulting date like 2005-01-01T13:31:06.000-0500. An object with this date is successfully indexed into elasticsearch.

Do you know why 2005-01-01T13:31:06.000-0500 index successfully but 2005-01-01T13:35:50.596-0500 does not? Aren't these the same format?

Ole V.V.
  • 81,772
  • 15
  • 137
  • 161
Gabriel
  • 624
  • 1
  • 7
  • 20

2 Answers2

2

Update: I did a quick test on my ES 5.2 and it worked fine.

$curl -XPUT localhost:9200/myindex1 -d '
{"mappings": {"type1": {"properties": {"field1": {
    "type": "date", 
    "format": "yyyy-MM-dd'\''T'\''HH:mm:ss.SSSZ"
}}}}}
'

$curl -XPUT localhost:9200/myindex1/type1/id1 -d '
    { "field1":"2005-01-01T13:35:50.000-0500" }
'

$curl -XPUT localhost:9200/myindex1/type1/id2 -d '
    { "field1":"2005-01-01T13:35:50.596-0500" }
'

To confirm we have the same mapping:

$curl localhost:9200/myindex1/type1/_mapping
{"myindex1":{"mappings":{"type1":{"properties":
    {"field1":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}}}}}}

From this thread on https://discuss.elastic.co

Currently, Elasticsearch is limited to millisecond precision, see this GH issue.

The post is from 2015 but the issue remains open. It looks like nanosecond precision is not yet supported.

kgf3JfUtW
  • 13,702
  • 10
  • 57
  • 80
  • 1
    Thanks! but isn't .001 a millisecond? Whereas .000000001 is a nanosecond? – Gabriel Dec 14 '17 at 20:30
  • You're absolutely correct. Apologies for my oversight - I naturally assumed you are doing micro/nanoseconds since I ran into this problem before. That said, I tried creating a mock index (using default mapping), inserting two documents, each having just one field. One has value `2005-01-01T13:31:06.000-0500` and one `2005-01-01T13:31:06.596-0500`. Both were inserted successfully. Let us see whether your mapping is causing the issue. – kgf3JfUtW Dec 14 '17 at 20:41
  • @Gabriel I tried on my local ES 5.2: Created same mapping, created index, inserted two documents with those values, everything worked fine. I updated my answer to reflect the logs. Is it possible for you to try a simple curl command? So that you could isolate the issue to either ES itself or the Java API. – kgf3JfUtW Dec 14 '17 at 20:59
0

I'm not sure if my "fix" will be appropriate for anyone seeing this problem. But it was getting worse. I couldn't parse out any dates that ended in 0. So yyyy-MM-dd'T'HH:mm:00 and yyyy-MM-dd'T'HH:00 would fail, but yyyy-MM-dd'T'HH:mm:01 or yyyy-MM-dd'T'HH:01 would be successful. I destroyed my entire elasticsearch database and upgraded to 6.1 and reindexed all of my data. I don't see the issue anymore.

Gabriel
  • 624
  • 1
  • 7
  • 20