Java.util.Timestamp => ElasticSearch long => C# NEST DateTime

Question

I use java.sql.Timestamp 2014-12-27 00:00:00

and ElasticSearch saves it as long 1419634800000 with mapping:

"EventDateLocal" : {
     "type" : "long"
}

and I want to read it in C# (via NEST) in System.DateTime

I tried

[Date(NumericResolution = NumericResolutionUnit.Milliseconds)]
public DateTime? EventDateLocal { get; set; }

but I got:

    Unhandled Exception: Elasticsearch.Net.UnexpectedElasticsearchClientException: U
    nexpected token parsing date. Expected String, got Integer.

....

Path 'hits.hits[0]._
        source.EventDateLocal', line 1, position 350. ---> Newtonsoft.Json.JsonSerializa
        tionException: Unexpected token parsing date. Expected String, got Integer. Path
         'hits.hits[0]._source.EventDateLocal', line 1, position 350.
           at Newtonsoft.Json.Converters.IsoDateTimeConverter.ReadJson(JsonReader reader
        , Type objectType, Object existingValue, JsonSerializer serializer)
           at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConv
        ertable(JsonConverter converter, JsonReader reader, Type objectType, Object exis
        tingValue)...

What should I put in annotation to get this effect automatically (long + unix => EventDate) :

foreach (var e in request.Documents)
{
    var start = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
    var date = start.AddMilliseconds(e.EventDateLocal.Value).ToLocalTime();
}

My NEST configuration is:

var client = new ElasticClient(new ConnectionSettings(new Uri(url))
      .DefaultIndex(index)
      .DefaultTypeNameInferrer(t => type)
);

If not with annotation is it possible somehow to tell NEST about the long?

client.Map<Event>(m => m
  .AutoMap()
  .Properties(ps => ps
      .Date(e => e
          .Name("EventDateLocal")
          .Format(????)
      )
  )
);

score 2 · Accepted Answer · answered Aug 29 '16 at 00:04

The default serializer used by NEST, Json.NET, does not handle milliseconds since epoch as a serialized form of a date (System.DateTime/System.DateTimeOffset). We can however apply our own converter for handling this, either just for these properties or globally.

First, define the converter

public class EpochDateTimeConverter : DateTimeConverterBase
{
    private static readonly DateTime Epoch = new DateTime(1970,1,1,0,0,0,DateTimeKind.Utc);

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        if (value == null)
        {
            writer.WriteNull();
            return;
        }

        long millisecondsSinceEpoch;
        if (value is DateTime)
        {
            millisecondsSinceEpoch = Convert.ToInt64((((DateTime)value).ToUniversalTime() - Epoch).TotalMilliseconds);
        }
        else
        {
            if (!(value is DateTimeOffset))
                throw new JsonSerializationException("Expected date object value.");
            millisecondsSinceEpoch = Convert.ToInt64((((DateTimeOffset)value).ToUniversalTime().UtcDateTime - Epoch).TotalMilliseconds);
        }
        writer.WriteValue(millisecondsSinceEpoch);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        if (reader.TokenType == JsonToken.Null)
        {
            if (objectType != typeof(DateTime?) && objectType != typeof(DateTimeOffset?))
                throw new JsonSerializationException($"Cannot convert null value to {objectType}");

            return null;
        }
        if (reader.TokenType == JsonToken.Integer || reader.TokenType == JsonToken.Float)
        {
            var millisecondsSinceEpoch = (long)reader.Value;
            var dateTime = Epoch.AddMilliseconds(millisecondsSinceEpoch);
            if (objectType == typeof(DateTime) || objectType == typeof(DateTime?))
            {
                return dateTime;
            }
            else
            {
                return new DateTimeOffset(dateTime);
            }
        }

        throw new JsonSerializationException($"Cannot convert to DateTime or DateTimeOffset from token type {reader.TokenType}");
    }
}

Now apply it to you POCO properties

public class MyDocument
{
    [JsonConverter(typeof(EpochDateTimeConverter))]
    public DateTime DateTime { get; set;}

    [JsonConverter(typeof(EpochDateTimeConverter))]
    public DateTimeOffset DateTimeOffset { get; set; }

    [JsonConverter(typeof(EpochDateTimeConverter))]
    public DateTime? NullableDateTime { get; set; }

    [JsonConverter(typeof(EpochDateTimeConverter))]
    public DateTimeOffset? NullableDateTimeOffset { get; set; }
}

and test it

var client = new ElasticClient();

var document = new MyDocument
{
    DateTime = new DateTime(2016, 8, 29, 9, 46, 0),
    NullableDateTime = null,
    DateTimeOffset = new DateTimeOffset(2016, 8, 29, 9, 46, 0, TimeSpan.Zero),
    NullableDateTimeOffset = new DateTimeOffset(2016, 8, 29, 9, 46, 0, TimeSpan.Zero),
};

client.Index(document);

serializes

{
  "dateTime": 1472427960000,
  "dateTimeOffset": 1472463960000,
  "nullableDateTimeOffset": 1472463960000
}

NEST doesn't send null values by default, but this can be changed by adding [JsonProperty(NullValueHandling = NullValueHandling.Include)] to the POCO property in question, if you really want to send null.

To test deserialization

var json = @"{
  ""dateTime"": 1472427960000,
  ""dateTimeOffset"": 1472463960000,
  ""nullableDateTimeOffset"": 1472463960000
}";

MyDocument deserializedDocument = null;
using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(json)))
    deserializedDocument = client.Serializer.Deserialize<MyDocument>(stream);

note that all DateTime and DateTimeOffset instances are in UTC so may want to modify to convert to local time.

Java.util.Timestamp => ElasticSearch long => C# NEST DateTime

1 Answers1