Elastic Search mapping of logstash @timestamp to strongly typed in .NET

Question

I use Serilog and the ElasticSearch Sink with AutoRegisterTemplate = true

That stores the following in ElasticSearch

{
"@timestamp": "2016-08-17T08:57:37.3487446+02:00",
"level": "Information",
"messageTemplate": "User login {UserId}",
"message": "User login ...,
"fields": {
  "UserId": "...",
  "ProcessId": 15568,
  "ThreadId": 14,
  "MachineName": "...",
  "EnvironmentUserName": "...",
  "HttpRequestId": "...",
  "HttpRequestClientHostIP": "::1",
  "HttpRequestType": "POST",
  "Version": "1.0.0.0"
  }
}

I can query the index with DynamicResponse

            var searchResponse = client.Search<DynamicResponse>(s => s
            .Index("logstash-*")
            .AllTypes()
            .Size(50)
            .Query(q => q.Bool(b => b.Must(bs => bs.Term(p => p.Field("fields.UserId.raw").Value("..."))))
            )
        );

But I would like to use a strongly typed class for the result.

I have tried with the following class.

        public class LogResponse
    {
        public string Message { get; set; }
        [Date(Name = "@timestamp")]
        public DateTime Timestamp { get; set; }
        public LogResponseFields Fields { get; set; }

        public class LogResponseFields
        {
            public Guid UserId { get; set; }
        }
    }

But the timestamp is not set, it defaults to DateTime.Min, both Message and UserId are mapped correctly.

How to I map the @timestamp field?

what does the mapping in Elasticsearch look like? – Russ Cam Aug 18 '16 at 01:41 — Russ Cam, Aug 18 '16 at 01:41

score 1 · Accepted Answer · answered Aug 18 '16 at 02:26

The mapping for the timestamp field is working correctly; here's a complete example to demonstrate

void Main()
{
    var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
    var defaultIndex = "logstash-" + DateTime.UtcNow.ToString("yyyy-MM-dd");
    var connectionSettings = new ConnectionSettings(pool)
            .DefaultIndex(defaultIndex)
            .InferMappingFor<LogResponse>(m => m
                .TypeName("log")
            );

    var client = new ElasticClient(connectionSettings);

    // delete the index if it exists. Useful for demo purposes
    if (client.IndexExists(defaultIndex).Exists)
    {
        client.DeleteIndex(defaultIndex);
    }

    // use the lowlevel client to send the exact json as 
    // it would be sent from the source
    var createResponse = client.LowLevel.Index<string>(
        defaultIndex, 
        "log",
        @"{
            ""@timestamp"": ""2016-08-17T08:57:37.3487446+02:00"",
            ""level"": ""Information"",
            ""messageTemplate"": ""User login {UserId}"",
            ""message"": ""User login .."",
            ""fields"": {
              ""UserId"": ""29a35806-15d2-4eed-a3bf-707760e426b8"",
              ""ProcessId"": 15568,
              ""ThreadId"": 14,
              ""MachineName"": ""..."",
              ""EnvironmentUserName"": ""..."",
              ""HttpRequestId"": ""..."",
              ""HttpRequestClientHostIP"": ""::1"",
              ""HttpRequestType"": ""POST"",
              ""Version"": ""1.0.0.0""
            }
        }");

    if (!createResponse.SuccessOrKnownError)
    {
        Console.WriteLine("Error indexing document");
    }

    // Refresh the index after indexing. Useful for demo purposes.
    client.Refresh(defaultIndex);

    var searchResponse = client.Search<LogResponse>(s => s
        .AllTypes()
        .Size(50)
        .Query(q => q
            .MatchAll()
        )
    );

    foreach (var document in searchResponse.Documents)
    {
        Console.WriteLine(document.Timestamp);
    }
}

public class LogResponse
{
    public string Message { get; set; }
    [Date(Name = "@timestamp")]
    public DateTime Timestamp { get; set; }
    public LogResponseFields Fields { get; set; }

    public class LogResponseFields
    {
        public Guid UserId { get; set; }
    }
}

This outputs

17/08/2016 4:57:37 PM

for me (it has been deserialized into my local DateTime).

score 0 · Answer 2 · answered Aug 17 '16 at 20:06

0

Try DateTimeOffset that appears to be a timestamp with embedded timezone information. (+02:00 specifically)

answered Aug 17 '16 at 20:06

Guvante

18,775
1
33
64

score 0 · Answer 3 · answered Aug 18 '16 at 06:19

0

Thanks for the input.

The problem was fixed when I made the example to post to SO.

My real code was:

public class LogResponse : DynamicResponse

So by removing DynamicResponse inheritence

public class LogResponse

Every thing is working.

answered Aug 18 '16 at 06:19

MlyMly

23
3

Elastic Search mapping of logstash @timestamp to strongly typed in .NET

3 Answers3