How do I serialize properties of type JToken or JObject in Elasticsearch NEST?

Question

I'm introducing Elasticsearch into a C# API project. I'd like to leverage existing API models as search documents, many of which allow for adding custom data points. These are implemented using the JObject type from Json.NET. For example:

public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public JObject ExtraProps { get; set; }
}

This allows users to send JSON request bodies like this, which works great:

{
   "Id": 123,
   "Name": "Thing",
   "ExtraProps": {
      "Color": "red",
      "Size": "large"
   }
}

However, if I use this as a document type in NEST, those extra properties are losing their values somehow, serializing as:

{
   "Id": 123,
   "Name": "Thing",
   "ExtraProps": {
      "Color": [],
      "Size": []
   }
}

Adding a [Nest.Object] attribute to ExtraProps didn't change the behavior. As I understand it, NEST uses Json.NET internally, so I wouldn't expect it to have problems with Json.NET types. Is there a relatively simple fix for this?

Here are some options I'm weighing:

1. Use custom serialization. I started down this path, it got to feeling way more complicated than it should be, and I never did get it working.

2. Map JObjects to Dictionary<string, object>s. I have verified this works, but if there are nested objects (which there could be), I'll need to enhance it with recursion. And, ideally, I'd like this to work with the more general JToken type. This is the option I'm leaning toward, but again, it feels more complicated than it should be.

3. Use the "Low Level" client or even raw HTTP calls. Admittedly I haven't explored this, but if it's really simpler/cleaner than the alternatives, I'm open to it.

4. Report this as a bug. I'll probably do this regardless. I just have a hunch this should work with JObject or any JToken out of the box, unless there is some reason that this is intended behavior.

Answer 1

This is expected behaviour with NEST 6.x.

NEST uses Json.NET for serialization. In NEST 6.x however, this dependency was internalized within the NEST assembly by

• IL-merging all Json.NET types into the NEST assembly
• renamespacing the types within Newtonsoft.Json to Nest.Json
• marking all types internal

There's a blog post with further details explaining the motivations behind this change.

When it comes to handling Json.NET types such as Newtonsoft.Json.Linq.JObject, Json.NET has special handling for these types for serialization/deserialization. With NEST 6.x, the internalized Json.NET does not know how to specially handle Newtonsoft.Json.Linq.JObject because all types within the internalized Json.NET have been renamespaced to the Nest.Json namespace.

To support Json.NET types, a serializer that uses Json.NET to serialize your documents needs to be hooked up. The NEST.JsonNetSerializer nuget package was created to help with this. Simply add a reference to NEST.JsonNetSerializer to your project, then hook up the serializer as follows

// choose the appropriate IConnectionPool for your use case
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings =
    new ConnectionSettings(pool, JsonNetSerializer.Default);
var client = new ElasticClient(connectionSettings);

With this is place, documents with JObject properties will be serialized as expected.