Is it possible to map Elastic document ID value to a field in your mapping

Question

I know its possible to map your personal id so that the document id would be the same as your personal id, but is it possible to have it other way around ?

How to map document id value to a property in your mapping ?

Answers could be preferably using C# NEST library but not necessary, thank you.

What prevents you from accessing `_id` directly? Can you explain the need behind this? — Val, May 12 '16 at 06:48
In order for me to get the document id in my results I'd need to re-iteriate through returned results and assign the document id to a custom field in my mapping POCO class. When dealing with big sets of data (7k+) this additional step is a bit clunky, so I thought I could populate this custom field on document creation (when indexing data) with Elastic (if possible of course). Hope this makes sense — nlv, May 12 '16 at 06:48
So you're letting ES generate the ID and you'd like that ID to be available in your documents as well. correct? — Val, May 12 '16 at 06:51
Yes, im letting ES generate the ID but Id like to map (on document creation/ indexing data) that ID value to a field in my mapping called MyDocId. As mentioned above - you can do map custom MyDocId value to elastic's document value, but I would like to do this other way around. — nlv, May 12 '16 at 06:54

score 4 · Answer 1 · answered May 12 '16 at 10:38

4

NEST won't currently map a generated id onto a property of a POCO; you can assign the id manually from response.Hits metadata using

var response = client.Search<Poco>();

var pocos = response.Hits.Select(hit =>
    {
        hit.Source.Id = h.Id;
        return hit.Source;
    }).ToList();

This was discussed again in January and we decided to keep the current implementation; the POCO maps to the _source in Elasticsearch.

answered May 12 '16 at 10:38

Russ Cam

124,184
33
204
266

Shame, will have to go and generate my own id and push it on to elastic to use it as a doc id... Re to assigning it manually - very expensive move when receiving thousands of results. Thanks anyway – nlv May 12 '16 at 11:04
@NeilVarnas You can have ES generate it for you, but need to ensure that you map it onto your POCO after deserialization and use the id when making any update operations to a document in ES. If you feel there's a nice way of making this configurable functionality, feel free to discuss - https://github.com/elastic/elasticsearch-net/issues – Russ Cam May 12 '16 at 11:21
I think You miss-understood me... I know ES can generate an ID for em but thats not an objective. And again - grabbing id field value on de-serialidation at runtime is more expensive operation than generating my own id and telling ES to use that as doc ID when indexing new data. Thanks – nlv May 12 '16 at 11:25
You want to map the document id to a property in your mapping? – Russ Cam May 12 '16 at 11:35
Yes, I think Val's answer below is the closest, thanks – nlv May 12 '16 at 11:37
OK, NEST does that already :) Set an `Id` property on your POCO and it'll be used as the `_id` for the document, as well as stored in `_source` of the document. Granted, you set the `Id` property that sets the `_id` rather than set the `_id` that sets the `Id` property, but the end result is the same. – Russ Cam May 12 '16 at 11:43
Yes yes, thats what im going to go for at the end. The problem witht his options is - indexing documents with custom IDs is slower than having ES generate an ID for You :) – nlv May 12 '16 at 11:46
1

Maybe, would need to measure that. One thing to bear in mind is how you generate those ids - http://blog.mikemccandless.com/2014/05/choosing-fast-unique-identifier-uuid.html – Russ Cam May 12 '16 at 11:53
0 padded is the fastest ?... how do You get that when you dont know how many documents you gonna end up with ... Nevertheless - good read, thanks – nlv May 12 '16 at 11:58

Val · Answer 2 · 2016-05-12T07:10:32.313

I'm not aware of any feature that can modify the source that is indexed into ES.

The (deprecated) transform feature could add a new arbitrary field at indexing time, but would not modify the source, which means that when retrieving the results, you'd not get the created field.

I'd suggest you create your own IDs and assign them to your MyDocId field and don't let ES generate them.

Note that in ES 5, we'll have a new type of node called Ingest node, which will allow to define transformation pipelines, similar to what can be done with Logstash filters. At that point, you'll be able to use the set processor in order to set/create an arbitrary field and achieve what you want:

{
  "set": {
    "field": "MyDocId",
    "value": "_id"
  }
}

an options - yes. An option which Ill most likely turn to at the end. It would be interesting to know whether [routing](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-fields.html#_routing_meta_fields) could work between the doc _id field and a mapping field — nlv, May 12 '16 at 07:12

Is it possible to map Elastic document ID value to a field in your mapping

2 Answers2