Elasticsearch update the whole `_source` field based on search query

Question

"_source": {
         "id": "5b1676493d21784208c36041",
         "label": "name",
         "properties": {
           "name": "patrick"
         },
         "updatedAt": 1528259039542
       }

I want to update this document based on id (not _id) with a new document.

Something like this:

    "_source": {
             "dataSource": "ELASTIC",
             "entity": "vertices",
             "label": "vertices",
             "id": "5b1676493d21784208c36041",
             "properties": {
                     "name": "patrick"
                  },
             "updatedAt": 1528259039542
           }

elasticsearch version: 6.2, ES Java api: 6.2

Since there is no such thing as an update in a search engine anyway, just delete the old document, then add the new one. https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html may help. — Harald, Jun 07 '18 at 14:08

Val · Accepted Answer · 2018-06-13T12:38:11.713

5

You can achieve what you want using the update by query API, basically like this:

POST index/_update_by_query
{
  "query": {
    "match": {
      "id": "5b1676493d21784208c36041"
    }
  },
  "script": {
    "source": "ctx._source = params",
    "params": {
       "dataSource": "ELASTIC",
       "entity": "vertices",
       "label": "vertices"
    }
  }
}

UPDATE: Using the Java API

Map<String, String> params = new HashMap<>();
params.put("dataSource", "ELASTIC");
params.put("entity", "vertices");
params.put("label", "vertices");

UpdateByQueryRequestBuilder updateByQuery = UpdateByQueryAction.INSTANCE.newRequestBuilder(client);
updateByQuery.source("index")
    .filter(QueryBuilders.matchQuery("id", "5b1676493d21784208c36041"))
    .size(1000)
    .script(new Script(ScriptType.INLINE, "painless", "ctx._source.putAll(params)", params));
BulkByScrollResponse response = updateByQuery.get();

More details on using the UpdateByQuery Java API and Java high level rest client

edited Jun 13 '18 at 12:38

answered Jun 07 '18 at 14:11

Val

207,596
13
358
360

It's the name of the scripting language used in the script present in the second parameter – Val Jun 07 '18 at 14:31
Please update your question with the results... more legible than here – Val Jun 08 '18 at 09:38
Can you provide a meaningful portion of the stacktrace ? – Val Jun 08 '18 at 09:46
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/172750/discussion-between-abhijith-s-and-val). – Abhijith S Jun 08 '18 at 09:51
`"ctx._source = params"` This also is not supporting – Abhijith S Jun 08 '18 at 11:50
I'm curious to know what your `introspect()` method does... Can you create a simple HashMap like I did in my example and try it out? Start simple and then iterate. – Val Jun 08 '18 at 11:51
when I tried using HashMap as you said "ctx._source = params" is not supporting – Abhijith S Jun 08 '18 at 11:57
Setting ctx.op to any other value generates an error. Setting any other field in ctx generates an error. from the es documentation: https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-update-by-query.html – Abhijith S Jun 10 '18 at 18:18
Small change @Val `.script(new Script(ScriptType.INLINE, "painless", "ctx._source = params", params));` – Abhijith S Jun 10 '18 at 18:53
You're right, I've inverted the "painless" and the code. I don't think that's a reason for downvoting, though. I've updated the code. Works better now? – Val Jun 11 '18 at 04:56
No, Actually my requirement is different this is creating a field `ctx` in `_source` – Abhijith S Jun 12 '18 at 09:41
What the script does is replace all fields of the `_source` with the ones present in `params`, which is what you asked for. `ctx._source` is the way to access the original source document and we assign it the params values – Val Jun 12 '18 at 09:44
hey @Val can you pleas check my updated question? That is what my requirement. – Abhijith S Jun 12 '18 at 09:46
You should have created a new question as now all the comments are out of sync with the question and answer :-( – Val Jun 12 '18 at 09:47
Why can't we delete the comments and you can edit your answer corresponding to my updated question right? – Abhijith S Jun 12 '18 at 09:49
Because I prefer spending time helping people than curating comments based on edited questions ;-) – Val Jun 12 '18 at 09:49
Nice :P . btw I tried by searching with Id and updating but with 2 api(get and put) calls, Actually my intution says this can be solved using single script query OR in a single api call. – Abhijith S Jun 12 '18 at 09:53
Simply replace the script by this one `ctx._source.putAll(params)`, it will keep the original source, but overwrite it with all fields present in params – Val Jun 12 '18 at 09:54
Getting closer @Val "_source": { "ctx": { "_routing": null, "_parent": null, "_index": "vertices", "_type": "vertices", "_id": "AeZz82MBrouOs8r6VAyE", "_version": 1 }, "id": "5b1676493d21784208c36041", "label": "updatedLabel", "properties": { "name": "sheji" }, "updatedAt": 1528797832451 } . But I don't want that `ctx`field. Can I remove that? – Abhijith S Jun 12 '18 at 10:08
Anyway update your answer @Val I will accept the answer. :) – Abhijith S Jun 12 '18 at 10:18
let me figure out why first – Val Jun 12 '18 at 10:19
In the future, please make sure to not modify the question in such a way that the posted answers don't make sense anymore. – Val Jun 13 '18 at 12:38
Sure @Val I ll definitely take that into consideration – Abhijith S Jun 13 '18 at 12:41

jetnet · Answer 2 · 2019-03-14T08:49:02.637

1

"ctx._source.putAll(params)" was a nice try, but unfortunately it moves all existing fields under _source.ctx.

So, the following works for me (ES 6.1):

"for (k in params.keySet()){if (!k.equals('ctx')){ctx._source.put(k, params.get(k))}}"

edited Mar 14 '19 at 08:49

answered Mar 13 '19 at 12:48

jetnet

571
5
8

score 0 · Answer 3 · answered Sep 04 '20 at 16:16

Thanks @jetnet for providing an idea to loop through all the entries in the map. When I used the script, it was still replacing the entire object, so had to adapt it as per the below to preserve the fields not supplied during an update operation. For context, if I first insert a document that has three objects each with 2 fields, if I then updated that document with the same 3 objects, but with 1 field each, the output is that I end up with a document with 3 objects each with 1 field instead of 3 objects with 2 fields each as I had originally.

I had to adapt @jetnet's script to give me the desired outcome of not overwriting object properties

POST /transaction_index/_update/33384637-3137-3132-5543-31304c4c3151
{
   "script": {
      "source": "if (ctx._source.Metadata == null || params.Metadata.Version >= ctx._source.Metadata.Version) { for (k in params.keySet()){ if (ctx._source[k] != null) { ctx._source[k].putAll(params.get(k)) } else { ctx._source.put(k, params.get(k)) } } } else { ctx.op = 'none' }",
      "params": {
         "Transaction": {
            "TransactionId": "33384637-3137-3132-5543-31304c4c3151",
            "TransactionKey": "Key1"
         },
         "Message": {
            "MessageId": "505a5953-374a-385a-4a48-52353549445a",
            "Context": "This is a test context"
         },
         "MessageDefinition": {
            "MessageDefinitionId": "a1c05e06-fa6b-40ce-992f-d083ff6c0243",
            "Code": 1010101010
         },
         "Metadata": {
            "Version": 1,
            "CreateTime": "2020-09-04T14:27:51.1986439+01:00",
            "IsLatest": true,
            "IsDummy": false,
            "VersionString": "20200903111111"
         }
    }
 },
 "scripted_upsert": true,
 "upsert": {}
}

Elasticsearch update the whole `_source` field based on search query

3 Answers3

Linked