Bulk Partial Upsert in Elasticseach with python

Question

I want to send n upsert partial requests to ES, is such a thing possible? So if the document doesn't exist, insert my partial doc. If it already exists, update it with the partial doc.

Using the bulk helpers, I've tried a ton of variations, but they all wipe out existing values in favour of the new ones.

data = [{
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "doc": {"newkey": 'newvalue'}
}]
helpers.bulk(es, data, index='my_index', doc_type='my_type')

or

data = [{
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "_source": {"newkey": 'newvalue'}
}]
helpers.bulk(es, data, index='my_index', doc_type='my_type')

Does not work either.

score 17 · Answer 1 · answered Mar 18 '17 at 14:44

17

As J. Ku answered, he has given the right answer but the code provided is incomplete. So posting the complete answer.

data = [{
    "_op_type": 'update',
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "doc": {"newkey": 'newvalue'},
    "doc_as_upsert":True
}]

helpers.bulk(es, data, index='my_index', doc_type='my_type')

answered Mar 18 '17 at 14:44

jhandei

183
1
12

what if you dont know the _id? – Kailegh Jul 02 '21 at 12:06
Haven't worked on elastic search for a long time. However I feel, _id is necessary in upsert, else you cannot identify whether the doc is there or not. And if you don't know it, you have to make another query to fetch all the ids you want to update. – jhandei Jul 18 '21 at 15:28

J.Ku · Answer 2 · 2015-12-23T20:19:24.110

1

I think you need to include the action as mentioned in the documentation and set upsert as true

data = [{
    "_op_type": 'update',
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "doc": {"newkey": 'newvalue'}
}]
helpers.bulk(es, data, index='my_index', doc_type='my_type')

edited Dec 23 '15 at 20:19

answered Dec 23 '15 at 19:48

J.Ku

123
7

`doc_as_upsert` does not make a difference. Also I want the action to be `index`, which is default if you leave it out. Finally, I'm using the bulk API which is different syntax from what you posted. – L-R Dec 23 '15 at 19:51
I believe the `index` action always overwrites the existing value if exist. – J.Ku Dec 23 '15 at 19:53
not sure how to do it with `index`. my answer uses `update` and it would behave like you expected – J.Ku Dec 23 '15 at 20:16
sorry, I had the syntax wrong, I think the above works. – L-R Dec 23 '15 at 20:18

Bulk Partial Upsert in Elasticseach with python

2 Answers2