7

Assuming I have a long running update query where I am updating ~200k to 500k, perhaps even more.Why I need to update so many documents is beyond the scope of the question.

Since the client times out (I use the official ES python client), I would like to have a way to check what the status of the bulk update request is, without having to use enormous timeout values.

For a short request, the response of the request can be used, is there a way I can get the response of the request as well or if I can specify a name or id to a request so as to reference it later.

For a request which is running : I can use the tasks API to get the information.

But for other statuses - completed / failed, how do I get it. If I try to access a task which is already completed, I get resource not found .

P.S. I am using update_by_query for the update

Rami
  • 8,044
  • 18
  • 66
  • 108
Shivendra Soni
  • 672
  • 2
  • 6
  • 19
  • Is there a good reason to not break your query up into chunks? You could still use the bulk API, and maybe just break them into like chunks of 500 each. – MrName Mar 22 '18 at 22:29
  • Yes. So this is a update_by_query request, and I want to update a property of all documents which have a particular value of this property. Eg. Every doc which has `company_name` as 'flipkart' update it to 'flipkart group' – Shivendra Soni Mar 22 '18 at 22:35
  • Are you using sliced scroll with _update_by_query, which would parallelize this operation? You can also increase the request timeout value. – sramalingam24 Mar 23 '18 at 03:09
  • Yes, I am using slice as well (set to around 80% of the number of shards) and can increase the timeout as well. However, like I mentioned in the question, I want to avoid doing it and at a later point of time know whether the query was a success or not. If this is not supported out of the box, is there an approach or implementation I can use to do this. – Shivendra Soni Mar 23 '18 at 03:25
  • You can use the task api to do that https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-task-api but test it first. When I tried it and got the status before the update_by_query operation finished it broke my data – Alkis Kalogeris Mar 23 '18 at 07:00
  • @alkis please clarify a bit. Because from what I tried and understood, the Task API gives only the tasks which are currently running and not the ones which have finished running. Also as a side question, if possible, can you tell what happened in your case and how it broke your data. – Shivendra Soni Mar 23 '18 at 09:10

2 Answers2

4
With the task id you can look up the task directly:

GET /_tasks/taskId:1

The advantage of this API is that it integrates with wait_for_completion=false to transparently return the status of completed tasks. If the task is completed and wait_for_completion=false was set on it them it’ll come back with a results or an error field. The cost of this feature is the document that wait_for_completion=false creates at .tasks/task/${taskId}. It is up to you to delete that document.

From here https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-task-api

My use case went like this, I needed to do an update_by_query and I used painless as the script language. At first I did a reindex (when testing). Then I tried using the update_by_query functionality (they resemble each other a lot). I did a request to the task api (the operation hasn't finished of course) and I saw the task being executed. When it finished I did a query and the data of the fields that I was manipulating had disappeared. The script worked since I used the same script for the reindex api and everything went as it should have. I didn't investigate further because of lack of time, but... yeah, test thoroughly...

Alkis Kalogeris
  • 17,044
  • 15
  • 59
  • 113
4

I feel GET /_tasks/taskId:1 confusing to understand. It should be 

GET http://localhost:9200/_tasks/taskId

A taskId looks something like this NCvmGYS-RsW2X8JxEYumgA:1204320.

Here is my trivial explanation related to this topic.

To check a task, you need to know its taskId.

A task id is a string that consists of node_id, a colon, and a task_sequence_number. An example is taskId = NCvmGYS-RsW2X8JxEYumgA:1204320 where node_id = NCvmGYS-RsW2X8JxEYumgA and task_sequence_number = 1204320. Some people including myself thought taskId = 1204320, but that's not the way how the elasticsearch codebase developers understand it at this moment.

A taskId can be found in two ways.

  1. wait_for_deletion = false. When sending a request to ES, with this parameter, the response will be {"task" : "NCvmGYS-RsW2X8JxEYumgA:1204320"}. Then, you can check a status of that task like this GET http://localhost:9200/_tasks/NCvmGYS-RsW2X8JxEYumgA:1204320
  2. GET http://localhost:9200/_tasks?detailed=false&actions=*/delete/byquery. This example will return you the status of all tasks with action = delete_by_query. If you know there is only one task running on ES, you can find your taskId from the response of all running tasks.

After you know the taskId, you can get the status of a task with this.

GET /_tasks/taskId

Notice you can only check the status of a task when the task is running, or a task is generated with wait_for_deletion == false.

More trivial explanation, wait_for_deletion by default is true. Based on my understanding, tasks with wait_for_deletion = true are "in-memory" only. You can still check the status of a task while it's running. But it's completely gone after it is completed/canceled. Meaning checking the status will return you a 'resouce_not_found_exception'. Tasks with wait_for_deletion = false will be stored in an ES system index .task. You can still check it's status after it finishes. However, you might want to delete this task document from .task index after you are done with it to save some space. The deletion request looks like this 

http://localhost:9200/.tasks/task/NCvmGYS-RsW2X8JxEYumgA:1204320

You will receive resouce_not_found_exception if a taskId is not present. (for example, you deleted some task twice, or you are deleting an in-memory task, whose wait_for_deletetion == true).

About this confusing taskId thing, I made a pull request https://github.com/elastic/elasticsearch/pull/31122 to help clarify the Elasticsearch document. Unfortunately, they rejected it. Ugh.

Jinggang
  • 401
  • 7
  • 13