ElasticSearch Bulk Delete by id

Question

I'm attempting to do a bulk delete by ids and it appears to be not working. When I run my test suite and try via the Rails console, it seems to work OK. However, I'm running into occurrences where there are documents that simply don't get removed. The response however, indicates it was successful.

Using tire 0.5.8, curb 0.8.4, ElasticSearch 0.90.2.

query = Tire::Search::Search.new do
  query do
    filtered do
      query { all }
      filter(:terms, _id: ids_array)
    end
  end
end

Curl::Easy.http_delete("#{index.url}/_query?source=#{Tire::Utils.escape(query.to_hash[:query].to_json)}") do |curb|
  curb.http_auth_types = :basic
  curb.username = ENV["ELASTICSEARCH_USERNAME"]
  curb.password = ENV["ELASTICSEARCH_PASSWORD"]
end

The model mapping looks like:

tire.settings ElasticSearchAnalysis do
  mapping _all: { enabled: false } do
    indexes :id, type: "integer"
  end
end

Sample HTTP DELETE:

DELETE /production_shifts/_query?source=%7B%22filtered%22%3A%7B%22query%22%3A%7B%22match_all%22%3A%7B%7D%7D%2C%22filter%22%3A%7B%22and%22%3A%5B%7B%22terms%22%3A%7B%22_id%22%3A%5B587965789%2C587965791%7D%7D%5D%7D%7D%7D

When unescaped, the query looks like:

{\"filtered\":{\"query\":{\"match_all\":{}},\"filter\":{\"and\":[{\"terms\":{\"_id\":[587965789,587965791}}]}}}

In the past I've had problems with deleting objects from elasticsearch but the problem was with ES and not Tire. — jspooner, Jul 12 '13 at 16:10
What does your failing test look like? Do you add documents and then immediately delete them? — javanna, Jul 15 '13 at 16:04
Can you answer the questions of the people or at least accept some answer? It would be useful for other users. Or if you have solved it yourself just add the solution as answer and accept the same. You can accept your own answer. — Vamsi Krishna, Jul 29 '13 at 07:00

score 0 · Answer 1 · answered Jul 17 '13 at 18:12

0

That 'and' in the json query looks wrong. Nothing is going to match that query.

In any case, try the same query with a get to see what you get back.

answered Jul 17 '13 at 18:12

Jilles van Gurp

7,927
4
38
46

score 0 · Answer 2 · answered Jul 26 '13 at 10:25

Why do you need a filtered query there?

You have indexed the id field as id not _id... The below query should work for sure, this is a tested query.

You can just use this,

query = Tire.search do |search|
        search.query { |q| q.terms :id, ids_array }
      end

Reference: https://github.com/karmi/tire/issues/309

ElasticSearch Bulk Delete by id

2 Answers2