elastic search not found error in ruby on rails

Question

I'm going to do elastic search on DoctorProfile and Subspeciality table. The error that I'm dealing with is that it gives the not found result. It takes a list of ids from doctor table but it doesn't gives desire result which is the doctor and with subspeciality. this is what everything that i did: I used these gems:

gem 'elasticsearch-model', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
gem 'elasticsearch-rails', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
gem 'elasticsearch-extensions', git: 'git://github.com/elasticsearch/elasticsearch-ruby.git'

my search method:

 def search
    query = params[:query]
    #query.encode("UTF-8")
    if query.nil?
      render json: { message: "no query is provided" }, status: :unprocessable_entity
      return
    end
    profiles = DoctorProfile.search(query).results.map { |r| r._source.id }
    subspecialties = Subspecialty.search(query).results.map { |r| r._source.title }
    subspecialties.uniq!
    profiles = profiles + DoctorProfile.where("subspeciality in (?)", subspecialties).ids
    profiles.uniq!
    logger.info "################  The profile is #{profiles} ########################"
    @doctors = DoctorProfile.find(profiles)
    @cleaned_doctors = @doctors.select { |u| !u.user.nil? }

    render json: @cleaned_doctors
  end

in the doctor model:

  after_commit on: [:create] do
    __elasticsearch__.index_document if self.enabled?
  end

  after_commit on: [:update] do
    __elasticsearch__.update_document if self.enabled?
  end

  after_commit on: [:destroy] do
    __elasticsearch__.delete_document
  end

  settings index: {
    number_of_shards: 1,
    number_of_replicas: 0,
    analysis: {
      filter: {
        autocomplete_filter: {
          type: "edge_ngram",
          min_gram: 1,
          max_gram: 20
        }
      },
      analyzer: {
        autocomplete: {
          type: "custom",
          tokenizer: "standard",
          filter: [
            "lowercase",
            "autocomplete_filter"
          ]
        }
      }
    }
  }

  mappings dynamic: 'false' do
    indexes :first_name, type: "string"
    indexes :last_name, type: "string"
    indexes :medical_code, type: "string"
    indexes :expertise, type: "string", analyzer: "autocomplete", search_analyzer: "standard"
    indexes :subspeciality, type: "string", analyzer: "autocomplete", search_analyzer: "standard"
  end

  def self.search(query)
    __elasticsearch__.search(
      {
        query: {
          multi_match: {
            query: query,
            fields: ['medical_code^10', 'subspeciality^5', 'expertise^2', 'first_name', 'last_name']
          }
        }
      }
    )

and in subspeciality model:

 after_commit on: [:create] do
    self.services = self.services.each {|str| str.force_encoding("UTF-8")}
    __elasticsearch__.index_document if self.enabled?
  end

  after_commit on: [:update] do
    self.services = self.services.each {|str| str.force_encoding("UTF-8")}
    __elasticsearch__.update_document if self.enabled?
  end

  after_commit on: [:destroy] do
    __elasticsearch__.delete_document
  end

 settings index: {
    number_of_shards: 1,
    number_of_replicas: 0,
    analysis: {
      filter: {
        autocomplete_filter: {
          type: "edge_ngram",
          min_gram: 1,
          max_gram: 20
        }
      },
      analyzer: {
        autocomplete: {
          type: "custom",
          tokenizer: "standard",
          filter: [
            "lowercase",
            "autocomplete_filter"
          ]
        }
      }
    }
  }

  mappings dynamic: 'false' do
    indexes :description, type: "string", analyzer: "autocomplete", search_analyzer: "standard"
    indexes :services, type: "string"
  end

  def self.search(query)
    __elasticsearch__.search(
      {
        query: {
          multi_match: {
            query: query,
            fields: ['description', 'services']
          }
        }
      }
    )
  end

and this is my error in log :

Couldn't find all DoctorProfiles with 'id': (031addd8-9df8-4a53-974d-da0067302ad0, ff890720-4bfb-47d8-bdb8-3dc712b27f29, 869b28e1-cdd7-4bb6-b1d0-c7296e4b0637, 6dd6a784-c54b-4bb7-a0e1-337474ec4114, 234ccc87-f0c7-42f7-b96f-cf8d85487929, 543b621d-87aa-4a34-b6d6-62144c6a387e, 77e35144-9b93-48a0-a5bb-7b3addb99dff, d368f1df-3d1a-49ce-b6f5-f791df3294b1, d3dca8de-3143-4b03-90ec-e73a27c88960, 24abb0b3-2d11-457b-b95d-972462c4a37f) (found 2 results, but was looking for 10

i changed this line of code

@doctors = DoctorProfile.find(profiles)

to

@doctors = DoctorProfile.where("id in (?)",profiles)

and remove this line:

@cleaned_doctors = @doctors.select { |u| !u.user.nil? }

now i want to know what does this method do eactly.

@cleaned_doctors = @doctors.select { |u| !u.user.nil? }

to be mention, i have a table named user that doctorProfile has reference to it

The activerecord seems to be missing in DB, but the elasticsearch index contains the same document. Try reindexing entire ES index if possible...The best solution is to delete the document in ES when it is deleted in db. You can use the after_destroy callback for this. There is a remove api in ES for this purpose. — Vamsi Krishna, Jun 28 '17 at 06:02
activerecord is every single entry in DB related to the rails model tables. You create a doctorprofile using rails create action and post the data to ES after save. The entry is created in db as well as ES (expecting that you might have written an after_commit or after_save). Now you delete that doctorprofile from the app. But, you forget to delete the same in ES. That is when this problem occurs. Now when you search the deleted object, it searches in ES and tries to load the object from MySQL. The object doesn't exist. So, you might get this error. — Vamsi Krishna, Jun 28 '17 at 06:14
can u give me a tutorial to do this, i'm so beginner in this case — Hussein Ojaghi, Jun 28 '17 at 06:16
I use postgresql, should i have to change something in my postgresql database — Hussein Ojaghi, Jun 28 '17 at 06:20
of course i forget to say i added include Elasticsearch::Model include Elasticsearch::Model::Callbacks in doctorprofile and subspeciality model — Hussein Ojaghi, Jun 28 '17 at 06:34
Check this out https://berislavbabic.com/refresh-your-elasticsearch-index-with-zero-downtime/ — Vamsi Krishna, Jun 28 '17 at 06:44
I fetch the index data by curl -XGET 'http://127.0.0.1:9200/doctor_profiles/_search?pretty=1' command and i see there is a few rows of my doctor profile table are over there but i don't know it doesn't show in the search result. in fact all of the outputs results are 404 not found — Hussein Ojaghi, Jun 28 '17 at 08:28
You can double check if you are doing something wrong in Rails by running the query you think you need from a REST client or the Dev Tools from Kibana (or if you are using an older version - the Sense client plugin). If the query gives you 2 documents back then you know the query/mappings are wrong. — Andrei Stefan, Jul 01 '17 at 14:43
@Sandro, as a note, if you're using ```ElasticsearchModel``` you don't need to include all those callbacks, just include in your models the Callbacks module, like this: `include Elasticsearch::Model::Callbascks` — Allam Matsubara, Jul 03 '17 at 14:28

score 0 · Accepted Answer · answered Jul 06 '17 at 10:13

Your code doesn't say much, but it seems that you have a boolean enabled on your model, which tells whether the record is to be indexed or not.

The issue is with the update callback, because if you change your model from enabled to not enabled, instead of removing it from the index, it just don't update existing information.

The correct callback would be

  after_commit on: [:update] do
    if enabled?
      if previous_changes['enabled'] &&
         !previous_changes['enabled'].first
        # previously not enabled, we need to index it
        __elasticsearch__.index_document
      else
        # previously enabled, we need to update it
        __elasticsearch__.update_document
      end
    else
      # not enabled
      if previous_changes['enabled'] &&
         previous_changes['enabled'].first
        # previously enabled, delete
        __elasticsearch__.delete_document
      end
      # if it wasn't enabled before, it's not in the index anyway.
      # do nothing
    end
  end

The previous_changes hash stores the attributes that did change when saving the model, so you can check the previous value of the enabled attribute. See http://api.rubyonrails.org/classes/ActiveModel/Dirty.html#method-i-previous_changes

Once you have the new callback in place, rebuild the indexes to remove the bogus data in production if needed:

DoctorProfile.where(enabled: true).find_each { |dp| dp.__elasticsearch__.index_document }
Subspecialty.where(enabled: true).find_each { |dp| dp.__elasticsearch__.index_document }

elastic search not found error in ruby on rails

1 Answers1