Elasticsearch Ruby Activerecord Persistence Model URL term search

Question

I'm trying to make a search on a field that contains URL using elastic search Term query. I use elasticsearch-rails the ActiveRecord Persistance Pattern. This is how I try to do it.

total_views = UserAction.search :query=> {
        :filtered=> {
            :filter=> {
                :term=> { action_path:"http://0.0.0.0:3000/tshirt/test" } 
            }
        }
    }

It works if there are no '/' or ':' characters. For example when the action_path is just 'tshirt'. The other fields are not analyzed and they work if there are no '/', ':' kinds of characters in the field. So obviously elastic search tries to analyze it but the problem is they should not be analyzed because mapping is already there.

This my user action class

class UserAction
  include Elasticsearch::Persistence::Model  
  extend Calculations
  include Styles

  attribute :user_id, Integer
    attribute :user_referrer, String, mapping: { index: 'not_analyzed' } 
    attribute :user_ip, String, mapping: { index: 'not_analyzed' } 
    attribute :user_country, String, mapping: { index: 'not_analyzed' }
    attribute :user_city, String, mapping: { index: 'not_analyzed' }
    attribute :user_device, String, mapping: { index: 'not_analyzed' }
  attribute :user_agent, String, mapping: { index: 'not_analyzed' }
    attribute :user_platform
  attribute :user_visitid, Integer
    attribute :action_type, String, mapping: { index: 'not_analyzed' } 
    attribute :action_css, String, mapping: { index: 'not_analyzed' }
  attribute :action_text, String, mapping: { index: 'not_analyzed' }
  attribute :action_path, String, mapping: { index: 'not_analyzed' } 
  attribute :share_url, String, mapping: { index: 'not_analyzed' } 
  attribute :tag 
  attribute :date

I also tried adding indexes using 'mapping do.." and then "create_index!" but result is the same. Because mapping is there it does create the mapping.

This is my gem file

   gem "elasticsearch-model", git: "git://github.com/elasticsearch/elasticsearch-rails.git", require: "elasticsearch/model"
          gem "elasticsearch-persistence", git: "git://github.com/elasticsearch/elasticsearch-rails.git", require: "elasticsearch/persistence/model"
          gem "elasticsearch-rails"

When I make the search I also see that those fields that are not analyzed.

       :reload_on_failure=>false,
         :randomize_hosts=>false,
         :transport_options=>{}},
       @protocol="http",
       @reload_after=10000,
       @resurrect_after=60,
       @serializer=
        #<Elasticsearch::Transport::Transport::Serializer::MultiJson:0x007fc4bf9e0e18
         @transport=#<Elasticsearch::Transport::Transport::HTTP::Faraday:0x007fc4bf9b35a8 ...>>,
       @sniffer=
        #<Elasticsearch::Transport::Transport::Sniffer:0x007fc4bf9e0dc8
         @timeout=1,
         @transport=#<Elasticsearch::Transport::Transport::HTTP::Faraday:0x007fc4bf9b35a8 ...>>,
       @tracer=nil>>,
   @document_type="user_action",
   @index_name="useraction",
   @klass=UserAction,
   @mapping=
    #<Elasticsearch::Model::Indexing::Mappings:0x007fc4bfab18d8
     @mapping=
      {:created_at=>{:type=>"date"},
       :updated_at=>{:type=>"date"},
       :user_id=>{:type=>"integer"},
       :user_referrer=>{:type=>"string"},
       :user_ip=>{:type=>"string"},
       :user_country=>{:type=>"string", :index=>"not_analyzed"},
       :user_city=>{:type=>"string", :index=>"not_analyzed"},
       :user_device=>{:type=>"string", :index=>"not_analyzed"},
       :user_agent=>{:type=>"string", :index=>"not_analyzed"},
       :user_platform=>{:type=>"string"},
       :user_visitid=>{:type=>"integer"},
       :action_type=>{:type=>"string", :index=>"not_analyzed"},
       :action_css=>{:type=>"string", :index=>"not_analyzed"},
       :action_text=>{:type=>"string", :index=>"not_analyzed"},
       :action_path=>{:type=>"string", :index=>"not_analyzed"}},
     @options={},
     @type="user_action">,
   @options={:host=>UserAction}>,
 @response={"took"=>1, "timed_out"=>false, "_shards"=>{"total"=>4, "successful"=>4, "failed"=>0}, "hits"=>{"total"=>0, "max_score"=>nil, "hits"=>[]}}>
(END)

the initializer file has nothing other than the elastichq connection url.

Data is there in elastichq so I should get the results but can't get any.

    user_action 1   AUzH9xKDueQ8OtBQuyQC    http://example.org/api/analytics/track
user_actions    user_action 1   AUzIAUsvueQ8OtBQuyQg    http://0.0.0.0:3000/tshirt/funnel_test2
user_actions    user_action 1   AUzH7ay5ueQ8OtBQuyP2    http://example.org/api/analytics/track
user_actions    user_action 1   AUzH-HAdueQ8OtBQuyQU    http://0.0.0.0:3000/tshirt/test
user_actions    user_action 1   AUzIJbCGueQ8OtBQuyQ4    http://example.org/api/analytics/track
user_actions    user_action 1   AUzIJbCjueQ8OtBQuyQ5    http://example.org/api/analytics/track

Curl Results from Elastichq

curl -XGET "https://YYYYY:XXXXX@xxxx.qbox.io/user_actions/_mapping"
{
  "user_actions": {
    "mappings": {
      "user_action": {
        "properties": {
          "action_css": { "type": "string" },
          "action_path": { "type": "string" },
          "action_text": { "type": "string" },
          "action_type": { "type": "string" },
          "created_at": { "format": "dateOptionalTime", "type": "date" },
          "date": { "type": "string" },
          "share_url": { "type": "string" },
          "tag": { "type": "string" },
          "updated_at": { "format": "dateOptionalTime", "type": "date" },
          "user_agent": { "type": "string" },
          "user_city": { "type": "string" },
          "user_country": { "type": "string" },
          "user_device": { "type": "string" },
          "user_id": { "type": "long" },
          "user_ip": { "type": "string" },
          "user_referrer": { "type": "string" },
          "user_visitid": { "type": "long" }
        }
      }
    }
  }
}

can anybody help me on getting url term search work?

Does the mapping that was created (look at /_mapping?pretty) have what you expect? — Frederick Cheung, Apr 18 '15 at 12:05
http://prntscr.com/6v7aqp this is the mapping section on elastichq ...I don't know if I should see not analyzed fields in here but they are not here. But as I written above I see the mappings on the response when I do the search. — Caner, Apr 18 '15 at 12:37
The mapping you see when you do the query isn't part of the response - ES doesn't return that information. I'm not sure how elastichq displays mapping information but I would expect to see details such as analyzer - that looks like it could be a dynamic mapping (would happen if you inserted data before setting up the mapping) — Frederick Cheung, Apr 18 '15 at 12:58
Can you run `curl -XGET "http://localhost:9200/useraction/user_action/_mapping"` and provide the output? — Andrei Stefan, Apr 20 '15 at 23:08
there is only one user actions mapping on xbox and I don't run local ES...I updated the result above looks like it doesn't get the not_analyzed fields. But they are in my persistence model...I can do it custom — Caner, Apr 22 '15 at 15:41
And how did you create the index because, obviously, it doesn't match your model? — Andrei Stefan, Apr 22 '15 at 20:52
check above "This my user action class". Elastic search rails supposed to create it — Caner, Apr 23 '15 at 05:58
Did you change the mapping after the index was first created? — Frederick Cheung, Apr 23 '15 at 06:20

score 2 · Answer 1 · answered Apr 23 '15 at 15:02

2

From elasticsearch curl at the end it seems that your field is analyzed (does not have not_analyzed flag). Maybe try to rebuild your index with the mapping you want.

answered Apr 23 '15 at 15:02

Piotr Jakubowski

1,760
10
16

score 1 · Accepted Answer · answered Apr 26 '15 at 17:18

I did what I didn't want to do. Created the index with its mapping manually with the below post request so elasticsearch-rails can't create it wrong. Now everything works fine

curl -XPOST https://xxxxxx.qbox.io/user_actions -d '{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "user_action" : {
            "_source" : { "enabled" : false },
            "properties" : {
                "action_path" : { "type" : "string", "index" : "not_analyzed" }
            }
        }
    }
}'

score 0 · Answer 3 · answered Apr 24 '15 at 16:57

As a rule of thumb, if you want to search for something, you should not leave it not_analyzed.

In this case in particular, you should definitely try the Keyword Analyzer, setting the relevant fields mapping to keyword.

As long as you search for the full string, i.e., "http://0.0.0.0:3000/tshirt/test", there is a great chance going with the Keyword Analyzer will do the trick.

score 0 · Answer 4 · answered Apr 26 '15 at 03:16

0

Try a raw query:

total_views = UserAction.search :query=> {
    :filtered=> {
        :filter=> {
            :term=> { "action_path.raw" => "http://0.0.0.0:3000/tshirt/test" } 
        }
    }
}

answered Apr 26 '15 at 03:16

Abraham Sangha

336
1
9

Elasticsearch Ruby Activerecord Persistence Model URL term search

4 Answers4