Search in multiple indexes in elastica

Question

I am looking for a way to search in more than one index at the same time using Elastica.

I have an index products, and an index user.

products contains {product_id, product_name, price} and user contains {product_id, user_name, date}. Knowing that the product_id in both of them is the same, in products each products_id is unique but in user they're not as a user can buy the same product multiple times.

Anyway, I want to automatically get the price of a product from the products index while searching through the user index.

I know that we can search over multiple indexes like so (correct me if I'm wrong) :

$search = new \Elastica\Search($client);
$search->addIndex('users')
        ->addType('user')
        ->addIndex('products')
        ->addType('product');

But the problem is, when I write an aggregation on the products_id for example and then create a new query with some filters :

$products_agg = new \Elastica\Aggregation\Terms('products_id');
$products_agg->setField('products_id')->setSize(0);

$query = new \Elastica\Query();
$query->addAggregation($products_agg);
$query->setQuery($bool);

$search->setQuery($query);

How does elastica know in which index to search? How can I link this products_id to the other index?

score 1 · Answer 1 · edited Sep 04 '20 at 10:05

1

Basically you want to join two indexes based on a common field as in sql.

What you can do is model you data in the same index using join datatype

https://www.elastic.co/guide/en/elasticsearch/reference/master/parent-join.html

Index all documents in the same index ,

Make all product documents - parent.
Make all user documents as child

And the use parent-child aggregations and queries

https://www.elastic.co/guide/en/elasticsearch/reference/master/parent-join.html#_parent_join_queries_and_aggregations

NOTE: make sure of the performance implication of parent-child mapping https://www.elastic.co/guide/en/elasticsearch/reference/master/parent-join.html#_parent_join_and_performance

One more thing you can do is put all the information of the product with every user that buys it.

But this can unnecessarily waste you space and is not a good practice as per data rules are concerned.

But since this is a search engine and elasticsearch suggests that best is to normalise and duplicate data rather that using parent-child.

edited Sep 04 '20 at 10:05

Alexandre Elshobokshy

10,720
6
27
57

answered Oct 10 '18 at 14:53

gaurav9620

1,147
12
30

Thank you for your answer! :) I think tho that there might be a simpler way to do this.. I'm also not sure that that datatype exists in my version of elastica. – Alexandre Elshobokshy Oct 10 '18 at 14:55
Which version of elasticsearch are you using? – gaurav9620 Oct 10 '18 at 16:01
I'm using version `1.6.2` – Alexandre Elshobokshy Oct 11 '18 at 07:25
In your case it is parent-child mapping with data of two types in same index in different types . Or else you can choose to duplicate the data – gaurav9620 Oct 11 '18 at 09:34

Gasol · Answer 2 · 2018-10-22T03:15:55.127

The Elastica library has support for Multi Search API, The multi search API allows to execute several search requests within the same API. The endpoint for it is _msearch.

The format of the requests is similar to the bulk API, The first line is header part that includes which index / indices to search on, The second line includes the typical search body requests.

{"index" : "products", "type": "products"}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 10} // write your own query to get price
{"index" : "uesrs", "type" : "user"}
{"query" : {"match_all" : {}}} // query for user

Check test case in Multi/SearchTest.php to see how to use.

score 0 · Answer 3 · answered Oct 16 '18 at 15:52

you can try the following: 1- naming indexes with specific name like the following

myFirstIndex-myProjectName
mySecIndex-myProjectName
myThirdIndex-myProjectName

and so on.

2- that's give me the ability using * in the field of indexes to search because it accepts wildcard so i can search across multiple fields like this using kibana Dev Tools

GET *-myProjectName/_search
{
  "_source": {
    "excludes": [ "*" ]
  },
  "query": { "match_all": {} },
}

this will search on each index includes -myProjectName.

I'm looking for an Elastica answer, but thank you for your response. — Alexandre Elshobokshy, Oct 17 '18 at 07:00

score 0 · Answer 4 · edited Oct 19 '18 at 09:36

0

You can't query two indices with different mappings. Best way to solve your problem is to just do two queries (application-side joins). First query you do the aggregations on the user and the second you get the prices.

Another option would be to add the price to the user index. Sometimes you have to sacrifice a little space for better usability.

edited Oct 19 '18 at 09:36

Alexandre Elshobokshy

10,720
6
27
57

answered Oct 18 '18 at 06:03

ZiadM

374
2
6

Thank you for your answer. `You can't query two indices with different mappings` source please? – Alexandre Elshobokshy Oct 19 '18 at 07:32
Also what I am asking is possible in Elasticsearch for sure (`curl -H "Content-Type: application/json" -X POST 'elasticsearchServer.com/products,user/_search' --data '{"query":{"match_all":{}}}'`), so my question is how to do it using Elastica. – Alexandre Elshobokshy Oct 19 '18 at 09:55
Yes you right, maybe my answer isn't very clear. What I understood from your question is that you are trying to do joins between the indicies, which is better done by adding the products fields inside the orders, thus sacrificing space. – ZiadM Oct 19 '18 at 10:03

Search in multiple indexes in elastica

4 Answers4