Elasticsearch - generic facets structure - calculating aggregations combined with filters

Question

In a new project of ours, we were inspired by this article http://project-a.github.io/on-site-search-design-patterns-for-e-commerce/#generic-faceted-search for doing our “facet” structure. And while I have got it working to the extent the article describes, I have run into issues in getting it to work when selecting facets. I hope someone can give a hint as to something to try, so I don’t have to redo all our aggregations into separate aggregation calculations again.

The problem is basically that we are using a single aggregation to calculate all the “facets” at once, but when I add a filter (fx. checking a brand name), then it “removes” all the other brands when returning the aggregates. What I basically want is that it should use that brand as filter when calculating the other facets, but not when calculating the brand aggregations. This is necessary so the user can, for example, choose multiple brands.

Looking at https://www.contorion.de/search/Metabo_Fein/ou1-ou2?q=Winkelschleifer&c=bovy (which is the site described in the above article), I have selected the “Metabo” and “Fein” manufacturer (Hersteller), and unfolding the Hersteller menu it shows all manufacturers and not just the ones selected. So I know it’s possible somehow and I hope some one out there has a hint as to how to write the aggregations / filters, so I get the "correct e-commerce facet behavior".

On the products in ES I have the following structure: (the same as in the original article, though “C#’ified” in naming)

"attributeStrings": [
    {
        "facetName": "Property",
        "facetValue": "Organic"
    },
    {
        "facetName": "Property",
        "facetValue": "Without parfume"
    },
    {
        "facetName": "Brand",
        "facetValue": "Adidas"
    }
]

So the above product has 2 attributes/facet groups – Property with 2 values (Organic, Without parfume) and Brand with 1 value (Adidas). Without any filters I calculate the aggregations from the following query:

  "aggs": {
    "agg_attr_strings_filter": {
      "filter": {},
      "aggs": {
        "agg_attr_strings": {
          "nested": {
            "path": "attributeStrings"
          },
          "aggs": {
            "attr_name": {
              "terms": {
                "field": "attributeStrings.facetName"
              },
              "aggs": {
                "attr_value": {
                  "terms": {
                    "field": "attributeStrings.facetValue",
                    "size": 1000,
                    "order": [
                      {
                        "_term": "asc"
                      }
                    ]
   } } } } } } } }

Now if I select Property "Organic" and Brand "Adidas" I build the same aggregation, but with a filter to apply those two constraints (which is were it kind of goes wrong...):

  "aggs": {
    "agg_attr_strings_filter": {
      "filter": {
        "bool": {
          "filter": [
            {
              "nested": {
                "query": {
                  "bool": {
                    "filter": [
                      {
                        "term": {
                          "attributeStrings.facetName": {
                            "value": "Property"
                          }
                        }
                      },
                      {
                        "terms": {
                          "attributeStrings.facetValue": [
                            "Organic"
                          ]
                        }
                      }
                    ]
                  }
                },
                "path": "attributeStrings"
              }
            },
            {
              "nested": {
                "query": {
                  "bool": {
                    "filter": [
                      {
                        "term": {
                          "attributeStrings.facetName": {
                            "value": "Brand"
                          }
                        }
                      },
                      {
                        "terms": {
                          "attributeStrings.facetValue": [
                            "Adidas"
                          ]
                        }
                      }
                    ]
                  }
                },
                "path": "attributeStrings"
              }
            }
          ]
        }
      },
      "aggs": {
        "agg_attr_strings": {
          "nested": {
            "path": "attributeStrings"
          },
          "aggs": {
            "attr_name": {
              "terms": {
                "field": "attributeStrings.facetName",
              },
              "aggs": {
                "attr_value": {
                  "terms": {
                    "field": "attributeStrings.facetValue",
                    "size": 1000,
                    "order": [
                      {
                        "_term": "asc"
                      }
                    ]
   } } } } } } } }

The only way I can see forward with this model, is to calculate the aggregation for each selected facet and somehow merge the result. But it seems very complex and kind of defeats the point of having the model as described in the article, so I hope there's a more clean solution and someone can give a hint at something to try.

Answer 1

The only way I can see forward with this model, is to calculate the aggregation for each selected facet and somehow merge the result.

This is exactly right. If one facet (e.g. brand) is selected than you can not use global brand filter if you also want to fetch other brands for multi-selection. What you can do is apply all other filters on selected facets, and all filters on non-selected facets. As a results you will have n+1 separate aggregations for n selected filters - first one is for all facets and the rest are for selected facets.

In your case query might look like:

{
  "aggs": {
    "agg_attr_strings_filter": {
      "filter": {
        "bool": {
          "filter": [
            {
              "nested": {
                "query": {
                  "bool": {
                    "filter": [
                      {
                        "term": {
                          "attributeStrings.facetName": {
                            "value": "Property"
                          }
                        }
                      },
                      {
                        "terms": {
                          "attributeStrings.facetValue": [
                            "Organic"
                          ]
                        }
                      }
                    ]
                  }
                },
                "path": "attributeStrings"
              }
            },
            {
              "nested": {
                "query": {
                  "bool": {
                    "filter": [
                      {
                        "term": {
                          "attributeStrings.facetName": {
                            "value": "Brand"
                          }
                        }
                      },
                      {
                        "terms": {
                          "attributeStrings.facetValue": [
                            "Adidas"
                          ]
                        }
                      }
                    ]
                  }
                },
                "path": "attributeStrings"
              }
            }
          ]
        }
      },
      "aggs": {
        "agg_attr_strings": {
          "nested": {
            "path": "attributeStrings"
          },
          "aggs": {
            "attr_name": {
              "terms": {
                "field": "attributeStrings.facetName"
              },
              "aggs": {
                "attr_value": {
                  "terms": {
                    "field": "attributeStrings.facetValue",
                    "size": 1000,
                    "order": [
                      {
                        "_term": "asc"
                      }
                    ]
                  }
                }
              }
            }
          }
        }
      }
    },
    "special_agg_property": {
      "filter": {
        "nested": {
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "attributeStrings.facetName": {
                      "value": "Brand"
                    }
                  }
                },
                {
                  "terms": {
                    "attributeStrings.facetValue": [
                      "Adidas"
                    ]
                  }
                }
              ]
            }
          },
          "path": "attributeStrings"
        }
      },
      "aggs": {
        "special_agg_property": {
          "nested": {
            "path": "attributeStrings"
          },
          "aggs": {
            "agg_filtered_special": {
              "filter": {
                "query": {
                  "match": {
                    "attributeStrings.facetName": "Property"
                  }
                }
              },
              "aggs": {
                "facet_value": {
                  "terms": {
                    "size": 1000,
                    "field": "attributeStrings.facetValue"
                  }
                }
              }
            }
          }
        }
      }
    },
    "special_agg_brand": {
      "filter": {
        "nested": {
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "attributeStrings.facetName": {
                      "value": "Property"
                    }
                  }
                },
                {
                  "terms": {
                    "attributeStrings.facetValue": [
                      "Organic"
                    ]
                  }
                }
              ]
            }
          },
          "path": "attributeStrings"
        }
      },
      "aggs": {
        "special_agg_brand": {
          "nested": {
            "path": "attributeStrings"
          },
          "aggs": {
            "agg_filtered_special": {
              "filter": {
                "query": {
                  "match": {
                    "attributeStrings.facetName": "Brand"
                  }
                }
              },
              "aggs": {
                "facet_value": {
                  "terms": {
                    "size": 1000,
                    "field": "attributeStrings.facetValue"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

This query looks super big and scary but generating such query can be done with few dozen lines of code. When parsing query results, you need to first parse general aggregation (one that uses all filters) and after special facet aggregations. From the upper example, first parse results from agg_attr_strings_filter but those results will also contain aggregation values for Brand and Property that should be overwritten by aggregation values from special_agg_property and special_agg_brand Also, this query is efficient since Elasticsearch does good job in caching separate filter clauses so applying same filters in different parts of query should be cheap.

But it seems very complex and kind of defeats the point of having the model as described in the article, so I hope there's a more clean solution and someone can give a hint at something to try.

There is really no way around the fact that you need to apply different filters to different facets and at the same time have different query filters. If you need to support "correct e-commerce facet behavior" you will have complex query :)

Disclaimer: I'm coauthor of the mentioned article.

Answer 2

The issue comes from the fact that you are adding a filter on Property and Organic inside your aggregation, hence the more facets you pick, the more you will restrain the terms you will get. In that article, the filter they use is in fact a post_filter, both names were allowed until recently, but filter got removed because that was causing ambiguities.

What you need to do is to move that filter outside the aggregations into the post_filter section, so that the results get correctly filtered out by whatever facets have been picked, but all your facets still get computed correctly on the whole document set.

{
  "post_filter": {
    "bool": {
      "filter": [
        {
          "nested": {
            "query": {
              "bool": {
                "filter": [
                  {
                    "term": {
                      "attributeStrings.facetName": {
                        "value": "Property"
                      }
                    }
                  },
                  {
                    "terms": {
                      "attributeStrings.facetValue": [
                        "Organic"
                      ]
                    }
                  }
                ]
              }
            },
            "path": "attributeStrings"
          }
        },
        {
          "nested": {
            "query": {
              "bool": {
                "filter": [
                  {
                    "term": {
                      "attributeStrings.facetName": {
                        "value": "Brand"
                      }
                    }
                  },
                  {
                    "terms": {
                      "attributeStrings.facetValue": [
                        "Adidas"
                      ]
                    }
                  }
                ]
              }
            },
            "path": "attributeStrings"
          }
        }
      ]
    }
  },
  "aggs": {
    "agg_attr_strings_full": {
      "nested": {
        "path": "attributeStrings"
      },
      "aggs": {
        "attr_name": {
          "terms": {
            "field": "attributeStrings.facetName"
          },
          "aggs": {
            "attr_value": {
              "terms": {
                "field": "attributeStrings.facetValue",
                "size": 1000,
                "order": [
                  {
                    "_term": "asc"
                  }
                ]
              }
            }
          }
        }
      }
    },
    "agg_attr_strings_filtered": {
      "filter": {
        "bool": {
          "filter": [
            {
              "nested": {
                "path": "attributeStrings",
                "query": {
                  "bool": {
                    "filter": [
                      {
                        "term": {
                          "attributeStrings.facetName": {
                            "value": "Property"
                          }
                        }
                      },
                      {
                        "terms": {
                          "attributeStrings.facetValue": [
                            "Organic"
                          ]
                        }
                      }
                    ]
                  }
                }
              }
            },
            {
              "nested": {
                "path": "attributeStrings",
                "query": {
                  "bool": {
                    "filter": [
                      {
                        "term": {
                          "attributeStrings.facetName": {
                            "value": "Brand"
                          }
                        }
                      },
                      {
                        "terms": {
                          "attributeStrings.facetValue": [
                            "Adidas"
                          ]
                        }
                      }
                    ]
                  }
                }
              }
            }
          ]
        }
      },
      "aggs": {
        "nested": {
          "path": "attributeStrings"
        },
        "aggs": {
          "attr_name": {
            "terms": {
              "field": "attributeStrings.facetName"
            },
            "aggs": {
              "attr_value": {
                "terms": {
                  "field": "attributeStrings.facetValue",
                  "size": 1000,
                  "order": [
                    {
                      "_term": "asc"
                    }
                  ]
                }
              }
            }
          }
        }
      }
    }
  }
}