Elasticsearch Cardinality Aggregation giving completely wrong results

Question

I am saving each page view of a website in an ES index, where each page is recognized by an entity_id. I need to get the total count of unique page views since a given point in time. I have the following mapping:

{
   "my_index": {
      "mappings": {
         "page_views": {
            "_all": {
               "enabled": true
            },
            "properties": {
               "created": {
                  "type": "long"
               },
               "entity_id": {
                  "type": "integer"
               }
            }
         }
      }
   }
}

According to the Elasticsearch docs, the way to do that is using a cardinality aggregation. Here is my search request:

GET my_index/page_views/_search
{
  "filter": {
    "bool": {
      "must": [
        [
          {
            "range": {
              "created": {
                "gte": 9999999999
              }
            }
          }
        ]
      ]
    }
  },
  "aggs": {
    "distinct_entities": {
      "cardinality": {
        "field": "entity_id",
        "precision_threshold": 100
      }
    }
  }
}

Note, that I have used a timestamp in the future, so no results are returned. And the result I'm getting is:

{
   "took": 2,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 0,
      "max_score": null,
      "hits": []
   },
   "aggregations": {
      "distinct_entities": {
         "value": 116
      }
   }
}

I don't understand how the unique page visits could be 116, giving that there are no page visits at all for the search query. What am I doing wrong?

Sloan Ahrens · Accepted Answer · 2014-12-12T23:50:23.920

Your aggregation is returning the global value for the cardinality. If you want it to return only the cardinality of the filtered set, one way you could do that is to use a filter aggregation, then nest your cardinality aggregation inside that. Leaving out the filtered query for clarity (you can add it back in easily enough), the query I tried looks like:

curl -XPOST "http://localhost:9200/my_index/page_views/_search " -d'
{
   "size": 0, 
   "aggs": {
      "filtered_entities": {
         "filter": {
            "bool": {
               "must": [
                  [
                     {
                        "range": {
                           "created": {
                              "gte": 9999999999
                           }
                        }
                     }
                  ]
               ]
            }
         },
         "aggs": {
            "distinct_entities": {
               "cardinality": {
                  "field": "entity_id",
                  "precision_threshold": 100
               }
            }
         }
      }
   }
}'

which returns:

{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 4,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "filtered_entities": {
         "doc_count": 0,
         "distinct_entities": {
            "value": 0
         }
      }
   }
}

Here is some code you can play with:

http://sense.qbox.io/gist/bd90a74839ca56329e8de28c457190872d19fc1b

I used Elasticsearch 1.3.4, by the way.

Elasticsearch Cardinality Aggregation giving completely wrong results

1 Answers1