How to get value from aggregation buckets in Java for elasticsearch aggregation query result

Question

So, I have been able to replicate the elasticsearch query as desired in Java with the elasticsearch high level restclient. The problem is that I cannot retrieve the values I want. Before I give the code, I want to address the overarching goal in case there is a much easier solution (seems like this shouldn't have to be so difficult.)

Overarching goal: get number of documents where 'visited'==true for each unique value in the 'recommender' field.

My current status: I have been able to write a query with the desired output in kibana/elasticsearch, but when I replicate this query in Java, I am unable to access the data I need.(verified with searchRequest.source().toString()).

Here is the query:

{
  "aggs":{
    "recommenderIDs": {
      "terms": {
        "field": "recommender"
      },
      "aggs": {
        "visit_stats": {
          "filters": {
            "filters": {
              "visited": {
                "match":{
                  "visited": true
                }
              }
            }
          }
        }
      }
    }
  }
}

And this is what I have in my java code:

// ...
        SearchRequest searchRequest = new SearchRequest(INDEX_REC_RECOMMENDATIONS);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        String aggregationName = "recommenderId";
        String filterName = "wasVisited";
        TermsAggregationBuilder aggQuery = AggregationBuilders
                .terms(aggregationName)
                .field(RecommendationRepoFieldNames.RECOMMENDER);
        AggregationBuilder aggFilters = AggregationBuilders.filters(
                filterName,
                new FiltersAggregator.KeyedFilter(
                        RecommendationRepoFieldNames.RECOMMENDER,
                        QueryBuilders.termQuery(RecommendationRepoFieldNames.VISITED, true))
        );
        aggQuery.subAggregation(aggFilters);
        searchSourceBuilder.aggregation(aggQuery);
        searchRequest.source(searchSourceBuilder);
//        System.out.println(searchRequest.source().toString());
        try {
            SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
            Aggregations aggregations = searchResponse.getAggregations();
            Terms byRecommenderId = aggregations.get(aggregationName);
            Filters filterResponses = searchResponse.getAggregations().get(aggregationName);
//            for (Filters.Bucket entry : filterResponses.getBuckets()) {
//                String key = entry.getKeyAsString();
//            }
            for (Terms.Bucket bucket : byRecommenderId.getBuckets()) {
                String bucketKey = bucket.getKeyAsString();
                long totalDocs = bucket.getDocCount();
                Aggregation visitedDocs = bucket.getAggregations().get(filterName);
                //long visitedDocsCount = visitedDocs.getValue();
                System.out.println();
            }
        } catch (IOException e) { //...

I've been fiddling with this all day and cannot make any progress. It's especially frustrating because I can see the doc count for each recommender bucket when I am debugging in my IDE, but I have no idea how to access it. I realize that there are approximately 180 classes extending Aggregation and I have tried a few, but failed every time.

Additionally, if you know any decent resource for the elasticsearch java high level rest client, please let me know. Thank you!

---------EDIT 5/4/21 -------------

Example output from elasticsearch:

// searchResponse (documents returned have been truncated to show only part we are interested in)

  "aggregations": {
    "sterms#recommenderId": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "AdjacentActivityRecommender",
          "doc_count": 3,
          "filters#wasVisited": {
            "buckets": {
              "recommender": {
                "doc_count": 2
              }
            }
          }
        },
        {
          "key": "DefaultProfileDBRecommender",
          "doc_count": 2,
          "filters#wasVisited": {
            "buckets": {
              "recommender": {
                "doc_count": 2
              }
            }
          }
        },
        {
          "key": "PSTR_SC_DI",
          "doc_count": 2,
          "filters#wasVisited": {
            "buckets": {
              "recommender": {
                "doc_count": 1
              }
            }
          }
        },
        {
          "key": "SignificantCategories",
          "doc_count": 2,
          "filters#wasVisited": {
            "buckets": {
              "recommender": {
                "doc_count": 2
              }
            }
          }
        }
      ]
    }
  }

searchResponse.getAggregations() is then saves to aggregations. Eventually, we are able to loop through the buckets for each recommenderID, but I am never able to get into the aggregations inside of each bucket which is what I need to do.

Could you post an example of how elasticsearch gives the result? — Vitor Santos, May 04 '21 at 01:35
Example elastic response added. The doc count is 0 for each bucket which is actually expected in this result set. — redgrengrumbholt, May 04 '21 at 13:53
The Java API is a bit cumbersome. From my experience I would try to get your aggregation and cast it to "MultiBucketsAggregation". Ex: ((MultiBucketsAggregation)aggregations.get("sterms#recommenderId")) . Then you can loop through each bucket using the getBuckets method. Inside each bucket you should be able to call getAggregations(), and that should give you the data you need. That's my best guess. — Vitor Santos, May 04 '21 at 20:29
Thank you @VitorSantos! I actually figured this out a few days ago and it did work. The solution was right under my nose the entire time (agh!!). I need my work machine and I will post the solution here on Monday for posterity. — redgrengrumbholt, May 08 '21 at 03:07

score 5 · Answer 1 · answered May 10 '21 at 21:13

Solution code posted below:

    try {
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        Aggregations aggregations = searchResponse.getAggregations();
        Terms byRecommenderId = aggregations.get(aggregationName);
        for (Terms.Bucket bucket : byRecommenderId.getBuckets()) {
            String recommenderId = bucket.getKeyAsString();
            double totalDocs = bucket.getDocCount();
            // next two lines are the solution:
            Aggregations subAggregations = bucket.getAggregations();
            Filters byWasVisited = subAggregations.get(filterName);
            // always only one item from getBuckets()
            double totalVisited = byWasVisited.getBuckets().get(0).getDocCount();
            double percentVisited = totalVisited / totalDocs;
            recommenderViews.put(recommenderId, percentVisited);
        } 
        // ...

The issue was that I needed to extract the next inner level of aggregations (subAggregations) which is done by calling getAggregations() once more, this time inside of the loop. At this point, we simply call get(filterName) from the subAggregations.

How to get value from aggregation buckets in Java for elasticsearch aggregation query result

1 Answers1