Elastic - Filter after selecting top 5 hits

Question

I'm using the alerting feature in Kibana and I want to check if the last 5 consecutive values of a field exceed a threshold x but if I use a filter in my elastic query, it gets applied before the top N aggregation.

Is there a way in which I can apply the filter after or check if the last consecutive values exceed a threshold using some other selector or method? I don't want to check this in the trigger condition in painless because that will return all the documents in the ctx and not just the ones which exceeded the threshold which I want to display in my alert message.

I've been stuck with this for a while and I have only seen blog posts saying sub aggregation is not possible on top N so any help or work around would be much appreciated.

This is my query :

{ 
    "size": 500, 
    "query": { 
        "bool": { 
            "filter": [ 
                { 
                    "match_all": {
                        "boost": 1 
                    } 
                }, 
                {
                    "match_phrase": { 
                        "client.id": { 
                            "query": "42", 
                            "slop": 0, 
                            "zero_terms_query": "NONE", 
                            "boost": 1 
                        } 
                    } 
                }, 
                { 
                    "range": { 
                        "@timestamp": { 
                            "from": "{{period_end}}||-10m", 
                            "to": "{{period_end}}", 
                            "include_lower": true, 
                            "include_upper": true, 
                            "format": "epoch_millis", 
                            "boost": 1 
                        } 
                    }
                } 
            ], 
            "adjust_pure_negative": true, 
            "boost": 1 
        } 

    }, 
    "aggs": {
        "2": {
          "terms": {
            "field": "component.name",
            "order": {
              "_key": "desc"
            },
            "size": 50
          },
          "aggs": {
            "3": {
              "terms": {
                "field": "client.name.keyword",
                "order": {
                  "_key": "desc"
                },
                "size": 5
              },
              "aggs": {
                "1": {
                  "top_hits": {
                    "docvalue_fields": [
                      {
                        "field": "gc.oldgen.used",
                        "format": "use_field_mapping"
                      }
                    ],
                    "_source": "gc.oldgen.used",
                    "size": 5,
                    "sort": [
                      {
                        "@timestamp": {
                          "order": "desc"
                        }
                      }
                    ]
                  }
                }
              }
            }
          }
        }
      }
    }       
}

score 0 · Answer 1 · answered Aug 18 '20 at 12:03

0

Did you try to use a sub filter aggregation: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html

Or you can use a pipeline aggregation to manipulate your aggregations results https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html

by the way, a term query on the client id looks more appropriate.

answered Aug 18 '20 at 12:03

Jaycreation

2,029
1
15
30

I want to check if all 5 values of component.stats.gc.oldgen.used.pct fetched in the above query are greater than 40. I don't know how I can do this using a sub aggregation. – amat_coder Aug 18 '20 at 12:56
Hum, after some tests, the issue is that we can't to a sub aggregation on top hits https://stackoverflow.com/questions/38195420/elasticsearch-aggregation-over-top-hits So you probably won't be able to test the value on the 5 last results of your query directly and would need to script this externally – Jaycreation Aug 19 '20 at 08:49

Elastic - Filter after selecting top 5 hits

1 Answers1