0

I am currently using elastic for storing time series of of data where I am storing the incoming packets( it is incremental counter) on a port->network device .

I am doing aggregations to calculate number of packets received during a time interval(histogram) and also I want to calculate the total number of packets received on the queried interval .

Number of packets received during time interval is derivative of maximum running counter . Total number of packets received on the queried interval is cumulative sum of derivatives .

{
  "query" : {
    "bool" : {
      "must" : {
        "range" : {
          "exportTimeStamp" : {
            "from" : 1523826005514,
            "to" : 1523862005514,
            "include_lower" : true,
            "include_upper" : true
          }
        }
      }
    }
  },

aggregations" : {
        "dataPoints" : {
          "date_histogram" : {
            "field" : "exportTimeStamp",
            "interval" : "5m",
            "min_doc_count" : 0
          },
          "aggregations" : {
            "Max" : {
              "max" : {
                "field" : "ingressPackets"
              }
            },
            "Der" : {
              "derivative" : {
                "buckets_path" : [ "Max" ],
                "gap_policy" : "insert_zeros"
              }
            },
            "CumSum" : {
              "cumulative_sum" : {
                "buckets_path": "Der"
              }
            }
          }
        }

    }
}

I am getting following error since for the first aggregation there will no derivative populated 

{
    "error": {
        "root_cause": [],
        "type": "reduce_search_phase_exception",
        "reason": "[reduce] ",
        "phase": "fetch",
        "grouped": true,
        "failed_shards": [],
        "caused_by": {
            "type": "null_pointer_exception",
            "reason": null
        }
    },
    "status": 503
}

Result with out "CumSum" aggregation snippet .

"buckets": [
                            {
                                "key_as_string": "2018-04-16T06:20:00.000Z",
                                "key": 1523859600000,
                                "doc_count": 1,
                                "Max": {
                                    "value": 58
                                }
//I think the problem is here where we dont have derivative and hence when we add cumulative sum aggregation we are getting NPE.
                            },
                            {
                                "key_as_string": "2018-04-16T06:25:00.000Z",
                                "key": 1523859900000,
                                "doc_count": 3,
                                "Max": {
                                    "value": 169
                                },
                                "Der": {
                                    "value": 111
                                }.....

Please clarify how to solve NPE in this case ?

Naidu
  • 245
  • 1
  • 4
  • 13
  • following is the stack trace from elastic – Naidu Apr 16 '18 at 07:58
  • It's not the most elegant but you could add a [Bucket Script](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/search-aggregations-pipeline-bucket-script-aggregation.html) that gives back the value of `Der` or `0` if it's not there and point the `cumulative_sum` aggregation to that. – Frank Apr 16 '18 at 09:38
  • Thanks Frank, I will give a try..Between What could be better approch here ..My requirement is to calculate rate of of change(derivative) on the running counter (gauge )and sum of all the rate of changes(cumulative sum) based on time interval. – Naidu Apr 17 '18 at 04:50
  • You can just do the same with a min and max aggregation on the whole time frame and a bucket script that would do `value of max agg - value of min agg` right? – Frank Apr 17 '18 at 09:48

0 Answers0