Getting null_pointer_exception when accessing _aggs elements in Scripted Metric Aggregation

Question

I have an index of stock ticks for every second that look like the following and I am trying to aggregate them together into larger ticks of 5 minutes. Aggregating the high and low is easy using min/max. However the open and close are harder. I am trying to use the Scripted Metric Aggregation but I am getting a null pointer exception that I can't explain.

{ "time": "2017-12-06 12:02:00", "high": 10, "low": 1, "open": 1, "close": 5}

My query looks like the following, although I've removed a lot from combine_script and reduce_script just so it is easy to read.

{
    "size": 0,
    "aggs": {
        "ticks": {
            "date_histogram": {
                "field":"time",
                "interval":"5m"
            },
            "aggs": {
                "open": {
                    "scripted_metric": {
                        "init_script": "params._agg.opens = []",
                        "map_script": "params._agg.opens.add(['time': doc.time.value.getMillis(),'open':doc.open.value])",
                        "combine_script": "return params._agg.opens[0]",
                        "reduce_script":  "return params._aggs[0]"
                    }
                }
            }
        }
    }
}

If I run the query you can see it outputs a Map (with the keys time and open) for the "open" value as you'd expect.

{
  "aggregations": {
    "ticks": {
      "buckets": [
        {
          "key_as_string": "2017-12-06 14:20",
          "key": 1512570000000,
          "doc_count": 6,
          "volume": {
            "value": 84.09597664
          },
          "high": {
            "value": 12886
          },
          "low": {
            "value": 12874.99
          },
          "open": {
            "value": {
              "time": 1512570420000,
              "open": 12874.99
            }
          }
        }
      ]
    }
  }
}

But if I try and access any properties of the map (as a simple example just returning params._aggs[0]['open']) I get a null pointer exception. I've tried accessing it with params._aggs[0].open and even setting it to a map and then accessing that map Map myMap = params._aggs[0]; return myMap['open']; but I get the same error.

{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "",
    "phase": "fetch",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "script_exception",
      "reason": "runtime error",
      "script_stack": [
        "return params._aggs[0]['open']",
        "                   ^---- HERE"
      ],
      "script": "return params._aggs[0]['open']",
      "lang": "painless",
      "caused_by": {
        "type": "null_pointer_exception",
        "reason": null
      }
    }
  },
  "status": 503
}

This is a problem because I need to do some additional stuff in reduce_script but can't seem to access any properties of the maps in _aggs without getting an NPE.

the problem seems to be `_aggs` vs `_agg`. You create `_agg` but try to access `_aggs` — Val, Dec 07 '17 at 07:41
Thanks @Val but I don't think that is it because it does work until I try and access a property on `_aggs[0]`. They show it as the same way in their docs (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html) as well. I think it is because `_agg` is the shard aggregation, and `_aggs` is a combination of all the `_agg`'s — dustins, Dec 07 '17 at 12:53
Doooh! What happens if you simply return `params._aggs` or leave out the reduce_script entirely? — Val, Dec 07 '17 at 13:01

score 2 · Answer 1 · answered Dec 07 '17 at 13:35

I was able to solve the problem by using the ?. operator. I am guessing because I had some null fields, it was blowing up by trying to access properties on null. So by just adding that, I get what I expect. So it is just the simple matter of params._aggs[0] ?. open

{
    "size": 0,
    "aggs": {
        "ticks": {
            "date_histogram": {
                "field":"time",
                "interval":"5m"
            },
            "aggs": {
                "open": {
                    "scripted_metric": {
                        "init_script": "params._agg.opens= []",
                        "map_script": "params._agg.opens.add(['time': doc.time.value.getMillis(),'open':doc.open.value])",
                        "combine_script": "return params._agg.opens[0]",
                        "reduce_script":  "return params._aggs[0]?.open"
                    }
                }
            }
        }
    }
}

{
  "aggregations": {
    "ticks": {
      "buckets": [
        {
          "key_as_string": "2017-12-07 12:20",
          "key": 1512649200000,
          "doc_count": 2,
          "open": {
            "value": 13
          }
        }
      ]
    }
  }
}

Getting null_pointer_exception when accessing _aggs elements in Scripted Metric Aggregation

1 Answers1