Scripting in AWS elasticsearch

Question

I read online about scripting in aws elacticsearch service. It said that AWS ES doesn't support dynamic scripting so I am writing aggregations using scripts stored in my disk. I wrote the following query

{
    "query":{
        "match_all":{}
    },
    "aggs":{
        "inBoundRecieved":{
            "scripted_metric":{
                "init_script":{
                    "file": "init.groovy"
                },
                "map_script": {
                    "file": "map.groovy"
                },
                "combine_script": {
                    "file":"comb.groovy"
                },
                "params":{
                    "field":"call_direction"
                },
                "reduce_script": {
                    "file": "red.groovy"
                }
            }
        }
    }

}

But I keep getting this error.

Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]: [init_script]

I have searched a lot online but couldn't find a good solution. Full Error ->

{ 
 "error" : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][0]:                 RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[plivoredshift][0]: query[ConstantScore(*:*)],  from[-1],size[-1]: Parse Failure [Failed to parse source [{    \"query\":{        \"match_all\":{}    },    \"aggs\":{        \"inBoundRecieved\":{                       \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                \"combine_script\": {                    \"file\":\"comb.groovy\"                },                \"params\":  {                    \"field\":\"call_direction\"                },                \"reduce_script\": {                    \"file\": \"red.groovy\"                       }            }        }    }}]]]; nested: SearchParseException[[plivoredshift][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a          START_OBJECT in [inBoundRecieved]: [init_script].]]; }{[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][1]: RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/ search[phase/query]]]; nested: SearchParseException[[plivoredshift][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{             \"query\":{        \"match_all\":{}    },    \"aggs\":{        \"inBoundRecieved\":{            \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                                \"combine_script\": {                    \"file\":\"comb.groovy\"                },                \"params\":{                    \"field\":                             \"call_direction\"                },                \"reduce_script\": {                    \"file\": \"red.groovy\"                }            }        }    }}]]];     nested: SearchParseException[[plivoredshift][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]:         [init_script].]]; }{[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][2]: RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested:      SearchParseException[[plivoredshift][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{    \"query\":{        \"match_all\":{}     },    \"aggs\":{        \"inBoundRecieved\":{            \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.                      groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                \"combine_script\":              {                    \"file\":\"comb.groovy\"                },                \"params\":{                    \"field\":\"call_direction\"                               },                \"reduce_script\": {                    \"file\": \"red.groovy\"                }            }        }    }}]]]; nested:                               SearchParseException[[plivoredshift][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]: [init_script].  ]]; }{[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][3]: RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested:                    SearchParseException[[plivoredshift][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{    \"query\":{        \"match_all\":{}     },    \"aggs\":{        \"inBoundRecieved\":{            \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.                      groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                \"combine_script\":              {                    \"file\":\"comb.groovy\"                },                \"params\":{                    \"field\":\"call_direction\"                               },                \"reduce_script\": {                    \"file\": \"red.groovy\"                }            }        }    }}]]]; nested:                               SearchParseException[[plivoredshift][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]: [init_script].
"status":400
}

Here are my scripts init.groovy

_agg['transactions'] = []

map.groovy

if (doc['call_direction']=="inbound" {_agg.transactions.add(1)} else {_agg.transactions.add(0)}

comb.groovy

inBoundRecieved=0; for( t in _agg.transactions) {inBoundRecieved+=t}; return inBoundRecieved

red.groovy

inBoundRecieved=0; for( a in _aggs) {inBoundRecieved += a}; return inBoundRecieved

I have been following this tutorial from the ElacticSearch website

Dynamic scripting is not available in the Amazon managed ES service, whether inline or via script files. [see this](http://kirankoduru.github.io/elasticsearch/moving-from-aws-elasticsearch-service.html) — Val, May 30 '16 at 14:15
so you mean to say there is no way of adding scripts to aws search queries? — sidd607, May 30 '16 at 14:18
That's correct. As far as my experience goes, the AWS managed service is good enough for storing logs or for some very simple ES usage, but if you need some advanced ES features, you're better off rolling our own. — Val, May 30 '16 at 14:19
In that case can you help me. I have a field "CallDirection" that contains either INBOUND or OUTBOUND. I need to return the number of records that have callDirection=INBOUND and number of records where callDirection=OUTBOUND, can you help me create a query for this result? — sidd607, May 30 '16 at 14:23
Why not simply using a `terms` aggregation on the `callDirection` field? — Val, May 30 '16 at 14:24

score 4 · Answer 1 · answered Mar 24 '17 at 15:36

Looks like scripting can be used in AWS elasticsearch service for version 5 now:

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-resources.html

https://forums.aws.amazon.com/thread.jspa?threadID=217896&start=25&tstart=0

Scripting in AWS elasticsearch

1 Answers1