4

I wish to update my ES version from 2.3 to 5.0.0-alpha4 to be able to use Ingest nodes and remove Logstash out of the question. But it seems ES 5.x version won't start without me setting vm.max_map_count to 262144. I don't want to set that value..I am okay with default value 65530. Can anyone guide me how to get ES 5.x started without tampering memory settings at all. I don't have access to root user on the host on which i wish to install ES.
Error:

java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:347)
at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
[2016-08-08 07:49:55,436][INFO ][node ] [data-cum-ingest-node] initializing ...
[2016-08-08 07:49:56,048][INFO ][plugins ] [data-cum-ingest-node] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-08-08 07:49:56,601][INFO ][env ] [data-cum-ingest-node] heap size [7.9gb], compressed ordinary object pointers [true]
[2016-08-08 07:49:57,582][INFO ][node ] [data-cum-ingest-node] initialized
[2016-08-08 07:49:57,582][INFO ][node ] [data-cum-ingest-node] starting ...
[2016-08-08 07:49:57,635][ERROR][bootstrap ] [data-cum-ingest-node] Exception
java.lang.RuntimeException: bootstrap checks failed
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
at org.elasticsearch.node.Node.start(Node.java:373)
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
at org.elasticsearch.cli.Command.main(Command.java:53)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
Suppressed: java.lang.IllegalStateException: max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:126)
The Real Bill
  • 14,884
  • 8
  • 37
  • 39
Mrunal Gosar
  • 4,595
  • 13
  • 48
  • 71

3 Answers3

3

Starting with ES 5, there are some bootstrap checks that are enforced.

One of those checks is the Maximum map count check (on Linux only), which requires 262144 memory-mapped areas or it will refuse to start.

Note that this check is enforced whatever the environment ES runs in (dev or prod)

Val
  • 207,596
  • 13
  • 358
  • 360
  • 1
    is there any way to bypass all the checks enforced by ES 5.x. I am using ES 5.x to try and use Ingest Nodes feature so that i can remove Logstash as it is quite heavy and does not gives me desired performance? – Mrunal Gosar Aug 09 '16 at 06:36
  • Not related, but have you been through the Logstash [performance guidelines](https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html)? – Val Aug 09 '16 at 06:40
  • Yes I had been through the guidelines..but tuning of Logstash required more hardware and high end config machines..I was wondering if this post holds true https://sematext.com/blog/2016/04/25/elasticsearch-ingest-node-vs-logstash-performance/ then i may not need to procure new hardware. Logstash is really CPU extensive and if i deploy to our prod..it might happen that it eats up CPU performance – Mrunal Gosar Aug 09 '16 at 06:48
  • Update: I tried it with 5.1.1 and it is still having that check, even in dev mode. This prevents me from running tests in travis.ci :( Has anyone an idea how to override this? – vanthome Dec 16 '16 at 14:26
1

From documentation:

If Elasticsearch is in development mode, any bootstrap checks that fail appear as warnings in the Elasticsearch log. If Elasticsearch is in production mode, any bootstrap checks that fail will cause Elasticsearch to refuse to start.

Again from there:

Thus, we consider Elasticsearch to be in development mode if it does not bind nor publish to an external interface (the default), and is otherwise in production mode if it does bind or publish to an external interface.

To disable checks completely just try to set bootstrap.ignore_system_bootstrap_checks to true.

https://www.elastic.co/guide/en/elasticsearch/reference/master/bootstrap-checks.html

alpert
  • 4,500
  • 1
  • 17
  • 27
  • are you referring to network.host property..I've set it to _global_. Not sure which binding is it. is there any way to bypass all the checks enforced by ES 5.x. I am using ES 5.x to try and use Ingest Nodes feature so that i can remove Logstash as it is quite heavy and does not gives me desired performance – Mrunal Gosar Aug 09 '16 at 06:35
  • Yes, that property. By default it should bind to the loopback and elasticsearch will start in development mode, any custom setttings will let es start in production mode: https://www.elastic.co/guide/en/elasticsearch/reference/master/important-settings.html#network.host – alpert Aug 09 '16 at 06:58
  • If i set that property to localhost.. then it may not accept connections from other hosts from where i wish to send logs via filebeat. Are there any hacks or backdoors in fooling ES? – Mrunal Gosar Aug 09 '16 at 07:03
  • 1
    Further seaarching just try to set `bootstrap.ignore_system_bootstrap_checks` to `true`. – alpert Aug 09 '16 at 07:22
  • Good find @alpert! – Val Aug 09 '16 at 08:19
  • I tried using bootstrap.ignore_system_bootstrap_checks: true but it still checks. I had ES_JAVA_OPTS set to -Xms1g -Xmx8g and it started crying – Mrunal Gosar Aug 09 '16 at 10:20
  • weird you can see: https://www.elastic.co/blog/elasticsearch-5-0-0-alpha3-released and https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/bootstrap/BootstrapSettings.java. I dont understand what is wrong. Did you set it on config file? Is the error same? – alpert Aug 09 '16 at 10:33
  • the property i've set it in config file at the bottom..but ES_JAVA_OPTS i am setting it as environment variable and trying to run – Mrunal Gosar Aug 10 '16 at 05:44
  • Same in the sense..it shows java.lang.RuntimeException: bootstrap checks failed minimum and maximum should be set same..although previous error went away..but this is not good for the bootstrap checks as well..why does it needs to keep startingup memory same as maximum memory? – Mrunal Gosar Aug 10 '16 at 06:10
0

Development Mode: If you are running elastic search in development mode we don't need to set the max_map_count. Just run as follows

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:5.6.4

Production Mode: : I f you run in production you need to set the property as below.

Linux

The vm.max_map_count setting should be set permanently in /etc/sysctl.conf:

grep vm.max_map_count /etc/sysctl.conf

vm.max_map_count=262144

OR

sysctl -w vm.max_map_count=262144

Community
  • 1
  • 1
Jinna Balu
  • 6,747
  • 38
  • 47