installing x-pack stops elasticsearch from running

Question

Centos VM guest on Win10 host. Attempting to follow installation guide for x-pack (and step by step from the training course). ES and K running fine, but once x-pack plugin installed, kibana will start (and complains ES is missing - that's fine), then elastic search starts and immediately fails. It looks like the same error as 3, but in a local VM, not a production setting. Here's the tail of the log:

[2017-01-31T16:52:03,191][ERROR][o.e.b.Bootstrap          ] Exception
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:384) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.1.1.jar:5.1.1]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
        ... 14 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/x-pack/system_key
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
        at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
        at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_111]
        at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:160) ~[?:?]
        at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
        at org.elasticsearch.xpack.security.Security.<init>(Security.java:191) ~[?:?]
        at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:193) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
        ... 14 more
[2017-01-31T16:52:03,215][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elkmaster1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]]; nested: InvocationTargetException; nested: AccessDeniedException[/etc/elasticsearch/x-pack/system_key];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.1.jar:5.1.1]
Caused by: org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:384) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/x-pack/system_key
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
        at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
        at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_111]
        at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:160) ~[?:?]
        at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
        at org.elasticsearch.xpack.security.Security.<init>(Security.java:191) ~[?:?]
        at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:193) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
    ... 6 more

score 1 · Answer 1 · answered Jan 31 '17 at 06:14

1

try it out:

open config/elasticsearch.yml file
change xpack configuration line

xpack.security.enabled: false xpack.graph.enabled: false xpack.watcher.enabled: false

answered Jan 31 '17 at 06:14

Vignesh Ammasi

381
1
4
10

score 0 · Accepted Answer · answered Jan 31 '17 at 19:50

0

Well since the log contains

Caused by: java.nio.file.AccessDeniedException:

The problem is possibly the user permissions. Who is the owner of this directory?

/etc/elasticsearch/x-pack

What does ls -l /etc/elasticsearch/x-pack give you for an output?

If you have sudo permissions or have root access, you might just change the ownership of this directory.

answered Jan 31 '17 at 19:50

cinhtau

1,002
8
16

thanks! this was a step forward for sure. the system_key file was : `-rw-------. 1 root root 128 Dec 9 16:29 system_key` so chmod 604 on the file to get at least read rights for elastic, and the service was able to start. The default user accounts don't login yet, but ill review the correct permissions further. This certainly helped solve the main problem – Thomas Humphries Feb 01 '17 at 00:42
Since it is a key it is better to change the ownership to elastic with `sudo chown elastic:elastic system.key`. Most security programs check the permission set. I would recommend not to give others a read access. This should work for you. HTH – cinhtau Feb 01 '17 at 16:51
understood, and will verify. the chmod was to prove the problem area... Now, however, having the 'too many local redirects', so may need to change my platform to allow for more nodes to troubleshoot. – Thomas Humphries Feb 05 '17 at 00:01

score 0 · Answer 3 · answered Sep 12 '17 at 12:24

0

The problem is with the user accounts you used to install elasitcsearch and x-pack. Make sure you used the same account and have the right permissions to do so!

You might want to grant access using chown.

answered Sep 12 '17 at 12:24

rohithnama

239
2
5

installing x-pack stops elasticsearch from running

3 Answers3