I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper 3.4.8
I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more than one node in cloud mode with Solr 6.2.0.
I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper, and three Solr nodes.
The first Solr node connects to Zookeeper as expected. I can access the admin and see the Cloud info there.
But when I try and connect with a second node, something goes wrong.
I get the following error in the Solr log:
2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized @490ms
2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor
[file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0
2016-08-30 20:12:12.906 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support
for /solr, did not find org.apache.jasper.servlet.JspServlet
2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
has uncovered http methods for path: /
2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init():
WebAppClassLoader=1465085305@57536d79
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured
for solr (NoInitialContextEx)
2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property
solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader
for directory: '/home/vagrant/solr-6.2.0/server/solr'
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured
for solr (NoInitialContextEx)
2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property
solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client
to connect to ZooKeeper
2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181
got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected
to ZooKeeper
2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from
SolrHome (not found in ZooKeeper)
2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration
from /home/vagrant/solr-6.2.0/server/solr/solr.xml
2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core
root directory: /home/vagrant/solr-6.2.0/server/solr
2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New CoreContainer 182259421
2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer
[instanceDir=/home/vagrant/solr-6.2.0/server/solr]
2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from /home/vagrant/solr-6.2.0/server/solr/lib
to classpath: /home/vagrant/solr-6.2.0/server/solr/lib
2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with
socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize
: 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries
: false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime : 40000,
2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler
HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j
(org.slf4j.impl.Log4jLoggerFactory)]
2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181
2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new OnReconnect listener
org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client
to connect to ZooKeeper
2016-08-30 20:12:13.549 INFO (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [
] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected
to ZooKeeper
2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating cluster state
from ZooKeeper...
2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (1)
2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr.
Check solr/home property and the logs
2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException:
A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other
Solr process using the same port is running already.
at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init()
done
2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms
It does start in the end, but it's not working properly, I just get errors in the Solr admin.
The only reference I can find to the the error:
"A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already."
,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
https://issues.apache.org/jira/browse/SOLR-8777
Has anyone else experienced this issue? Any hints on resolving?
EDIT: There are definitely no existing connections to Solr before trying to connect the second node. Each node is on a totally separate VM, so the ports are not clashing (which would result in a different failure anyway). I have even used pgrep -f solr to double check. Nothing.
