0

I am running Apache Ignite in a Kubernetes environment as a stateful set. Everything is working as expected in Ignite 2.10. When I tried to upgrade the Ignite version to 2.11 and create a new Ignite cluster, the nodes are not starting up and I get the following error:

[20:47:02,014][SEVERE][main][IgniteKernal] Exception during start processors, node will be stopped and close connections class org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2006) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1270) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2141) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1787) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1172) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1066) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:952) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:851) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:721) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690) at org.apache.ignite.Ignition.start(Ignition.java:353) at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:367) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start Jetty HTTP server. at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:321) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:153) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1012) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:983) at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:534) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2003) ... 11 more Caused by: java.lang.IllegalStateException: No matching constructor class org.eclipse.jetty.server.Server in file:///opt/ignite/apache-ignite/config/custom-jetty.xml at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:454) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:380) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:318) ... 16 more [20:47:02,017][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine). class org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2006) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1270) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2141) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1787) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1172) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1066) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:952) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:851) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:721) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690) at org.apache.ignite.Ignition.start(Ignition.java:353) at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:367) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start Jetty HTTP server. at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:321) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:153) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1012) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:983) at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:534) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2003) ... 11 more Caused by: java.lang.IllegalStateException: No matching constructor class org.eclipse.jetty.server.Server in file:///opt/ignite/apache-ignite/config/custom-jetty.xml at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:454) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:380) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:318) ... 16 more

I use a jetty.xml file which is included in the Ignite configuration as a connector configuration. The jetty.xml has the default settings as provided in the apache ignite webpage. Also the ignite-rest-http folder has been copied to lib folder; the OPTION_LIBS environment variable also has this module to enable it. None of the settings has changed, so not sure what is different in 2.11 to cause this issue. Everything works in Ignite 2.10.

Olaf Kock
  • 46,930
  • 8
  • 59
  • 90
Ana
  • 3
  • 3
  • What version of Jetty? and what does your `file:///opt/ignite/apache-ignite/config/custom-jetty.xml` contents look like? – Joakim Erdfelt Nov 06 '21 at 16:28
  • Jetty comes as part of the apache-ignite libs. Apache Ignite 2.11 used jetty-server-9.4.39.v20210325. The jetty.xml I tried with is the same that is given here: https://ignite.apache.org/docs/latest/restapi#example-jetty-xml-configuration – Ana Nov 08 '21 at 05:13

1 Answers1

2

You can fix it by changing threadPool argument -> threadpool in jetty XML configuration.

This is a bug in Ignite documentation. I've created a JIRA for this. See details here

solveMe
  • 1,866
  • 1
  • 18
  • 20
  • Thanks for the guidance, it worked. I am jus wondering what changed in Jetty server. We have been having the argument as threadPool in jetty.xml all along so far. I don't see any change in the Jetty Server class constructors between the releases too. – Ana Nov 08 '21 at 23:15
  • It seems, that starting from 9.4.27 version, Jetty has more strict validation of the provided configuration files. Before that version, an incorrectly spelled property in the configuration file had no effect. The following issue might be the cause https://github.com/eclipse/jetty.project/issues/4550 (I'm not 100% sure, though). – sk0x50 Nov 09 '21 at 20:18