Can't run two standalone instances on one server that use local Infinispan containers

Question

It seems I am unable to run two standalone servers on one server, for an application that uses local Infinispan containers. This fails because both instances will start up the same default local Infinispan cache containers, and one tries to register them within the JGroups context of the other (because even though JGroups is not included in the standalone configuration, using Infinispan seems to boot up a JGroups instance listening to a local port that disobeys the jboss.socket.binding.port-offset).

It fails with:

  2018-02-16 15:25:51,045 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "iris-server.war")]) - failure description: {  
      "WFLYCTL0080: Failed services" => {"jboss.persistenceunit.\"iris-server.war#athena\"" => "org.jboss.msc.service.StartException in service jboss.persistenceunit.\"iris-server.war#athena\": javax.persistence.PersistenceException: [PersistenceUnit: athena] Unable to build Hibernate SessionFactory  
      Caused by: javax.persistence.PersistenceException: [PersistenceUnit: athena] Unable to build Hibernate SessionFactory  
      Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.spi.CacheImplementor]  
      Caused by: org.hibernate.cache.CacheException: HHH025009: Unable to start region factory  
      Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport  
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport  
      Caused by: org.infinispan.commons.CacheException: Channel connected, but unable to register MBeans  
      Caused by: javax.management.MBeanRegistrationException: The @MBean objectName is not unique  
      Caused by: javax.management.InstanceAlreadyExistsException: org.infinispan:type=protocol,cluster=\"infinispan-hibernate-cluster\",protocol=UNICAST3"},  
      "WFLYCTL0412: Required services that are not installed:" => ["jboss.persistenceunit.\"iris-server.war#athena\""],  
      "WFLYCTL0180: Services with missing/unavailable dependencies" => undefined  
  }

And the full stack trace:

2018-02-16 15:25:41,404 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 66) MSC000001: Failed to start service jboss.persistenceunit."iris-server.war#athena": org.jboss.msc.service.StartException in service jboss.persistenceunit."iris-server.war#athena": javax.persistence.PersistenceException: [PersistenceUnit: athena] Unable to build Hibernate SessionFactory
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:179)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:121)
    at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:667)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:193)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: athena] Unable to build Hibernate SessionFactory
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.persistenceException(EntityManagerFactoryBuilderImpl.java:967)
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:892)
    at org.jboss.as.jpa.hibernate5.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:161)
    ... 7 more
Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.spi.CacheImplementor]
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:271)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:233)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:210)
    at org.hibernate.service.internal.SessionFactoryServiceRegistryImpl.getService(SessionFactoryServiceRegistryImpl.java:80)
    at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:244)
    at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:452)
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:889)
    ... 9 more
Caused by: org.hibernate.cache.CacheException: HHH025009: Unable to start region factory
    at org.hibernate.cache.infinispan.InfinispanRegionFactory.start(InfinispanRegionFactory.java:476)
    at org.hibernate.cache.spi.RegionFactory.start(RegionFactory.java:63)
    at org.hibernate.internal.CacheImpl.<init>(CacheImpl.java:71)
    at org.hibernate.engine.spi.CacheInitiator.initiateService(CacheInitiator.java:28)
    at org.hibernate.engine.spi.CacheInitiator.initiateService(CacheInitiator.java:20)
    at org.hibernate.service.internal.SessionFactoryServiceRegistryImpl.initiateService(SessionFactoryServiceRegistryImpl.java:59)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:259)
    ... 15 more
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
    at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:249)
    at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:663)
    at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:310)
    at org.hibernate.cache.infinispan.InfinispanRegionFactory.createCacheManager(InfinispanRegionFactory.java:517)
    at org.hibernate.cache.infinispan.InfinispanRegionFactory.createCacheManager(InfinispanRegionFactory.java:513)
    at org.hibernate.cache.infinispan.InfinispanRegionFactory.start(InfinispanRegionFactory.java:464)
    ... 21 more
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
    at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:859)
    at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:628)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:617)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:542)
    at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:236)
    ... 26 more
Caused by: org.infinispan.commons.CacheException: Channel connected, but unable to register MBeans
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:217)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:191)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
    ... 31 more
Caused by: javax.management.MBeanRegistrationException: The @MBean objectName is not unique
    at org.jgroups.jmx.JmxConfigurator.internalRegister(JmxConfigurator.java:184)
    at org.jgroups.jmx.JmxConfigurator.register(JmxConfigurator.java:122)
    at org.jgroups.jmx.JmxConfigurator.registerChannel(JmxConfigurator.java:59)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:214)
    ... 37 more
Caused by: javax.management.InstanceAlreadyExistsException: org.infinispan:type=protocol,cluster="infinispan-hibernate-cluster",protocol=UNICAST3
    at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.registerMBean(PluggableMBeanServerImpl.java:1527)
    at org.jboss.as.jmx.PluggableMBeanServerImpl.registerMBean(PluggableMBeanServerImpl.java:871)
    at org.jgroups.jmx.JmxConfigurator.internalRegister(JmxConfigurator.java:181)
    ... 40 more

Hopefully someone can help me with constructing a standalone.xml workaround so that both instances have an isolated JGroups instance as well.

Hmm, that's strange. Which WildFly version? Have you tried WildFly 11? Can you provide some more detailed instructions on replicating this? — Galder Zamarreño, Feb 26 '18 at 17:22
I am on WF 10.1, still have to migrate to 11. Steps to reproduce: (1) boot under standalone or standalone-full profile (2) deploy an application with second level caching enabled, (3) copy JBOSS_HOME, (4) boot up the instance from the copied folder with an arbitrary port offset set through the VM variables (5) notice that this second instance will fail to boot the application, because it tries to register its local Infinispan caches in the JGroups cluster of the other standalone server. — Jan-Willem Gmelig Meyling, Feb 26 '18 at 18:01
(it actually does surprise me that Infinispan causes an JGroups instance to be booted up even though JGroups is not configured under a non-HA profile) — Jan-Willem Gmelig Meyling, Feb 26 '18 at 18:03
Have you tried copying before starting the instance? Also try downloading a fresh copy of WF 10.1. I'd suggest downloading it again, unzip/untar it to the two locations, then do the booting... — Galder Zamarreño, Mar 07 '18 at 13:43
Yes my steps do include setting up a clean JBOSS_HOME folder booted with a port offset. — Jan-Willem Gmelig Meyling, Mar 07 '18 at 13:44

score 2 · Accepted Answer · answered Mar 16 '18 at 22:49

2

What Hibernate region factory are you using? Can you post your hibernate.cfg.xml?

Given the stack trace, I would guess that you are using org.hibernate.cache.infinispan.InfinispanRegionFactory, when you ought to be using org.jboss.as.jpa.hibernate5.infinispan.InfinispanRegionFactory. This implementation was designed to use the cache configurations included in WildFly's Infinispan subsystem.

answered Mar 16 '18 at 22:49

Paul Ferraro

326
1
3

That could very well be it! I found the documentation slightly unclear about what was the preferred RegionFactory. I am using: https://gist.github.com/JWGmeligMeyling/df4c1847bfb2eb45008a924a4c53f482#file-persistence-xml-L120-L126 – Jan-Willem Gmelig Meyling Mar 16 '18 at 22:51
Which documentation were you looking at? This should be documented here: https://docs.jboss.org/author/display/WFLY10/JPA+Reference+Guide#JPAReferenceGuide-UsingtheInfinispansecondlevelcache – Paul Ferraro Mar 18 '18 at 15:54
Hmm I see that the current documentation is pretty clear. Back in the day (I suppose half a year ago, as 10.x was still the newest version), I suppose I copied the properties from a quickstart example repository somewhere on Github (probably one of the wildfly or infinispan quickstart). Ill try to find the original resource back. Nevertheless, Ill rollout the updated configuration today and if the issue is resolved (what I do expect), Ill mark your answer as correct. Thanks for your help! – Jan-Willem Gmelig Meyling Mar 18 '18 at 18:46

Can't run two standalone instances on one server that use local Infinispan containers

1 Answers1