I am running WSO2 IS 5.0.0. I have the SP for IS 5.0.0 applied along with all the other security patches issued for that version for Identity Server and Carbon 4.2.0. My environment consists of 4 machines that are creating a cluster (using the WKA membership scheme and Load Balancer(AWS ELB) with sticky session enabled). I am using MySQL(not the default H2 database). The machines on which the IS is deployed are Windows Server 2012 R2 (EC2 AWS machines).
I am constantly receiving in the console log files "Deployment synchronization commit for tenant -1234 failed". After I applied the changes that @ycr proposed in order to disable the Dep-Sync:
<DeploymentSynchronizer>
<Enabled>false</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>false</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://svnrepo.example.com/repos/</SvnUrl>
<SvnUser>username</SvnUser>
<SvnPassword>password</SvnPassword>
<SvnUrlAppendTenantId>false</SvnUrlAppendTenantId>
and restarted all of my machines. I had no issues for about 2 weeks. Then suddenly I received the the same error but with additional stack trace on two of my machines today (29.09.2016):
[2016-09-29 04:42:24,000] ERROR {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - Deployment synchronization commit for tenant -1234 failed
com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! InvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=247, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[172.31.2.242]:4000}, done=false} No response has been send backups-expected: 0 backups-completed: 0 reinvocations: 0
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.newOperationTimeoutException(BasicInvocation.java:782)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.waitForResponse(BasicInvocation.java:760)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:697)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
at com.hazelcast.map.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:257)
at com.hazelcast.map.proxy.MapProxySupport.getInternal(MapProxySupport.java:161)
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53)
at org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:130)
at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:182)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache.getPathID(JDBCPathCache.java:299)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceID(JDBCResourceDAO.java:81)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:151)
at org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.resourceExists(EmbeddedRegistry.java:644)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:293)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777)
at org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108)
at org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ------ End remote and begin local stack-trace ------.(Unknown Source)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
at com.hazelcast.map.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:257)
at com.hazelcast.map.proxy.MapProxySupport.getInternal(MapProxySupport.java:161)
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53)
at org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:130)
at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:182)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache.getPathID(JDBCPathCache.java:299)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceID(JDBCResourceDAO.java:81)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:151)
at org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.resourceExists(EmbeddedRegistry.java:644)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:293)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777)
at org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108)
at org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Are there any other configuration files that I need to tweak?
Thanks in advance.
