0

I am trying to set up a device monitoring site using KAA application server. In a cluster setup.

We are using 4 server cluster.

Server 1 has the following components

  1. Monogo DB (Which is running fine)
  2. Kaa-node (Control and Admin --running fine)
  3. Zoo keeper (running fine)

Server 2 has the following components

  1. Monogo DB (Which is running fine)
  2. Mongo Shard server (running fine)
  3. Kaa-node (Operations and Bootstrap --running fine)
  4. Zoo keeper (running fine)

Server 3 has the following components

  1. Monogo DB (Which is running fine)
  2. Mongo Config server (running)
  3. Kaa-node (Operations,Bootstrap,Admin and Control --running fine)
  4. Zoo keeper (running fine)

Server 4 has following components

  1. Device monitoring app (custom java application)

  2. Event listener (not working) Following things are happening in event listener Allocated operational server When trying to issue Create Sync request, get a response line ConACK(REFUSE_SERVER_UNAVAILABLE) Connection got lost.

    Please see the exact event listener log

Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2]INFO o.k.k.c.c.i.DefaultOperationDataProcessor - Created Sync request: {"requestId": 19658, "syncReﾢ￁ﾈ#ﾱﾑ﾿￡'"}, "profileHash": {"bytes": "/c￷ￇﾎDﾎￇwￄﾬ@ﾼﾚﾨ.a9"}, "timeout": 60000}, "bootstrapSyncRequest": null, "profileSyncRequest": null, "configurationSyncRequest": {"configurationHash": {"bytes": "JL￢￶#ﾱ%ﾕﾍﾖXﾈﾲﾪ;ﾰAN*"}, "resyncOnly": true}, "notificationSyncRequest": {"topicListHash": 251, "topicStates": [{"topicId": 220, "seqNumber": 7}], "acceptedUnicastNotifications": null, "subscriptionCommands": []}, "userSyncRequest": {"userAttachRequest": {"userVerifierId": "41465538310519253645", "userExternalId": "CIOuserverifier", "userAccessToken": "41465538310519253645"}, "endpointAttachRequests": [], "endpointDetachRequests": []}, "eventSyncRequest": {"eventSequenceNumberRequest": {}, "eventListenersRequests": null, "events": null}, "logSyncRequest": {"requestId": 0, "logEntries": null}, "extensionSyncRequests": null} Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] INFO o.k.k.c.c.i.c.DefaultOperationTcpChannel - Channel [default_operation_tcp_channel] is reading data from stream using [1024] byte buffer Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] INFO o.k.k.c.c.i.c.DefaultOperationTcpChannel - ConnAck (REFUSE_SERVER_UNAVAILABLE) message received for channel [default_operation_tcp_channel] Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] ERROR o.k.k.c.c.i.c.DefaultOperationTcpChannel - Connection for channel [default_operation_tcp_channel] was rejected: REFUSE_SERVER_UNAVAILABLE Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] INFO o.k.k.c.c.i.c.DefaultOperationTcpChannel - [default_operation_tcp_channel] has failed Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] INFO o.k.k.c.c.i.c.DefaultOperationTcpChannel - Channel "default_operation_tcp_channel": closing current connection Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] INFO o.k.k.c.c.i.DefaultFailoverManager - Server [OPERATIONS, 161946944] failed Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] WARN o.k.k.c.b.DefaultBootstrapManager - Failed to find server for channel [TransportProtocolId [id=1456013202, version=1]] Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] WARN o.k.k.c.b.DefaultBootstrapManager - Attempt to receive operations server list will be made in 2000 ms, according to failover strategy decision Sat Oct 15 2016 12:19:54 PM [pool-6-thread-2] INFO o.k.k.c.c.i.c.DefaultOperationTcpChannel - Read Task is interrupted for channel [default_operation_tcp_channel]

Also see the Trace log for the Kaa operations server

2016-10-15 12:01:22,739 [EPS-io-dispatcher-7] TRACE o.k.k.s.sync.platform.AvroEncDec - Decoding client sync {"requestId": 18586, "syncRequestMetaDﾢ￁ﾈ#ﾱﾑ﾿￡'"}, "profileHash": {"bytes": "/c￷ￇﾎDﾎￇwￄﾬ@ﾼﾚﾨ.a9"}, "timeout": 60000}, "bootstrapSyncRequest": null, "profileSyncRequest": null, "configurationSyncRequest": {"configurationHash": {"bytes": "JL￢￶#ﾱ%ﾕﾍﾖXﾈﾲﾪ;ﾰAN*"}, "resyncOnly": true}, "notificationSyncRequest": {"topicListHash": 251, "topicStates": [{"topicId": 220, "seqNumber": 7}], "acceptedUnicastNotifications": null, "subscriptionCommands": []}, "userSyncRequest": {"userAttachRequest": {"userVerifierId": "41465538310519253645", "userExternalId": "CIOuserverifier", "userAccessToken": "41465538310519253645"}, "endpointAttachRequests": [], "endpointDetachRequests": []}, "eventSyncRequest": {"eventSequenceNumberRequest": {}, "eventListenersRequests": null, "events": null}, "logSyncRequest": {"requestId": 0, "logEntries": null}, "extensionSyncRequests": null} 2016-10-15 12:01:22,739 [EPS-io-dispatcher-7] TRACE o.k.k.s.sync.platform.AvroEncDec - Decoded client sync ClientSync [requestId=18586, clientSyncMetaData=ClientSyncMetaData [applicationToken=null, sdkToken=0DmQxaMY6JVI0d2fJSh1kCSB80U, endpointPublicKeyHash=java.nio.HeapByteBuffer[pos=0 lim=20 cap=20], profileHash=java.nio.HeapByteBuffer[pos=0 lim=20 cap=20], timeout=60000], bootstrapSync=null, profileSync=null, forceConfigurationSync=false, configurationSync=ConfigurationClientSync [configurationHash=java.nio.HeapByteBuffer[pos=0 lim=20 cap=20], resyncOnly=true], forceNotificationSync=false, notificationSync=NotificationClientSync [topicListHash=251, topicStates=[TopicState [topicId=220, seqNumber=7]], acceptedUnicastNotifications=null, subscriptionCommands=[]], userSync=UserClientSync [userAttachRequest=org.kaaproject.kaa.server.sync.UserAttachRequest@c48c69c1, endpointAttachRequests=[], endpointDetachRequests=[]], eventSync=EventClientSync [seqNumberRequest=true, eventListenersRequests=null, events=null], logSync=LogClientSync [requestId=0]] 2016-10-15 12:01:22,739 [EPS-io-dispatcher-7] DEBUG o.k.k.s.o.s.c.c.ConcurrentCacheService - Fetching result for sdk token: 0DmQxaMY6JVI0d2fJSh1kCSB80U to retrieve application token 2016-10-15 12:01:22,739 [EPS-io-dispatcher-7] DEBUG o.k.k.s.c.d.i.s.HibernateSdkProfileDao - Searching for an SDK profile by token: [0DmQxaMY6JVI0d2fJSh1kCSB80U] 2016-10-15 12:01:22,739 [EPS-io-dispatcher-7] TRACE o.k.k.s.c.d.i.s.HibernateAbstractDao - Searching SdkProfile entity by criterion [token=0DmQxaMY6JVI0d2fJSh1kCSB80U] 2016-10-15 12:01:22,740 [EPS-io-dispatcher-7] TRACE o.k.k.s.o.s.a.a.i.EncDecActorMessageProcessor - Request processing failed java.lang.IllegalStateException: Cache Operation Exception at org.kaaproject.kaa.server.operations.service.cache.concurrent.CacheTemporaryMemorizer.launderThrowable(CacheTemporaryMemorizer.java:106) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.cache.concurrent.CacheTemporaryMemorizer.compute(CacheTemporaryMemorizer.java:79) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.cache.concurrent.ConcurrentCacheService.getAppTokenBySdkToken(ConcurrentCacheService.java:760) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActorMessageProcessor.getAppToken(EncDecActorMessageProcessor.java:370) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActorMessageProcessor.addAppTokenToClientSyncMetaData(EncDecActorMessageProcessor.java:362) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActorMessageProcessor.decodePlatformLevelData(EncDecActorMessageProcessor.java:315) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActorMessageProcessor.decodeEncryptedRequest(EncDecActorMessageProcessor.java:251) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActorMessageProcessor.decodeRequest(EncDecActorMessageProcessor.java:238) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActorMessageProcessor.processSignedRequest(EncDecActorMessageProcessor.java:182) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActorMessageProcessor.decodeAndForward(EncDecActorMessageProcessor.java:99) ~[kaa-node-0.8.0.jar:na] at org.kaaproject.kaa.server.operations.service.akka.actors.io.EncDecActor.onReceive(EncDecActor.java:144) [kaa-node-0.8.0.jar:na] at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.4.1.jar:na] at akka.actor.Actor$class.aroundReceive(Actor.scala:480) [akka-actor_2.11-2.4.1.jar:na] at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.4.1.jar:na] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.1.jar:na] at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.1.jar:na] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.1.jar:na] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.1.jar:na] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.1.jar:na] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na] Caused by: java.util.concurrent.ExecutionException: org.springframework.orm.hibernate4.HibernateSystemException: No Session found for current thread; nested exception is org.hibernate.HibernateException: No Session found for current thread at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_91] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_91] at org.kaaproject.kaa.server.operations.service.cache.concurrent.CacheTemporaryMemorizer.compute(CacheTemporaryMemorizer.java:75) ~[kaa-node-0.8.0.jar:na] ... 21 common frames omitted Caused by: org.springframework.orm.hibernate4.HibernateSystemException: No Session found for current thread; nested exception is org.hibernate.HibernateException: No Session found for current thread at org.springframework.orm.hibernate4.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:216) ~[spring-orm-4.0.2.RELEASE.jar:4.0.2.RELEASE] at org.springframework.orm.hibernate4.HibernateExceptionTranslator.convertHibernateAccessException(HibernateExceptionTranslator.java:57) ~[spring-orm-4.0.2.RELEASE.jar:4.0.2.RELEASE] at org.springframework.orm.hibernate4.HibernateExceptionTranslator.translateExceptionIfPossible(HibernateExceptionTranslator.java:44) ~[spring-orm-4.0.2.RELEASE.jar:4.0.2.RELEASE] at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:59) ~[spring-tx-4.0.2.RELEASE.jar:4.0.2.RELEASE] at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:213) ~[spring-tx-4.0.2.RELEASE.jar:4.0.2.RELEASE] at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:147) ~[spring-tx-4.0.2.RELEASE.jar:4.0.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.0.2.RELEASE.jar:4.0.2.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.0.2.RELEASE.jar:4.0.2.RELEASE] at com.sun.proxy.$Proxy91.findSdkProfileByToken..

  1. Alert monitor
  2. Schedule monitor

Servers are in AWS instance.

Checked all ports.

Verified all component configurations.

Any help will be greatly appreciated.

Thanks Jinu

Jpec
  • 300
  • 2
  • 14

1 Answers1

0

Can you send all logs of clients and kaa nodes. Also please check transpot_public_intefase parameter on all nodes in /etc/kaa-node/conf/kaa-node.properties(transpot_public_intefase="ip of current machine") Also you can use Cluster setup guide

Vladimir Chizhevsky
  • 41
  • 3