0

I have upgraded DSE cluster with 2 nodes from 5.0.7 to 6.7.3. After upgrade with nodetool status shows both nodes are "UP NORMAL" with apprx 75 GB load on each and cluster works for applications with read write. but getting error during

  1. Nodetool repair -pr some repair failed
  2. Upgrade sstable makes node down.

and observing exception every 10 seconds in system.log file

WARN  [OptionalTasks:1] 2019-07-18 08:20:14,495  CassandraRoleManager.java:386 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2019-07-18 08:20:14,495  CassandraRoleManager.java:432 - Setup task failed with error, rescheduling
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
        at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:392)
        at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:214)
        at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:190)
        at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1541)
        at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1524)
        at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1447)
        at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1325)
        at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1274)
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:366)
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:574)
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:307)
        at org.apache.cassandra.cql3.QueryProcessor.lambda$processStatement$4(QueryProcessor.java:256)
        at io.reactivex.internal.operators.single.SingleDefer.subscribeActual(SingleDefer.java:36)
        at io.reactivex.Single.subscribe(Single.java:2700)
        at io.reactivex.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:34)
        at io.reactivex.Single.subscribe(Single.java:2700)
        at io.reactivex.Single.blockingGet(Single.java:2153)
        at org.apache.cassandra.concurrent.TPCUtils.blockingGet(TPCUtils.java:75)
        at org.apache.cassandra.cql3.QueryProcessor.processBlocking(QueryProcessor.java:352)
        at org.apache.cassandra.auth.CassandraRoleManager.hasExistingRoles(CassandraRoleManager.java:396)
        at org.apache.cassandra.auth.CassandraRoleManager.setupDefaultRole(CassandraRoleManager.java:370)
        at org.apache.cassandra.auth.CassandraRoleManager.doSetupDefaultRole(CassandraRoleManager.java:428)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
SLU
  • 81
  • 9
  • What errors are in the log when you run upgradesstables? I'm guessing something is going on that is bringing the node down. Assuming you can't log in with cqlsh? If both logs could be "attached", it could provide more information. – Jim Wartnick Jul 18 '19 at 12:29
  • During repair ERROR [AntiEntropyStage:1] 2019-07-18 06:11:52,515 VerbHandlers.java:77 - Unexpected error during execution of request REPAIR.VALIDATION_COMPLETE During sstabale upgrade INFO [RMI TCP Connection(84)-127.0.0.1] 2019-07-18 06:29:29,967 DseDaemon.java:831 - DSE shutting down... – SLU Jul 28 '19 at 07:34

0 Answers0