0

While starting kudu-master, I am getting the below error and unable to start kudu cluster.

F0706 10:21:33.464331 27576 master_main.cc:71] Check failed: _s.ok() Bad status: Invalid argument: Unable to initialize catalog manager: Failed to initialize sys tables async: on-disk master list (hadoop-master:7051, slave2:7051, slave3:7051) and provided master list (:0) differ. Their symmetric difference is: :0, hadoop-master:7051, slave2:7051, slave3:7051

It is a cluster of 8 nodes and i have provided 3 masters as given below in master.gflagfile on master nodes.

    --master_addresses=hadoop-master,slave2,slave3
Sangeeta
  • 491
  • 5
  • 22

1 Answers1

0

TL;DR

If this is a new installation, working under the assumption that master ip addresses are correct, I believe the easiest solution is to

  1. Stop kudu masters
  2. Nuke the <kudu-data-dir>/master directory
  3. Start kudu masters

Explanation

I believe the most common (if not only) cause of this error (Failed to initialize sys tables async: on-disk master list (hadoop-master:7051, slave2:7051, slave3:7051) and provided master list (:0) differ.) is when a kudu master node gets added incorrectly. The error suggests that kudu-master thinks it's running on a single node rather than 3-node cluster.

Maybe you did not intend to "add a node", but that's most likely what happened. I'm saying this because I had the same problem; after some googling and debugging, I discovered that during the installation, I started kudu-master before putting the correct IP address in master.gflagfile, so that kudu-master was spun up thinking it was running on a single node, not 3 node. Using steps above to clean install kudu-master again, my problem was solved.

clony
  • 136
  • 2