4

I've read the relevant documentation I could find, but I still have doubts.

What I read

From http://wiki.apache.org/cassandra/Operations#Moving_nodes

If you add nodes to your cluster your ring will be unbalanced and only way to get perfect balance is to compute new tokens for every node and assign them to each node manually by using nodetool move command.

and from http://www.datastax.com/docs/1.1/operations/cluster_management#adding-capacity-to-an-existing-cluster

If you need to increase capacity by a non-uniform number of nodes, you must recalculate tokens for the entire cluster, and then use nodetool move to assign the new tokens to the existing nodes. After all nodes are restarted with their new token assignments, run a nodetool cleanup to remove unused keys on all nodes

But I'm not clear on the order of these things.

Could you explain how to do it in the following scenario?

  • I'm using cassandra 1.1.9, so no virtual nodes are in use.
  • I have a cluster ring with 5 nodes, and each owns 20%
  • Their tokens are
    • 0
    • 34028236692093846346337460743176821145
    • 68056473384187692692674921486353642291
    • 102084710076281539039012382229530463436
    • 136112946768375385385349842972707284582

I want to add 2 additional nodes.

What steps do I have to follow? I know I should install and configure cassandra, use the original 5 as seeds, and calculate their new tokens, but in what order should I move the data using nodetool move? Is it one at a time?
What happens with the data when I move the first one? Is it available at all times?
Should I start the two new nodes before moving the original 5 to their new tokens?

A step by step guide would be ideal.

Please note that I need to do it pre version 1.2

juan
  • 80,295
  • 52
  • 162
  • 195

2 Answers2

4

The new tokens should be

  • 0
  • 24305883351495604533098186245126300818
  • 48611766702991209066196372490252601636
  • 72917650054486813599294558735378902454
  • 97223533405982418132392744980505203272
  • 121529416757478022665490931225631504090
  • 145835300108973627198589117470757804908

calculated using 2^127/7 * {0-7}.

What steps do I have to follow? in what order should I move the data using nodetool move?

You should

  1. Bootstrap in one node at 48611766702991209066196372490252601636
  2. Bootstrap the other node at 121529416757478022665490931225631504090
  3. Move 34028236692093846346337460743176821145 to 24305883351495604533098186245126300818
  4. Move 68056473384187692692674921486353642291 to 72917650054486813599294558735378902454
  5. Move 102084710076281539039012382229530463436 to 97223533405982418132392744980505203272
  6. Move 136112946768375385385349842972707284582 to 145835300108973627198589117470757804908

(I tried to minimise the amount of data transferred - might not be optimal but is close enough to not make much difference given the inbalance of data you probably have already.)

Is it one at a time?

You should bootstrap one node and once and move one token at once. This avoids placing excess load on the cluster while streaming data.

What happens with the data when I move the first one? Is it available at all times?

Data is fully available during the move. The node participates in reads and writes for the old and new range so you can read and write during the move.

Should I start the two new nodes before moving the original 5 to their new tokens?

Always better to have more nodes in the cluster - if you moved first, you'd have some nodes with twice as much data as the others.

Richard
  • 11,050
  • 2
  • 46
  • 33
  • So, I should add `48611766702991209066196372490252601636` to one of the new nodes, `121529416757478022665490931225631504090` to the other, start them up, and only then use `nodetool move` to migrate date from step 3 forward? – juan Mar 18 '13 at 21:25
  • Right, you never need to restart the original 5. Set the initial_token and seeds in the yaml for the new ones and start them up one by one. – Richard Mar 18 '13 at 21:25
0

From Cassandra 1.2, keeping a cluster balanced when adding nodes is very easy, because of the new vnodes (multiple seeds per node) feature. Cassandra now automatically balances the cluster for you. If you upgrade from an earlier version you will have to activate the vnode feature yourself

Community
  • 1
  • 1
Raedwald
  • 46,613
  • 43
  • 151
  • 237