0

I have a 6-node DSE Search datacenter with vnodes disabled. This is previously a 2-node DC and the other 4 nodes were just recently added. Now, to rebalance the DC, I want to move one of the original nodes into a different position in the ring. BUT, that node doesn't have enough free space to accommodate the move - the drive fills to 100% during the streaming phase.

Adding a new node to replace the node that I want to move is currently not possible for me. What other options do I have? My DC's RF is 2. I wonder if:

  1. I can safely run nodetool cleanup on the node before attempting to move (in the hope that cleanup would free up enough space to facilitate the move)*
  2. I can safely remove the node via nodetool removenode and re-add it as a new node

I'm using DSE 4.0.3 (Cassandra 2.0.7); Murmur3Partitioner

*I'm actually following this guide but the cleanup doesn't come until the move is completed. Can I interchange steps 6 and 7 for this particular node?

ir0nman143
  • 25
  • 1
  • 6
  • Are all the Nodes the same size? If they are not and you cant free UP enough space then you may need to migrate the smallest Node to the ring to balance the DC's. Did you check for any Archive data or Log Files you can Backup and then Delete before you Migrate to free up space? -- It looks like you can run the Cleanup Command from the nodeTool at any time --- http://www.datastax.com/docs/1.1/references/nodetool#nodetool – Tasos Jul 01 '14 at 09:22
  • The nodes have identical hardware. The only data the nodes contain are the sstable and log files. I'm not sure what archive data you are referring to. If there is anything that I can delete, it would probably just be a few gigs of old log files but that still wouldn't be enough to accommodate the move. My main concern about doing _cleanup_ before _move_ is that the move _might_ need to stream first some of the files that will be deleted by the cleanup – ir0nman143 Jul 01 '14 at 09:50
  • Just to give a perspective: Most of my nodes have 750 - 900 gb load (as shown by cassandra). The actual disk usage (df -h) is around 1.2 to 1.3 tb per node. Each node has 1.8 tb total space; so that leaves 500+ gb free space in the node that I want to move. But during the streaming phase, all of the free space was quickly exhausted. – ir0nman143 Jul 01 '14 at 09:56
  • Do the nodes have to be on the same hard drive, if not get another hard drive. – Tasos Jul 01 '14 at 22:57

1 Answers1

0

Did you run cleanup after expanding the cluster originally? If not, then running cleanup up front should help and reduce the data size. Your other option if the other nodes have room would be to decommission the node, and then bootstrap it back in the new location. But the other nodes in your cluster need to have enough room for the decommission to complete. The final option is to add another data folder in your cassandra.yaml if you can add some more storage to the node. DSE/Cassandra can spread the data across multiple mount points.

Zanson
  • 3,991
  • 25
  • 31