Changing commitlog directory in cassandra

Question

Currently commitlog directory is pointing to Directory1. I want to change it different directory D2. How should the migration be ?

Answer 1

This is how we did it. We have a load-balanced client that talks to Cassandra 1.1.2, and each client lives on each Cassandra node.

1. Drain your service.
   • Wait for the load balancer to remove the node.
2. Stop your service on the local node to halt direct Client-Cassandra writes: systemctl stop <your service name>
3. At this point there should be no more writes and greatly reduced disk activity:
   • iostat 2 - Disk activity should be near zero
   • nodetool gossipinfo
4. Disable Cassandra gossip protocol to mark the node dead and halt Cassandra-Cassandra writes: nodetool disablegossip
5. Flush all contents of the commit log into SSTables: nodetool flush
6. Drain the node – this command is more important than nodetool flush, (and might include all the behaviour of nodetool flush): nodetool drain
7. Stop the cassandra process: systemctl stop cassandra
8. Modify Cassandra config file(s), e.g. vi /etc/cassandra/default.conf/cassandra.yaml
9. Start Cassandra: systemctl start cassandra
10. Wait 10-20 minutes. Tail Cassandra logs to follow along, e.g. tail -F /var/log/cassandra/system.log
11. Confirm ring is healthy before moving on to next node: nodetool ring
12. Re-start client service: systemctl start <your service here>

Note that there was no need for us to do manual copying of the commitlog files themselves. Flushing and draining took care of that. The files then slowly reappeared in the new commitlog_dir location.

Answer 2

You can change the commit log directory in cassandra.yaml (key: "commitlog_directory") and copy all logs to the new destination (see docs) :

commitlog_directory

commitlog_directory The directory where the commit log is stored. Default locations:

    Package installations: /var/lib/cassandra/commitlog
    Tarball installations: install_location/data/commitlog

For optimal write performance, place the commit log be on a separate disk partition, or (ideally) a separate physical device from

the data file directories. Because the commit log is append only, an HDD is acceptable for this purpose.

---

If you are using bitnami/cassandra containers, this should be done using this env var (see docs):

CASSANDRA_COMMITLOG_DIR: Directory where the commit logs will be stored. Default: /bitnami/cassandra/data/commitlog