5

I have a single node Cassandra installation on my development machine (and very little experience with Cassandra). I always had very few data in the node and I experienced no problems. I inserted about 9,000 elements in a table today to experiment with a real world use case. When I start up the node the boot time is extremely long now. I get this in system.log

Replaying /var/lib/cassandra/commitlog/CommitLog-3-1388134836280.log
...
Log replay complete, 9274 replayed mutations

That took 13 minutes and is hardly bearable. I wonder if there is a way to store data in such a way that can be read at once without replaying the log. After all 9,000 elements are nothing and there must be a quicker way to boot. I googled for hints and searched into Cassandra's documentation but I didn't find anything. It's obvious that I'm not looking for the right things, would anybody be so kind to point me to the right documents? Thanks.

pmontrasio
  • 541
  • 6
  • 10

1 Answers1

9

There are a few things that might help. The most obvious thing you can do is flush the commit log before you shutdown Cassandra. This is a good idea to do in production too. Before I stop a Cassandra node in production I'll run the following commands:

nodetool disablethrift
nodetool disablegossip
nodetool drain

The first two commands gracefully shut down connections to clients connected to this node and then to other nodes in the ring. The drain command flushes memtables to disk (sstables). This should minimize what needs to be replayed on startup.

There are other factors that can make startup take a long time. Cassandra opens all the SSTables on disk at startup. So the more column families and SSTables you have on disk the longer it will take before a node is able to start serving clients. There was some work done in the 1.2 release to speed this up (so if you are not on 1.2 yet you should consider upgrading). Reducing the number of SSTables would probably improve your start time.

Since you mentioned this was a development machine I'll also give you my dev environment observations. On my development machine I do a lot of creating and dropping column families and key spaces. This can cause some of the system CFs to grow significantly and eventually cause a noticeable slowdown. The easiest way to handle this is to have a script that can quickly bootstrap a new database and blow away all the old data in /var/lib/cassandra.

psanford
  • 5,580
  • 1
  • 26
  • 25
  • 1
    Those three commands solved my problem. It replayed 0 mutations on startup and it booted quickly. I used them in the past but I switched to `sudo service cassandra start|stop` lately. It seems that it doesn't shutdown Cassandra in the right way. By the way, I'm on 2.0.3. One more question. The only way to terminate the process seems to kill it. Is there a nodetool command to do it more gracefully? – pmontrasio Dec 28 '13 at 08:10
  • There is no nodetool command to actually stop Cassandra. I run `service cassandra stop` after running the above commands to stop the process. – psanford Dec 28 '13 at 14:33