I have setup a 3 nodes cassandra cluster in single machine(Ubantu), And I do the following -
- All three nodes up and application is able to create keyspace.
- Shutdown one out of 3 running nodes and application is still able to create keyspace.
Shutdown one more node out of now 2 running nodes and start it immediately and now application is not able to create keyspace, it keeps on reporting (like infinite loop) about -
2017-09-25 14:42:40 INFO RequestHandler:561 - Query CREATE KEYSPACE IF NOT EXISTS MYKEYSPACE WITH replication = {'class':'NetworkTopologyStrategy', 'datacenter1':'2'}; is not prepared on /127.0.0.1:9042, preparing before retrying executing. Seeing this message a few times is fine, but seeing it a lot may be source of performance problems
My application log is flooded with these messages and it resulted java heap out error.
Note- I have Replication factor as 2, and the version of cassandra using is 2.2.6 with cassandra driver is 2.2.0-rc3. When I revert back to cassandra driver 2.1.3 everything is working fine, so just wondering if some issues at driver level side
Further more to add on this I debugged the code even at cassandra driver level and found this is because of the below given method of com.datastax.driver.core.Connection class -
private void flush(FlushItem item) {
EventLoop loop = item.channel.eventLoop();
Flusher flusher = flusherLookup.get(loop);
if (flusher == null) {
Flusher alt = flusherLookup.putIfAbsent(loop, flusher = new Flusher(loop));
if (alt != null)
flusher = alt;
}
flusher.queued.add(item);
flusher.start();
}
So after bringing back node online (2 nodes running now out of 3), the RequestHandler keeps on reporting about query not prepared, hence preparing before retry when this call is made -
public ResponseHandler write(ResponseCallback callback, boolean startTimeout)
{
....
....
....
flush(new FlushItem(channel, request, writeHandler(request, handler)));
....
....
}
