0

We are having a 5 cluster setup for Cassandra 3.0.9. We are seeing outofMemory exception in Cassandra. It is using thrift library API 0.9.2. These outof memory Exceptions are every 2-3 days on random nodes from Cluster.

The Max heap size for each Cassandra process is 8GB and RAM is 32GB.

We tried to Analyze the heap dump and it shows each of the thrift Thread Object is 128MB and there are around 55 threads. These thrift Object are consuming a lot of memory i.e. around 7GB.

Heap Dump Heap DUmp

We are not sure whether there is any Memory leak into the thrift API.

Any help would be really helpful.

JensG
  • 13,148
  • 4
  • 45
  • 55
Pushpak
  • 1
  • 2
  • I had similar problems when using thrift when there was a misbehaving client. There was one huge querry (basically loading of couple of GB into memory) and this caused the nodes to die when trying to respond to this. We added limitations on input query and then it was fine ... I don't know if this will help you but it kind of rings a bell so I wanted to mention it. – Marko Švaljek Feb 03 '17 at 11:04
  • Thanks for your reply Marko. We dont have any huge query in the application. Normally our query would fetch not more than 1MB of data. If we look at the Heap dump there are many such threads each consuming 128MB. Not sure why the thrift threads are consuming so much memory. – Pushpak Feb 03 '17 at 11:39
  • One of our Observation is that we are using a different version thrift library client 0.7.0 and server 0.9.2 . I am not sure whether this can cause any issue. – Pushpak Feb 03 '17 at 13:41

0 Answers0