2

We are using cassandra 2.1.14. Currently large partition warning is seen on system.hints table.

How to make sure that system.hints table doesn't have wide partitions ? Note that we don't want to upgrade to cassandra 3 now.

Is there a periodic way to clean up system.hints ?

Will this cause I/O spike in cassandra cluster ?

Log:

 Compacting large partition system/hints:
 10ad72eb-0240-4b94-b73e-eb4dc2aa759a (481568345 bytes)
Aaron
  • 55,518
  • 11
  • 116
  • 132
Knight71
  • 2,927
  • 5
  • 37
  • 63
  • 3
    https://www.slideshare.net/ClmentLARDEUR/deep-into-cassandra-data-repair-mechanisms – Ashraful Islam Oct 03 '17 at 10:32
  • As for cleaning up the hints table there should be a default 3 hours TTL. – raam86 Oct 03 '17 at 12:25
  • I was curious about why the hints table was so large? Did you have a node or nodes down for an extended period? Once your cluster is fully functioning, the hints should get passed to the correct node and then TTL'd out of the table. – LHWizard Oct 04 '17 at 18:45
  • It is getting TTL after 3hours . But even before that we are seeing this is getting filled up. From the logs we could see many unlogged batches. We are suspecting this to cause large system.hints partition. We are not sure though – Knight71 Oct 05 '17 at 08:07

1 Answers1

4

How to make sure that system.hints table doesn't have wide partitions?

There's not really much you can do about that. system.hints is partitioned on target_id which is the hostID of the target node. If 10000 hints build up for one node, there's really no other place for them to go.

Is there a periodic way to clean up system.hints?

As mentioned above, hints should TTL after 3 hours. This failsafe is meant to keep the system.hints table from getting too out of control. But it's not at all fool-proof.

One way to be sure, would be to clear them out via nodetool:

nodetool truncatehints

Will this cause I/O spike in cassandra cluster?

Running nodetool truncatehints is fairly innocuous. I haven't noticed a spike from running it before.

Aaron
  • 55,518
  • 11
  • 116
  • 132
  • Thanks for the answer. My question is will the I/O spike be caused by system.hints large partition ? – Knight71 Oct 06 '17 at 10:24