1

When Cassandra is doing the data integrity check, it does a validation compaction, but what does this mean exactly? My understanding is that it creates a single SSTable that will be stored temporarily (until the repair finishes), and then it generates the Merkle trees from that single created SSTable. If any of the Merkle trees leafs fails validation, then the partitions used to create that leaf (from the SSTable created during the validation compaction) will be streamed to the other node. However, a friend told me that the Merkle trees are generated from each (previously existing) SSTable.

So, how many Merkle trees are generated, one or as many as SSTables?

Pedro Gordo
  • 1,825
  • 3
  • 21
  • 45

1 Answers1

3

The validation compaction iterates over all the sstables that are included in the range to build the merkle tree. It doesn't actually write a new sstable, but the compaction interfaces perform same type of task (iterating over data) so its reused. The compaction manager is also used for cleanup, secondary index rebuild, MV building, scrubbing, and verify processes.

A single merkle tree is generated. Each node of it represents a hash of all the data in a token range, each child of the node is half of its token range. The depth of the tree is dynamic, ideally the leaf represents 1 partition each but it could end up representing much more if the root node represents a wide range containing many partitions. Since the depth of the merkle tree is capped at 20 (or else it will be too large, and cause issues transferring) you generally dont want to repair a range that has much more than 2^20 or 1 million partitions in it. Can use getsplits or the size_estimates table to determine this when picking how to subdivide range for a subrange repair.

Worth noting, that a repair can kick off many sub repairs, each will have its own validation compaction/merkle tree/streaming session.

Chris Lohfink
  • 16,150
  • 1
  • 29
  • 38
  • So if a leaf fails validation, where does the node reads the data that it needs to stream? Will it reread every SSTable to find those partitions used to generate that leaf? – Pedro Gordo Dec 05 '17 at 17:06
  • 1
    It will get all current sstables for the range that the leaf represents and stream it over. If there are new sstables thats Ok due to the idempotent, immutable nature of the data. sstables are sorted in token order so it can jump right to where the range starts. – Chris Lohfink Dec 05 '17 at 17:19
  • Cheers! Last question :) It doesn't stream the SSTables themselves, right? I suppose it reads all the SSTables for the leaf range, crunches the data (like it was a regular read operation) and then sends it over, right? – Pedro Gordo Dec 05 '17 at 17:23
  • 1
    the streaming session will have a task per sstable, but it will only stream over the part of that sstable that is in the range that was different. – Chris Lohfink Dec 05 '17 at 18:06