3

As everyone would agree, Cassandra repairs are necessary but are very expensive and failure prone, gets stuck most of the time if any node in the cluster go down while the repair is running on any other node in the cluster. I am running full sequential repair on primary range using the following command in a rolling fashion:

node repair -pr -full -seq

But have a doubt, Is it enough to run this repair on every node of a data-center (I have 4 different data-centers) or is it required to be run on every node of the whole cluster? I found some documents on this topic, but the language doesn't answer this question properly. For example 3.1 Primary range repair

Pankaj Yadav
  • 139
  • 1
  • 10

2 Answers2

3

Update: I was actually wrong here thinking ring as two DCs instead of single, the actual token Ring is more of:

    | DC  | Node | Token |
    |-----|------|-------|
    | DC1 |node1 |   1   |     
    | DC2 |node2 |   5   |
    | DC1 |node3 |   10  |
    | DC2 |node4 |   15  |
    | DC1 |node5 |   20  |
    | DC2 |node6 |   25  |

The primary range for node4 here is 11-15, not 6-15 (which is the primary range + local ranges). You must do -pr on each node. Deleting original so it doesn't cause any confusion.

Chris Lohfink
  • 16,150
  • 1
  • 29
  • 38
  • What part of the command makes it repair the token ranges owned outside of the DC you execute it in? – Chris Gerlt Oct 11 '17 at 00:58
  • repairs always involve all replicas regardless of DC. When your RF is `{DC1: 1, DC2: 1}` every token will have one replica in each DC. The tokens in arg determine which nodes of which dc are the involved in repair. We split them into two tables for visualization but its good to remember that its really a single list. – Chris Lohfink Oct 11 '17 at 05:03
  • I'm not sure we're talking about the same thing here. For above repair command (nodetool repair -pr) to repair all the replicas, it must be run on all the nodes: Note: If you use this option, you must run nodetool repair -pr on EVERY node in the cluster to repair all data. Otherwise, some ranges of data will not be repaired. (https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesManualRepair.html) – Chris Gerlt Oct 11 '17 at 14:18
  • 1
    `repair -pr` is kind of a shorthand for specifying the token range start/end to be the tokens that the repair coordinator owns as primary ranges (or in case of vnodes, multiple sub repairs for each token range). All the replicas for that range include multiple nodes across all DCs and are included. You also only have to actually do the `-pr` on all the nodes in a DC in the case of the above RF, not _every_ node. Since the sum of the primary ranges in one DC covers the entire token ring. If setup correctly its also possible to only have to do it within one rack of one DC. – Chris Lohfink Oct 11 '17 at 15:25
  • DS docs are wrong in this case btw. but safer to just do it on every node and not assume anything. – Chris Lohfink Oct 11 '17 at 15:34
  • I believe the docs are correct and have seen tests support the documentation. – Chris Gerlt Oct 11 '17 at 19:27
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/156497/discussion-between-chris-lohfink-and-chris-gerlt). – Chris Lohfink Oct 11 '17 at 19:49
  • The docs are correct. See this blog post I wrote for a full write up: https://www.datastax.com/dev/blog/repair-in-cassandra – Zanson Oct 11 '17 at 19:53
3

With repair -pr -full you must run repair on every node in the cluster. See this blog post I wrote a couple years ago for a detailed description of why.

Zanson
  • 3,991
  • 25
  • 31