0

So we had a node go down two nights ago, and it took the better part of a day to get it back up. I'm still learning so I'm not familiar with the repair options, but based on the article here: http://www.datastax.com/dev/blog/repair-service-opsc4

Can I just enable the repair service? Does this eliminate the need to run nodetool repair manually which is a pain because its manual, and apparently can be quite I/O intensive which gives me a lot of concern.

Also, assuming this service works in the real world as described in the article can I be confident that starting the service with their suggested interval isn't going to cause performance issues?

zoo_live_crew
  • 45
  • 7

1 Answers1

0

Regular repairs

It is recommended that you run repair often on a weekly basis as an anti-entropy mechanism. You can run into issues (zombie data) if you are not running repair more frequently than your gc grace seconds setting and have a lot of tombstones.

Running repair service in real life

The short answer is yes. The repair service makes your life easier by automating this operation, it also slices the repair into small parts so as to flatten the workload and prevent you from experiencing resource contention and performance issues during repairs. It should work out of the box for most clusters, in some scenarios you may have to tweak some of the default settings. Useful levers include -- max_parallel_repairs, min_repair_time, snapshot_override

Bringing back a downed node

For this particular scenario where you know one node is about a day or so out of sync, I would expect your cluster to be under stress during the repair. Creating Merkel trees is CPU intensive and io intensive. The smaller slices that the repair service calculates should help ease this process.

Community
  • 1
phact
  • 116
  • 3