0

Current setting, cassandra 2.2.5, gossip is 1 second default and phi threshold value is 8. The problem, I am facing is spikes in hints. And one of the reason hints goes up is when node is marked down (gossip has not communicated for phi threshold value).

I read one article, where it say phi threshold value of 8 corresponds to 18 seconds, it will be few seconds here or there. Now I need to understand what is the reason, what is blocking gossip to communicate for 18 seconds. What is the checklist that need to be satisfied for gossip to communicate?

Varun Gupta
  • 1,419
  • 6
  • 28
  • 53

1 Answers1

2

  • Re: "How does cassandra gossip protocol and phi_threshold works?": Phi is approximated as: phi = (tnow - tLast) / mean and a node is marked down when phi > phi_threshold / 0.434. For your settings (and assuming a mean of 1 [as in the node usually receives the heartbeat 1 second apart]) a node will be marked down if we didn't receive any heartbeats from it for 8 / 0.434 = 18.42 seconds. The paper documenting the algorithm can be found here.

  • Re: "What is the checklist that needs to be satisfied for gossip to communicate?": to me there are a few things:

    • the network: the gossip messages being dropped or the gossip port (7000/7001) being blocked;
    • the nodes themselves: the nodes is busy/unresponsive (i.e. doing GC, doing some heavy load operation) so they don't get to send any/much gossip messages.
Eugen Constantin Dinca
  • 8,994
  • 2
  • 34
  • 51
  • It is possible to configure Gossip to use different port? – Varun Gupta Oct 11 '16 at 17:52
  • It's the `storage_port` config value. To me it would make more sense to open the 7000 port in your firewall before changing the value. – Eugen Constantin Dinca Oct 11 '16 at 18:32
  • We have 7000 open, my concern is that due to congestion on storage port gossip messages are not communicated and it leads to believe other nodes that this node is down and as a result lot of hints are produced. So I was thinking, if gossip can work on other port it will make things simpler. – Varun Gupta Oct 11 '16 at 18:35
  • I doubt that changing the port would make much of a difference, it's going to be the same traffic. Or maybe I'm misunderstanding what you mean by congestion. Also, when you say `hints goes up`, which metric is that? `TotalHintsInProgress`? – Eugen Constantin Dinca Oct 11 '16 at 22:35
  • By metric, I mean total hints count. As of now, I believe 7000 port is used for gossip message and other data traffic (heavy traffic = congestion at that port). So if we can separate that then gossip will be communicated to other nodes and thereby all nodes will be seen as up, so no node will generate hint for that scenario. – Varun Gupta Oct 12 '16 at 17:25
  • 1
    Yes, port 7000 is used for any C* inter-node communication (gossip, data streaming, coordination, ...). Changing the value of `storage_port` will *not* result in just the gossip traffic happening on this other port, it just means all C* inter-node communication traffic happening now on this other port. – Eugen Constantin Dinca Oct 13 '16 at 16:33
  • I agree, so my suggestion is that going further can we separate ports for gossip, schema coordination & other light weight operations on one port and data streaming, hints replaying on other. – Varun Gupta Oct 13 '16 at 17:26
  • Afaik you *cannot* do that kind of separation of traffic. [as a side note keep in mind that the `total hints count` doesn't go down unless the node gets reset https://github.com/apache/cassandra/blob/trunk/doc/source/operating/metrics.rst] – Eugen Constantin Dinca Oct 13 '16 at 17:53
  • Link is broken. – Varun Gupta Oct 13 '16 at 18:16