4

My understanding is when using memcached in "consistent hashing" mode, when a node is down, the clients will remap the key with the algo.

Consider I have 2 clients (A,B) and two servers (C,D)

What if client A think server C is down, but B think C is still alive?

So A will keep using C & D, but B will only use D. How to solve the data inconsistency arise from this case?

Howard
  • 19,215
  • 35
  • 112
  • 184

2 Answers2

1

You have to make sure that servers A and B see the same state of memcached instances. I think that it can be achieved when memcached instances (C & D) will be hidden behind proxy (moxi or twemproxy). The proxy will be responsible for maintaining state of memcached instances. Application servers (A & B) should connect only to the proxy.

ragnor
  • 2,498
  • 1
  • 22
  • 25
0

I use a central "availability" server which runs its own memcached daemon and provides a key called "avail_servers", which returns an array of all available memcached servers. This is refreshed constantly.

My webservers first connect to this central server, get the list of available servers from it, and then use that to store/retrieve values. You could write in some additional checks so that if your "A" server still sees a server in the "available" list as down, it stops providing services until the problem is resolved.

ChunkyBaconPlz
  • 580
  • 2
  • 10
  • Ok fine but doing so you've introduced a single point of failure. How did you solve this issue then? – DevLounge Aug 31 '16 at 11:54
  • You can use something like Zookeeper, instead of a centralized server for keeping the mappings of the alive servers. All servers will have to heartbeat periodically to maintain their "leases". – Dimos Feb 02 '17 at 13:42