13

We are deploying memcached for our application, and I would like to make it as resistant as I can.

We plan to use the newer memcacheD extension.

One thing that I haven not fully figured out is, what happens if one of the servers dies. At the very least it seems like the memcached client just 'gives up' on that server, and doesn't store anything in it.

This behavior I'm fine with. We can deal with a bunch of cache-misses. However, it would be nice that after one of the servers is deemed 'failed' the subsequent sets and gets get re-distributed to the servers that are left over.

Since this does not seem to happen automatically; I guess the only way to approach this issue is to have an external system do health-checks on memcached systems, and update the list of servers appropriately.

But if there's a list of 10 servers, and lets say, the 5th dies.. even with the Ketama-hashing it would seem that this would trigger a big redistribution of keys (this is just based on common sense).

So ideally, I would simply like the PHP extension to figure out a server is down, mark it for down for a specified amount of time (10 minutes) and during those 10 minutes fall back to other servers (nicely distributed) for sets and gets.

How do other people solve this?

Edit : clarifying my libketama point.

Say we have 10 servers:

1,2,3,4,5,6,7,8,9,10

One of them dies. Libketama will then provide a very high likelyhood that the hits to the missing server get equally distributed to the remaining servers:

1,2,3,4,inactive,6,7,8,9,10

BUT: if we provide and manage this list manually, this is not the case:

1,2,3,4,6,7,8,9,10 // There are now 9 servers!

6 will get now 5's previous keys, 7 will get 6's. 8 will get 7's, 9 will get 8's and 10 will get 9's. All the hits the 10th servers used to get, will not be evenly distributed among the remaining ones. Resulting a high likelyhood of almost 50% of all keys being sent to new servers.

Evert
  • 93,428
  • 18
  • 118
  • 189
  • Is the goal to send all requests to one cache server and only switch if one is down? Or are certain requests going to certain servers, but still need a fail over? – endyourif Sep 11 '12 at 14:37
  • I'd like every value to distribute evenly, but elegantly deal with the situation if one instance is down. So, replication is not a factor in this scenario. – Evert Sep 11 '12 at 14:39
  • You could write a wrapper than sends the data to your list of memcached servers. As requests are made to for information, you cascade down to find the latest available, grab your data and be done with it. – wesside Sep 11 '12 at 14:49

4 Answers4

3

I generally store my list of available servers in APC, so I can modify it on the fly. You're correct in that systems will attempt to continue using the down server while it's listed, luckily with the new hashing methods it's not a big deal to pull it from rotation.

I would avoid using a brand new PHP extension, or trying to add new software to your deployment stack. You're likely already using something for monitoring (nagios?). Having it invoke a simple PHP script on each of your webservers to tweak the in-memory list seems like the best bet.

It's worth noting that under the Ketama hashing system, removing a server from rotation will result in its keys being re-hashed elsewhere on the ring (continuum), other servers will not see their keys assigned elsewhere. Visualize it as a circle, each server is assigned multiple points on the circle (100-200). Keys are hashed to the circle and continue clockwise until they find a server. Removing a server from the ring only results on those values continuing a bit further to find a new server. With luck the distribution of values will hit the remaining servers equally.

Demonstrating the hashing system:

<?php


$m = new Memcached();
$m->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);


$m->addServer('localhost', 11211);
$m->addServer('localhost', 11212);
$m->addServer('localhost', 11213);
$m->addServer('localhost', 11214);
$m->addServer('localhost', 11215);
$m->addServer('localhost', 11216);
$m->addServer('localhost', 11217);
$m->addServer('localhost', 11218);
$m->addServer('localhost', 11219);
$m->addServer('localhost', 11210);

$key = uniqid(); //You may change this to md5(uniqid()); if you'd like to see a greater variation in keys. I don't think it necessary.
$m->set($key, $key, 5);


var_dump($m->get($key));

unset($m);


$m = new Memcached();
$m->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
//one server removed. If assignment to the continuum is dependent based on add order, we would expect the get call here to fail 90% of the time, as there will only be a success if the value was stored on the first server. If the assignment is based on some hash of the server details we'd expect success 90% of the time. 
$m->addServer('localhost', 11211);
//$m->addServer('localhost', 11212);
$m->addServer('localhost', 11213);
$m->addServer('localhost', 11214);
$m->addServer('localhost', 11215);
$m->addServer('localhost', 11216);
$m->addServer('localhost', 11217);
$m->addServer('localhost', 11218);
$m->addServer('localhost', 11219);
$m->addServer('localhost', 11210);

var_dump($m->get($key));

unset($m);

$m = new Memcached();
$m->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
//2 servers removed
$m->addServer('localhost', 11211);
$m->addServer('localhost', 11212);
//$m->addServer('localhost', 11213);
//$m->addServer('localhost', 11214);
$m->addServer('localhost', 11215);
$m->addServer('localhost', 11216);
$m->addServer('localhost', 11217);
$m->addServer('localhost', 11218);
$m->addServer('localhost', 11219);
$m->addServer('localhost', 11210);

var_dump($m->get($key));

unset($m);

$m = new Memcached();
$m->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
//Out of order
$m->addServer('localhost', 11210);
$m->addServer('localhost', 11211);
$m->addServer('localhost', 11219);
$m->addServer('localhost', 11212);
$m->addServer('localhost', 11217);
$m->addServer('localhost', 11214);
$m->addServer('localhost', 11215);
$m->addServer('localhost', 11216);
$m->addServer('localhost', 11218);
$m->addServer('localhost', 11219);
$m->addServer('localhost', 11213);

var_dump($m->get($key));

unset($m);

If the hashing system cares about order, or omitted servers we would expect to get bool(false) on most of the secondary examples, since an early server was removed etc. However based on my quick, completely non-scientific tests, I only get a bool false in any particular slot one time in 10. I clearly just launched 10 servers on my test box. Giving each of them only 4mb of ram

preinheimer
  • 3,712
  • 20
  • 34
  • This does make sense; with a fallback to a real configuration file. We're very EC2 oriented though, so this would also force us to keep a list of active webservers to notify. – Evert Sep 11 '12 at 14:56
  • 1
    The second issue is.. if there's indeed 10 memcache servers, and one server in the middle gets marked as inactive, the consistent hashing system will fail because it's not just 1 addition/removal from an array, or an item marked as inactive. The array of servers gets completely re-ordered. – Evert Sep 11 '12 at 14:57
  • Evert, order the list numerically or something? Then your order will remain consistent. Good catch though. – preinheimer Sep 11 '12 at 14:59
  • If somehow we could call 'addServers' and give 1 server an 'inactive' flag, the Ketama hashing would still be effective though. – Evert Sep 11 '12 at 14:59
  • How will an ordered list of servers fail during addition or removal of a server? – preinheimer Sep 11 '12 at 15:03
  • Based on: http://docs.libmemcached.org/memcached_behavior.html?highlight=consistent I think the consistent hash values are based on the server details, not simply their place in the list. I'm prepared to be wrong though. – preinheimer Sep 11 '12 at 15:19
  • Good point, I don't know either.. the docs don't seem to be very clear on it. Gonna have to dig into the source :) – Evert Sep 11 '12 at 15:28
  • At this point, based on further readings I'd put like a 95% confidence that it's hashing the server info. The alternative is just silly, and not at all a "hash". It would also require the hashing system to maintain a static variable to remember how many times it had been called already if you add servers sequentially. – preinheimer Sep 11 '12 at 16:06
  • The static variable actually does exist. Dug into the codebase a bit, and 'ketama_create_continuum' is called whenever the list of servers is changed. The 'hash' that's created is (partly) based on the server address, but this only ensures that hashing is correct when servers are out of order, not when 1 server is missing entirely. – Evert Sep 11 '12 at 16:22
  • Let's simplify our server list to 5. If it can handle an out of order list, say 1, 3, 2, 4, 5. How can it not handle 1, 3, 4, 5 (omitting 2)? – preinheimer Sep 11 '12 at 16:24
  • Because the (hash-based) sorting of the server list is a simple 'usort'. The 'circle' is populated with items in this array. If hypothetically your 5 servers also get ordered 1-5 after the sort; Server 3 will still become index 1 in the array. – Evert Sep 11 '12 at 16:27
  • Basically.. if our server list is a simple PHP array, the data used to populate the 'circle' is : `array_keys(array_map('md5', $servers))` This simple example meats the criteria of your last example. – Evert Sep 11 '12 at 16:34
  • If your example is correct, omitting a server shouldn't affect the hashes of others, since each hash is carried out independent of all others. So you will not run into an issue if server 2 of 5 gets pulled. – preinheimer Sep 11 '12 at 17:47
  • Note that I'm also calling array_keys.. The assumption is that it would contain 0-4 in your first example, and 0-3 (inclusive) in your second. Only this index is used to populate the circle. The md5 is discarded. Seriously, take a look at the ketama source if you need proof. This is much better than me drawing up inferior examples. – Evert Sep 11 '12 at 17:53
  • My source reading is inferior (or at least different) to yours, so I tried it. Updated my post. – preinheimer Sep 11 '12 at 18:32
  • Well if your example gets the output you get, it would be pretty solid proof I'm wrong.. – Evert Sep 11 '12 at 18:53
  • I've pasted the source so you can try pretty easily, let me know what you get. – preinheimer Sep 11 '12 at 18:55
  • I will as soon as I get back to the office :) thanks for going this far – Evert Sep 11 '12 at 19:27
  • Finally got around trying this out; and it does indeed seem you were right. So that does make everything a whole lot simpler. Thanks for going as far to make a whole example out of it :) – Evert Sep 13 '12 at 16:39
2

You might wanna try out the Memcached::OPT_AUTO_EJECT_HOSTS option constant for PHP. It's not directly documented but there is a comment here naming it.

(I haven't tried it, so I can't tell you whether it'll work or not)

Louis Huppenbauer
  • 3,719
  • 1
  • 18
  • 24
0

Based on the answer to the comments, I would suggest along this lines:

You'll need to build a caching class.

This class will contain the following information:

  • List of cache servers

    • Status for online or offline
    • Count of requests to this server

  • List of keys currently stored and what server it is on

Next you'll need your standard functions to add, update, and delete keys.

Each time you execute one of these functions, you'll want to check if the key is already in the cache and what server it is on.

If it is not in a server, pick the server with the lowest requests to save it on after retrieving the actual DB value.

If any of these functions return an error from the cache server, I would mark that server as offline, reset the count, and remove any keys from the list that are on this server.

At this point, you could easily automatically move them to a new server or just delete them so they will be queried again.

endyourif
  • 2,186
  • 19
  • 33
  • where would you suggest storing the server, and 'list of keys' information? – Evert Sep 11 '12 at 17:01
  • Perhaps the database server. Otherwise a shared location on disk for all web servers to access. – endyourif Sep 11 '12 at 18:32
  • I have to be fair to you.. this is a terrible idea. The whole point of memcached is that it's distributed, based on a key. You're defeating this completely and either of the solutions (db, disk) have a TON of problem associated with them. Every memcached lookup would be a DB lookup, which is something we want to avoid :) – Evert Sep 11 '12 at 18:47
  • What you're proposing is actually something memcached solves already; so this functionality would be duplicated and made way, way worse. Don't do this in real life! – Evert Sep 11 '12 at 18:48
0

My 2 cents: It is NOT going to be easy to develop a robust HA module for Memcached. For example think about the following cases:

  • How will you determine which server is alive and which one died ? you should somehow sync between all your HA modules running your web/app servers
  • How do you publish this information between your web/app servers
  • Are you going to have an orchestrator ?

I suggest that you have a look at Redis Sentinel which is now in Beta, and has been developed and tested during the last few months specifically for solving these problem in Redis. You wil find a lot of corner cases there that you must be aware of before starting to develop a single line of code.

As for the other issues that were discussed here:

  • When you lose node you lose 1/N of your keys where N is the number of nodes you initially had, i.e. including the failed node. This is how Ketama works
  • Storing keys on the Memcached client using a new Memcached class is definitely not the way to go (IMO): (1_ where are you going to save all these keys? (2) how do you sync between your web/app nodes ? (3) how long it will take you to access these data-structures in-order to understand on which node each key is resided ? -- this is why Memcached is completely based on a hash function, to make it fast and simple.

Last but not least, I would suggest that you also check Memcached as-a-service solutions. For instance, we at Garantia Data have already solved the HA issues of Memcached.

Disclosure: I'm Co-Founder and CTO of Garantia Data.

Yiftach
  • 504
  • 3
  • 5
  • Thanks for the disclosure. There's no way in hell I would opt to store a map of keys on the client ;) – Evert Sep 19 '12 at 12:12