We are having a problem with a virtual machine that's running our frontend website. Once it's running everything is fine, but after a reboot memcached is going bonkers. What happens is that we put items in there set to expire in 15 to 30 seconds, but they don't expire for about an hour! So after a while all data we're serving is highly outdated.
We've been investigating the issue for a bit and found that during startup ntp is changing the clock a lot, putting it almost an hour forward.
We found that memcached doesn't use the system clock but has it's own clock, so once the system clock changes and sets the expiry in it's time, memcache is an hour behind and will keep the item for an hour.
We've already swapped the boot order of ntpd (now S58) and memcached (now S59), but that hasn't resolved the issue.
Restarting memcached manually after a reboot is not really option because our host reboots the server regularly after patches and we're not always there after that's happened.
Does anyone have any idea on how to resolve this? We've googled high and low, but can't find anyone with the same problem. Surely we're not the first to have this problem?
virt-what is reporting the VPS is running in VMWare.
