0

currently I have three different running memcached instances, each one for an environment (dev, test and live)

All three environments are running the same memcached version (1.4.5) and configurations with slightly different Linux distros and machines, as follows:

  • Dev: Linux version 2.6.17-13mdv (rtp@octopus.mandriva.com) (gcc version 4.1.2 20070302 (prerelease) (4.1.2-1mdv2007.1)) #1 SMP Fri Mar 23 15:18:36 EDT 2007 Mandriva Linux release 2007.1 (Official) for x86_64 Kernel 2.6.17-13mdv on a Dual-processor x86_64 / \l

  • Test: Linux version 2.6.33.2-desktop-2mnb (herton@celeste.mandriva.com) (gcc version 4.4.3 (GCC) ) #1 SMP Thu Apr 15 21:34:06 UTC 2010 Mandriva Linux release 2010.0 (Official) for x86_64 Kernel 2.6.33.2-desktop-2mnb on a 4-processor x86_64 / \l

  • Live: Linux version 2.6.31.13-server-1mnb (qateam@titan.mandriva.com) (gcc version 4.4.1 (GCC) ) #1 SMP Tue Apr 27 21:14:30 EDT 2010 Mandriva Linux release 2010.0 (Official) for x86_64 Kernel 2.6.31.13-server-1mnb on a 16-processor x86_64 / \l

My problem is that in the live environment all caching is perfect, I set some key/value and it responds to as expected, but in the test and dev env the application sets some key/value but fails to retrieve it.

Logging the output of memcached using -vvv gives me some insight about what's happenning when it tries to get the keys as follows:

<40 get qZPh2MQDIs%2BXTBrL3C7hiAcClZgOiklYwnz0a0LN6tY%3D FOUND KEY qZPh2MQDIs%2BXTBrL3C7hiAcClZgOiklYwnz0a0LN6tY%3D. 0 2700 ?? -nuked by expire

This nuked by expire is driving me a little crazy, fortunately my live env is OK.

I double checked the NTP configuration for the servers (I thought that maybe if the date was wrong the expiring was in the past or something) and it's all OK, my cache is expected to last 10 minutes from now and all servers are time-synced.

So, anyone could help me discover what's going wrong?

victorcampos
  • 185
  • 1
  • 6

1 Answers1

1

Ok, found it, the problem was that even with NTP enabled and "date" showing the correct local datetime there was a misconfiguration in the UTC datetime.

Solution: date +%T -u -s "actual-UTC-time"

I don't know why even with NTP it was like that, but just giving some feedback if anyone bumps into the same problem :D

victorcampos
  • 185
  • 1
  • 6
  • Are you sure you have your time-zone set correctly? Does "date +%Z%z" respond as you'd expect? I wonder this because the system stores all time as seconds since 1970 UTC, and NTP synchronizes time in UTC (which is why it doesn't need to sync with only clocks in your local timezone). So I wonder if the clocks were actually out of sync but that was masked by different machines having different timezones set making them look like the same time. I ask because I've never run into NTP synchronized systems that are correctly syncing having the sort of confusion you are reporting. – Sean Reifschneider Nov 25 '10 at 07:43
  • Sean, I don't know what happened (I'm a developer, not a sysadmin actually), I checked the NTP service and it was up and running in both machines. When I checked the date -u they were desynchronized... I warned my infrastructure team about it and they may have already fixed it (or not :/), but thanks for the info :D – victorcampos Dec 03 '10 at 21:12