0

My coworker was doing what should've been a straight-forward install of memcached this weekend on our Ubuntu 10.04.04 LTS server, but something went awry. I'm still trying to figure out the details, but I can tell you:

$ /lib/libc.so.6
GNU C Library (Ubuntu EGLIBC 2.11.1-0ubuntu7.12) stable release version 2.11.1, by Roland McGrath et al.
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.4.3.
Compiled on a Linux >>3.2.0-23-generic<< system on 2012-11-15.
Available extensions:
       crypt add-on version 2.1 by Michael Glad and others
       GNU Libidn by Simon Josefsson
       Native POSIX Threads Library by Ulrich Drepper et al
       BIND-8.2.3-T5B

However, I also see this:

$ ldd --version
ldd (Ubuntu EGLIBC 2.15-0ubuntu10.5) 2.15
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

With comparison to another Ubuntu 10.04.04 LTS machine, I know that 2.11.1 is where it ought to be, unless we try to use the upgrade tools to go to Ubuntu 12.04 LTS. But what is the best path to get there? Do I copy over libraries from the working machine until it all works, or is there a better way?

Michael H.
  • 543
  • 1
  • 5
  • 15

1 Answers1

0

OK, I think we know what happened. A couple of months ago, when reviewing the daily logwatch report, I saw that someone had started to run a dist-upgrade. I asked them about it, and they said they thought they were on their local linux machine, not my server, and ^C'd as soon as they realized something was wrong.

All appeared to be well, so I didn't think more of it... until a different coworker was installing memcached, and for some perhaps-related reason was having problems with the install, so he used one of the various force-this-upgrade options, and that's when all hell broke loose.

Many things were broken: apt/aptitude, anything complicated, even man. But dpkg worked! And if you look, there are archives of every standard library at every point release / update to the system. With careful iterations of downloading a package, install / force reversion to that version, recheck dependencies, download again, we were able to knit the system back together to exactly the same version of all relevant libraries we had at the time. I think we had to use a "hold" option once to keep a circular dependency from causing trouble. It took several hours, but we got the system back to the original state. Then we did a restart (just to make sure -- it had been 253 days since last reboot), and then a standard upgrade to get to the latest release of Ubuntu 10.04 LTS, and reboot again. Voila - saved.

What a nightmare!

Lessons learned:

  1. treat it more seriously when someone accidentally starts upgrading your system;
  2. be sure to make all machines clearly identifiable when you log in;
  3. be sure to stay on top of the smaller releases, since that we wouldn't have gone to the archives;
  4. thank the crazy people who keep those archives of every library version for every system release!
Michael H.
  • 543
  • 1
  • 5
  • 15