1

I am running a headless install of Ubuntu 10.0.4 LTS on a PVS. I was carrying out an OS upgrade earlier on today. I SSH'd into the sever and typed the following commands:

sudo aptitude update
sudo aptitude safe-upgrade

Whilst the second command (sudo aptitude safe-upgrade) was running, I (foolishly) decided to shut the server down (to bring down Apache and any other daemons that may be running on the server), whilst the OS and other software was being upgraded.

I lost my SSH connection to the server (unsuprisingly), and the install was aborted by the server being shut down. After rebooting the server, I have been unable to log into the server again.

Here is the console output when I attempt to login remotely using the Lish Ajax Console:

XENBUS: Device with no driver: device/console/0
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
REISERFS warning (device xvda): super-6502 reiserfs_getopt: unknown mount option "nobarrier"
EXT3-fs: barriers not enabled
kjournald starting. Commit interval 5 seconds
EXT3-fs (xvda): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
devtmpfs: mounted
Freeing unused kernel memory: 668k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 84k freed
Freeing unused kernel memory: 1356k freed
init: udevtrigger main process (1203) terminated with status 1
init: udevtrigger post-stop process (1205) terminated with status 1
init: udevmonitor main process (1202) killed by TERM signal

I thought there maybe something messed up with the filesystem, so I run fsck. Here is the output:

root@hvc0:~# fsck -fy /dev/xvdb
fsck from util-linux 2.19.1
e2fsck 1.42-WIP (02-Jul-2011)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/xvdb: 111349/1286144 files (6.5% non-contiguous), 1798314/5120000 blocks
root@hvc0:~#

AFAIK, this means that there is nothing wrong with the filesystem - so I have no idea what else to do. My server is currently unreachable, and I can't SSH in to try to rerun the install in case that was the cause of all of this.

Does anyone have any idea on what the issue could be and how I may resolve this and get the server back online?

Homunculus Reticulli
  • 133
  • 2
  • 10

2 Answers2

1

If you can mount the drive, chroot into it with

chroot /mount/point

and then try re-running the upgrade.

fizban
  • 111
  • 2
  • although I am a software developer, I have ZERO knowledge about sysadmin. Could you please explain WHICH drive to mount (how do I know which drive to mount?). I have a /dev/xdva, /dev/xdvb and /dev/xdvh - Iam not sure which one it is I need to mount. Could you please explain how I may find out which one to mount? – Homunculus Reticulli Jan 25 '12 at 17:32
  • I tried as you suggested, and I got the following error: chroot: cannot change root directory to /dev/xvdb: Not a directory – Homunculus Reticulli Jan 25 '12 at 17:40
  • You would mount whichever one is your root partition (*likely* the first one, xdva, but depends on how you set it up), and then chroot into the mount point, not the /dev/whatever device itself. – fizban Jan 25 '12 at 18:13
0

The answer was given to me from my posting in the Linode forum:

The upgrade was interupted while it was making changes to your filesystem, so your server was left in an undefined state and now it doesn't boot -- hence no access.

udev, the device manager that creates the device nodes in /dev, is hosed. Try this:

Boot into rescue mode, then run:

mkdir -p /mnt/rescue
mount /dev/xvda /mnt/rescue
chroot /mnt/rescue /bin/bash
dpkg --configure -a

Reboot the server normally.

Homunculus Reticulli
  • 133
  • 2
  • 10
  • I would suggest you mount /proc and /sys in the chroot environment when doing that, some upgrade scripts may get confused otherwise. – rackandboneman May 17 '12 at 15:24