4

CentOS 5.5

I just came across an instance where my /storage directory was supposedly a read-only file system. I get the error:

mkdir: cannot create directory `test': Read-only file system

df:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      141G   21G  114G  16% /
/dev/sda1              99M   13M   82M  14% /boot
tmpfs                1001M     0 1001M   0% /dev/shm
/dev/sdb1             1.4T  307G  999G  24% /storage
/dev/sdd1             1.4T  329G  978G  26% /media/hddblack
/dev/sdc1             1.4T  329G  978G  26% /media/hddsilver

mount:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /storage type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdd1 on /media/hddblack type ext3 (ro)
/dev/sdc1 on /media/hddsilver type ext3 (ro)

/storage is clearly a read-write file system. Next I tried this:

mount -o remount,rw /dev/disk/by-label/storage /storage

And now, the filesystem is writable again. Hooray. Except, why did it do this? Here is the mount now:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /storage type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdd1 on /media/hddblack type ext3 (ro)
/dev/sdc1 on /media/hddsilver type ext3 (ro)

Pretty much the same thing... any ideas what would cause this or what logs I should look into to find what the problem was?

Jake Wilson
  • 8,814
  • 29
  • 97
  • 125

2 Answers2

7

Most likely you had a disk IO error. Check /var/log/messages. Look for messages about aborting the journal and re-mounting read-only.

P.S.: If that is the case you best have backups of your data.

Mark Wagner
  • 18,019
  • 2
  • 32
  • 47
  • In which case, you will want to look into `fsck`. – Andrew M. Nov 24 '10 at 04:21
  • Bewarned: if you don't have backups try to salvage as much data as possible before the fsck. Fsck on a drive with I/O errors has a good chance of totally hosing the filesystem. – Mark Wagner Nov 24 '10 at 18:28
5

The problem with your diagnostic steps: if your root filesystem is read-only, /etc/mtab is also read-only. Guess what mount reads to get filesystem status :)

The solution is to check /proc/mounts, which is a dynamic kernel file that shows the true filesystem state. If you'd have done so at the time you had this problem, you'd very likely have found your root filesystem was remounted read-only. Know your utilities and where they get their data from.

How's that happen? Most /etc/fstab files will have a line similar to:

/dev/sda1 /               ext3    relatime,errors=remount-ro 0

Note the "errors=remount-ro" option. In the event of any filesystem errors, the root filesystem is remounted in read-only mode.

Assuming /var/logs was writeable, or you had offline syslogging enabled, you should find the event logged as well.

Disk errors capable of triggering a remount-ro are generally a Bad Thing. Hope you swapped your drive.

Dr. Edward Morbius
  • 879
  • 7
  • 13