2

So I'm very much a novice when it comes to Apache and Ubuntu, but I know the basics (file moving, copying, editing, etc).

I was trying to remedy a problem with PHP's permissions, so I set /tmp/ and /var/tmp/ permissions to 777 using chmod. Then, I modified both php.ini files, in /etc/php/ and /etc/php5/, to customize the temp directory. However, I ended up rebooting the server without actually creating the temp directory (oops).

Now I can still connect to my server through SFTP, but in Chrome when I try to connect it gives me an ERR_CONNECTION_REFUSED. I thought it was a problem with .php pages, so I tried loading an .html page. Nothing. I've tried rebooting several more times, but to no avail.

In addition, my whole filesystem is read-only. So I'm assuming I messed up something in config and as a safety measure, everything is locked down. I have tried sudo su, remounting the drive (blocked because I can't write to some config file), and a bunch of other stuff. I also get an error when doing fsck: fsck.ext4: unable to resolve UUID: xxxxxxxx. Again, can't auto fix any errors through remounting because I can't write to some config file.

I tried going through system logs in /var/log/, but I couldn't find any error messages. I believe this is a problem with PHP config, but I'm not sure. I also looked at my UFW, and saw it was disabled. When trying to enable it, again, it wouldn't let me because I couldn't write to the file system. All of this is in sudo.

Help would be greatly appreciated. Thanks!

EDIT: I managed to find my bash history as root, but nothing seems too weird: https://pastebin.com/yPh5rSE6

EDIT 2: One thing I remember is while modifying permissions for /var/tmp/, the folder had a sticky bit; I believe the permission was drwxrwxrwt. Was changing this a server breaking blunder? I cannot change this because I'm mounted as a read-only file system.

EDIT 3: So it seems like the issue is something with permissions, which caused an error on start up and locked the file system to read-only, messing up the web server. How plausible does this sound? The issue seems to be rooted with the UUID error showing up in two places, fsck and mount. How do I solve this?

shrey150
  • 23
  • 6
  • If the whole file system is read-only, the problem is something way wider than PHP's configuration. Also, giving wide `777` permissions even temporarily is reckless. Probably you have messed something up while doing this kind of random testing with file permissions. – Esa Jokinen Jul 16 '17 at 05:00
  • Bash history for root before this reboot could give a clue on what went wrong. It's also likely that restoring from backups is the easiest solution. Be more careful next time. – Esa Jokinen Jul 16 '17 at 05:03
  • Where can I find bash history for root? – shrey150 Jul 16 '17 at 05:29
  • @EsaJokinen I figured it out, added it to my question as a pastebin link: nothing seems too out of the ordinary. – shrey150 Jul 16 '17 at 06:04
  • Both `/tmp` and `/var/tmp` had sticky bit, that shouldn't under any circuimstances make root file system mounted read-only. You need to figure out why that happens first before fixing the web server. Your description of why remounting fails is very confusing - you mean sudo fails or the remount fails? Show us the errors! – Marek Rost Jul 16 '17 at 10:13
  • @MarekRost I tried `sudo mount -o remount,rw /` and it gives me `mount: can't find UUID=xxx`. This is the same UUID from `fsck`. Perhaps this is the main problem? Sorry about the confusing description, I got varying errors at different times. – shrey150 Jul 16 '17 at 10:56

1 Answers1

0

It is unusual for system to spontaneously start mounting root partition as read-only. The UUID error suggests something wrong with /etc/fstab file and hard drive detection.

First attempt to remount the root partition rw with explicitly setting the partition name. Use the command df to show which partition is mounted as /. Then use the sudo mount -o remount,rw with both location of the partition and /.

After this was hopefully successful, it's time to debug the read-only boot. Verify the state of /etc/fstab file. Is the syntax correct? Do all drives and partitions specified here really exist? Are there no explicit ro mount options for the / mount point? If the file contains any UUIDs, check that they exist in /dev/disk/by-uuid/ and check that these files are links to existing drive partitions.

After you've corrected any possible errors with /etc/fstab reboot the machine and fix the permissions on tmp folders if they haven't been fixed automatically.

If remounting the root partition in rw mode still fails with explicit drive location, check for possible damage to the hard drive. Look into output of dmesg command and consult internet for any messages suggesting I/O problems. If you have smart installed use the command sudo smartctl -a /dev/<drive> to verify it has not been damaged.

Get back to us after you've tried all this.

Marek Rost
  • 263
  • 3
  • 11
  • Okay, thanks for the help, I successfully remounted the drive (`sudo mount -o remount rw /`), but I don't know what to do with `fstab`. This is the one line written in the file: `UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 / ext4 errors=remount-ro 0 1`. What does this mean, and how do I fix it? I can confirm this is the same UUID from other errors. – shrey150 Jul 17 '17 at 04:22
  • So I looked in `/dev/disk/by-uuid` and there is only one UUID; however, it is different from the one before. – shrey150 Jul 17 '17 at 04:28
  • Nvm, I copied over the UUID from `by-uuid` into `fstab` and rebooted, works like a charm! Thanks so much! – shrey150 Jul 17 '17 at 04:39