PXE boot Linux - which directories must be writable?

Question

I'm planning to set up a small Hadoop cluster where the slave nodes boot and run from a central PXE server, to simplify deployment and updates, and to enable all of the disks on the slaves to be (almost) monopolized by HDFS.

However, I suppose I'll still have to reserve some space on the slave nodes for /tmp and /var/log, I don't want to just put these in a ramdisk because I'd like them there for debugging after crashes (and because RAM is scarcer than disk).

So the machines might boot off the remote PXE server, mount their / read-only from there, then mount /dev/sda1 through sdd1 for the HDFS data partitions, /dev/sda2 for /tmp, and /dev/sdb2 for /var/log.

My question is, are there any other directories that will need to be writable? Assuming we get Hadoop etc. to log into /var/log.

(And is this a sensible architecture in general?)

EDIT: don't worry about swap, I'm planning to make these swapless, the OOM killer is preferable to thrashing.

score 1 · Accepted Answer · answered May 10 '12 at 13:13

1

You can study livecd layout of your distro, but likely you need /var instead of /var/log and in some distros there are files in /etc that must be writable. /home as well unless you put home dir elsewhere.

answered May 10 '12 at 13:13

johnshen64

5,865
24
17

Thanks, I didn't think of reverse-engineering a live CD. – Andrew Clegg May 11 '12 at 08:58

score 1 · Answer 2 · answered May 10 '12 at 15:22

1

"(And is this a sensible architecture in general?)...."

I wouldn't say that your idea is wrong but seems interesting. In a nutshell your setting up a diskless architecture but still using the localdisk. To me your adding extra loops.

"to simplify deployment and updates .. "

If your goal is to make it more centralized and manageable. I would use some sort of automated distribution engine. Like in my case, I use puppet. The code is already available on github. Just customize to your needs. That should take care of your simplification & manageability. I built couple of clusters in no time using my puppet manifests.

answered May 10 '12 at 15:22

Chakri

1,070
6
8

I have accepted the other one because it was a more direct answer, but have upvoted yours as well because it has got me thinking. Perhaps Puppet is a better idea after all. My motivation was to reduce the usage of the local disks as well, though, to maximize the amount of storage and I/O that HDFS can use. So I'm still not sure. – Andrew Clegg May 11 '12 at 09:00
Andrew, thank you for the vote. I am little curious on the performance impact for not using local disks. – Chakri May 11 '12 at 13:48

score 0 · Answer 3 · answered Jun 27 '18 at 21:42

Here is a simple solution for Slackware 14.2 (BSD init, not systemd) booting via PXE with the root filesystem served read-only over NFS.

I simply modified /etc/rc.d/rc.S to copy the folders that need to be writable (/etc and /var) into a tmpfs filesystem, mount temporary filesystems over the original folders, and move the copies into the temporary filesystems:

# Copy /etc and /var as tmpfs prior to starting any processes.
echo "Creating volatile copies of /etc and /var in memory..."
mount -t tmpfs -o size=256m tmpfs /tmp
cp -a /etc /tmp/etc
mount -t tmpfs -o size=16m tmpfs /etc
mv /tmp/etc/* /etc/
rmdir /tmp/etc
cp -a /var /tmp/var
mount -t tmpfs -o size=16m tmpfs /var
mv /tmp/var/* /var/
rmdir /tmp/var

The above was inserted at the top of rc.S, preceded only by the line that sets the PATH variable. This runs at the very start of init, prior to any other partitions being mounted or services starting.

Any changes to these in-memory copies are simply discarded on reboot.

PXE boot Linux - which directories must be writable?

3 Answers3