This is more a brain dump than anything at this point, but I'm stuck and help would be appreciated!
I have a really complicated set of limitations on 3 of my servers that necessitated running the OS from a ram drive. Long story short, I have a group of blade servers that have a total of 4 hard drives each. I need to run a server application, called MinIO, in distributed mode. For distributed mode, MinIO requires 3 nodes with 4 pristine HDDs each. This means, there's not enough HDDs in the blades. There are work-around options for this situation, but for reasons we won't go into here, we can't do that. I've been tasked with booting these servers from a ram drive... I eventually got it working using FOG (basically just PXE booting and loading a squashfs file with the root file system over the network).
At the moment, the hardest part is done. the servers boot and I'm going through the squashfs file they use to configure MinIO. I've run into a snag with some persistent settings though. So basically, I need a way for the three servers to set their own static IPs based on the mac address of their network controllers.
Now I know I could just set DHCP reservations, and for 3 of the nics in question I have! but 3 nics live on a public network without a DHCP server... these nics have to dynamically set a static IP... yeah... I just cant seem to work out a good way to do this.
I'm going to keep researching and try to pick this one apart myself, but I'll post my progress here as I go. if anyone has an idea, I'm very open to suggestions!
What follows is my brain dump as I go through what I've done so far. please note, all of this except the DHCP portion was done on the PXE server itself. in theory, if I do this right, nothing needs be done on the destination servers except boot.
PART 1: through the FOG
Installed Debian standard took most of the default options, selected ssh server and core components only in tasksel
Some post install corrections and package installations:
$ su
# nano /etc/apt/sources.list # <--remove the line for the DVD
# apt-get update && apt-get install sudo neofetch apache2 bc build-essential cpp curl g++ gawk gcc genisoimage git gzip htmldoc isolinux lftp libapache2-mod-php7.3 libc6 libcurl4 liblzma-dev m4 mariadb-client mariadb-server net-tools nfs-kernel-server openssh-server php7.3 php7.3-bcmath php7.3-cli php7.3-curl php7.3-fpm php7.3-gd php7.3-json php7.3-ldap php7.3-mbstring php7.3-mysql php7.3-mysqlnd php-gettext sysv-rc-conf tar tftpd-hpa tftp-hpa unzip vsftpd wget xinetd zlib1g
# /usr/sbin/usermod -a -G sudo [username]
# exit
$ exec newgrp sudo
$ export PATH=$PATH:/usr/sbin
$ sudo export PATH=$PATH:/usr/sbin
set network config and reboot.
Install FOG:
$ wget https://github.com/FOGProject/fogproject/archive/1.5.8.tar.gz
$ mv 1.5.8.tar.gz fogproject-1.5.8.tar.gz
$ sudo cp fogproject-1.5.8.tar.gz /tmp/
$ cd /tmp/
$ sudo tar -zxvf fogproject-1.5.8.tar.gz
$ cd fogproject-1.5.8/bin/
$ sudo ./installfog.sh
Took the defaults during FOG installation...
GOTCHA#1: PAY ATTENTION TO THE INSTALLER!!! If fog has any errors or is canceled after it starts configuring MySQL, you might as well start over. Fixing MySQL login issues (in my environment at least) takes longer than reinstalling the OS. At some point toward the end of the install, FOG instructed me to log into a management website to finish config... it's easy; just open up a browser, go to the url fog provides, click "install", and you're done.
Disclaimer: I did not use ssl for the web page, nor did I configure the firewall. The PXE server is not on a public network. Also, the network we are on was given internet access for this installation only. Once everything was up and running, the PXE server was literally cut off from the internet (unplugged the cable entirely). Anyone using this as a guide for live PXE booting should consider setting up ssl, especially if there is a connection to the internet! At the least you should set up your firewall and create a self signed cert!
PART 2: Execute order 66 ...and 67
DHCP options 66 and 67 specify the location of the TFTP server and the bootable file to be served. I have a test environment and production environment. One uses an edgerouter while the other is just using server2012 r2 serving DHCP and DNS. I have been building this whole setup in both so we can deploy quickly once I get it all up and running in testing.
On the edgerouter I ran these commands:
set service dhcp-server shared-network-name VL1_dhcp subnet 10.10.10.0/24 tftp-server-name 10.10.10.100
set service dhcp-server shared-network-name VL1_dhcp subnet 10.10.10.0/24 bootfile-name ipxe.kpx
set service dhcp-server shared-network-name VL1_dhcp subnet 10.10.10.0/24 bootfile-server 10.10.10.100
On the windows server I did this:
- Open DHCP console
- Drill down to the scope in question
- Right Click in Scope Options -> Configure Options
- Scroll Down and check number 66.
- Enter the PXE server's IP address for the string value. Other instructions online say to use the PXE servers hostname. We've found it to be more reliable when using the servers IP address instead.
- Scroll down and check number 67.
- Enter the string value "ipxe.kpxe" (no quotes)
- Apply and close.
Fog server was functional at this point. Any network boot capable computers in the private network could boot to the FOG menu. We were even able to get VMs in HyperV to boot to it, provided we setup a legacy adapter in the test VM settings first.
It's kind of surreal seeing "downloading filesystem.squashfs" with actual download progress in dmesg, before seeing "waiting for network". More so when looking at lsblk! after this, I'm SO going to build a recovery image (knoppix comes to mind) for general use. No more thumb drives or burning DVDs, just boot to network and select the recovery option!
PART 3: Slaving over a hot keyboard
This part is pretty straight forward. I used this guide by george1421 to make it all happen... would NOT have gotten this far if not for his help!
download Debian live standard:
$ sudo mkdir /ISO
$ cd /ISO
$ sudo wget https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-10.5.0-amd64-standard.iso
Prep the boot area
$ sudo mkdir -p /images/os/debian/10.5L
$ sudo mkdir -p /tftpboot/debian/10.5L
$ sudo mkdir -p /var/www/html/os/debian/10.5L
$ sudo mount -o loop -t iso9660 /ISO/debian-live-10.5.0-amd64-standard.iso /mnt/loop
$ sudo cp -R /mnt/loop/* /images/os/debian/10.5L
$ sudo umount /mnt/loop
Copy the only files we actually care about...
$ sudo cp /images/os/debian/10.5L/live/vmlinuz-4.19.0-10-amd64 /tftpboot/debian/10.5L/vmlinuz
$ sudo cp /images/os/debian/10.5L/live/initrd.img-4.19.0-10-amd64 /tftpboot/debian/10.5L/initrd
$ sudo cp /images/os/debian/10.5L/live/filesystem.squashfs /var/www/html/os/debian/10.5L
GO TO fog webmanagement (in my case http://10.10.10.100/fog/management)
FOG Configuration->iPXE New Menu Entry
Menu Item: os.Debian10.5SL
Description: Debian 10.5 Standard Live
Parameters:
kernel tftp://${fog-ip}/debian/10.5L/vmlinuz-4.19.0-10-amd64
initrd tftp://${fog-ip}/debian/10.5L/initrd.img-4.19.0-10-amd64
imgargs vmlinuz boot=live vga=773 components fetch=http://${fog-ip}/os/debian/10.5L/filesystem.squashfs
boot || goto MENU
Menu Show with: All Hosts
save the entry and we're off to the ball games.
PART 4: Nice server you got there... be a shame if it rebooted
To build a custom image, I installed squashfs-tools on the PXE server and uncompressed the live squashfs file and chrooted in to make my changes.
$ cd /var/www/html/os/debian/10.5L
$ sudo mkdir squashfs-temp
$ cd squashfs-temp
$ sudo unsquashfs /var/www/html/os/debian/10.5L/filesystem.squashfs
$ sudo chroot /var/www/html/os/debian/10.5L/squashfs-temp/squashfs-root/
I replicated the changes noted above and installed some things in the chroot:
# nano /etc/apt/sources.list # <--had to create a fresh sources.list
deb http://ftp.us.debian.org/debian/ buster main
deb-src http://ftp.us.debian.org/debian/ buster main
For DNS to work inside the chroot, you need to edit resolv.conf. This change will be overwritten by other applications, but it doesn't matter. this is just for chroot.
# nano /etc/resolv.conf #
nameserver 8.8.8.8
moving on...
# apt-get update && apt-get install sudo neofetch ufw wget curl openssh-server # <- this is a good place to install any additional packages you might need on the client.
# adduser [username]
# /usr/sbin/usermod -a -G sudo [username]
# passwd
# su [username]
$ exec newgrp sudo
$ export PATH=$PATH:/usr/sbin
$ exit
# export PATH=$PATH:/usr/sbin
Note: Setting a root password breaks the image. As soon as the system boots up, you get a blinking cursor of death. We need to disable automatic login.
# echo "live-config.noautologin" >> /etc/live/config.conf
Next I set the firewall rules in the chroot (while the PXE server doesn't have internet, the PXE clients do)
# ufw allow in on enp0s10f0 to any port 22 # <--allow ssh over the private nic only
# ufw allow 80 <--allow port 80 over any network
# ufw allow 443 <--allow port 443 over any network
GOTCHA #2 Do NOT enable the firewall in the chroot unless you have physical access to the server (or have IPMI) the firewall will actually block traffic in the HOST OS! Not sure how to get around this one yet... we need the firewall enabled on the client on boot. My best idea is to enable it in chroot and immediately exit chroot and reboot the PXE server before the host firewall reacts. I'm sure there's a more elegant solution, but it eludes me at the moment.
To make minor changes to the client without having to update the squashfs file again, toss a script in to perform changes.
# nano /etc/network/if-up.d/00-onboot
the script in question:
#!/bin/sh
FLAGFILE=/var/lock/launch-script-complete
case "$IFACE" in
lo)
# The loopback interface does not count.
# only run when the any other interface comes up
# in theory, the first interface to actually come up will be the private interface...
exit 0
;;
*)
;;
esac
if [ -e $FLAGFILE ]; then
exit 0
else
touch $FLAGFILE
fi
wget http://10.10.10.100/os/debian/10.5L/launch.sh -O /var/run/launch.sh
sh /var/run/launch.sh
Make it executable
# chmod 755 /etc/network/if-up.d/00-onboot
You'll notice that script launches from ifup and does nothing more than download another script from the PXE server? yeah. that's where the magic is going to have to happen!
Finally, some quality of life changes (We have a very specific use caase where these make sense. YMMV, proceed with caution)
# rm /lib/live/config/1160-openssh-server
PART 5: Oh yeah... it's all coming together now.
Exit chroot, compress the temp folder back into a squashfs file, and replace the original:
# history -c # <--you dont need THIS history to persist over reboots.
# exit
$ cd /var/www/html/os/debian/10.5L/squashfs-temp
$ sudo mksquashfs squashfs-root/ filesystem.squashfs -noappend -always-use-fragments
$ cd ..
$ sudo mv filesystem.squashfs filesystem.squashfs.old # <-- OR: $ sudo rm filesystem.squashfs
$ sudo mv /var/www/html/os/debian/10.5L/squashfs-temp/filesystem.squashfs /var/www/html/os/debian/10.5L/filesystem.squashfs
Thats it! the live image is ready to boot... but then there's that launch script...
PART 6: Putting toothpaste back in the tube
Edit the launch script:
$ sudo nano /var/www/html/os/debian/10.5L/launch.sh
And this is where I'm stuck (maybe)... The trouble is, I have 3 servers with 3 different public IPs, and 12 drives with diferent uuids! how do I tell the script which to use?
I suppose the first thing the script needs to to is look at the network controllers on the client server. The hardware addresses found with cat /sys/class/net/*/address are a literal fingerprint of the server so it's a good idea to look there. I could create a list of hardware addresses to expect and tell the script to check what exists and act accordingly. But to be perfectly honest, I'm not sure how implement this. My scripting knowledge is very weak. hell, the script I put in above was literally copied from this post, but I think it can be adapted for this... right?
At this moment, I'm researching the script, and it appears I have a solution!
DISCLAIMER: the script that follows has not yet been tested... but it looks fantastically simple and straight forward.
#!/bin/sh
# Commands placed here will be run on ALL clients regardless of actual purpose or identity.
echo "# file deleted on reboot" > /etc/network/interfaces
echo "# The loopback network interface" >> /etc/network/interfaces
echo "auto lo" >> /etc/network/interfaces
echo "iface lo inet loopback" >> /etc/network/interfaces
echo "# The private network interface" >> /etc/network/interfaces
echo "auto eth0" >> /etc/network/interfaces
echo "iface eth0 inet dhcp" >> /etc/network/interfaces
echo "" >> /etc/network/interfaces
# from here on, the script will check the mac ID of the first network card in the server for hardware identification, and run the appropriate section below.
MAC=`ifconfig eth1 | awk '$1 == "ether" {print $2}'`
case "$MAC" in
#mac address of private network card of the first server
"##:##:##:##:##:##" )
echo "# The public network interface" >> /etc/network/interfaces
echo "auto eth1" >> /etc/network/interfaces
echo "iface eth1 inet static" >> /etc/network/interfaces
echo "address 20.0.0.21" >> /etc/network/interfaces
echo "netmask 255.255.255.240" >> /etc/network/interfaces
echo "gateway 20.0.0.1" >> /etc/network/interfaces
echo "" >> /etc/network/interfaces
# additional commands to perform here
;;
#mac address of private network card of the second server
"##:##:##:##:##:##" )
echo "# The public network interface" >> /etc/network/interfaces
echo "auto eth1" >> /etc/network/interfaces
echo "iface eth1 inet static" >> /etc/network/interfaces
echo "address 20.0.0.22" >> /etc/network/interfaces
echo "netmask 255.255.255.240" >> /etc/network/interfaces
echo "gateway 20.0.0.1" >> /etc/network/interfaces
echo "" >> /etc/network/interfaces
# additional commands to perform here
;;
#mac address of private network card of the third server
"##:##:##:##:##:##" )
echo "# The public network interface" >> /etc/network/interfaces
echo "auto eth1" >> /etc/network/interfaces
echo "iface eth1 inet static" >> /etc/network/interfaces
echo "address 20.0.0.23" >> /etc/network/interfaces
echo "netmask 255.255.255.240" >> /etc/network/interfaces
echo "gateway 20.0.0.1" >> /etc/network/interfaces
echo "" >> /etc/network/interfaces
# additional commands to perform here
;;
esac
Thank you @Brandon Xavier for the awsome suggestion. I was putting a significant dent in my forehead over this one. honestly, it was the ifconfig part, I was trying to get that same info using ip a... if it works, I'll mark you reply as the solution.
