I'm spinning up a VM in Google Compute Engine running Debian Buster, and configuring it with two network interfaces.
The first uses an ephemeral IP config, and is assigned to a publicly routable network, configured via DHCP. I'd like to keep this as is.
The second interface uses a static IP config, and is assigned to a private network. I'd like to stop GCE from using DHCP to configure this interface, and instead use systemd-networkd to configure it myself, for the purpose of easily adding some custom routes.
While the systemd-networkd configuration does work successfully, the problem is that when the machine is rebooted, GCE's DHCP setup is running after the systemd-networkd setup, and overwriting my custom config.
I've tried numerous things to resolve this issue so far, including:
- Disable the entry for /var/run/interfaces.d in /etc/network/interfaces
- Add a custom systemd service to run after GCE's networking services
The only thing that's worked thus far is a hideous script hack to wait 30 seconds after boot, then kick off systemd-networkd again to overwrite the GCE config.
Is there some cleaner configuration change I can make at the OS level, or a configuration setting during network/server setup that will prevent GCE from auto configuring this second network interface?
EDIT #1:
This is an example of the systemd-networkd configuration I'd prefer to use:
[Match]
Name=ens5
[Network]
Address=10.1.3.30/32
LinkLocalAddressing=no
[Route]
Destination=10.1.3.1/32
Scope=link
[Route]
Gateway=10.1.3.1
Destination=10.1.3.0/24
GatewayOnlink=yes
[Route]
Gateway=10.1.3.1
Destination=10.1.1.0/24
GatewayOnlink=yes
[Route]
Gateway=10.1.3.1
Destination=10.1.2.0/24
GatewayOnlink=yes
[Route]
Gateway=10.1.3.1
Destination=10.10.0.0/24
GatewayOnlink=yes
Using this configuration allows me to set additional routes on the interface to other subnets. Note that these other subnets may or may not be on GCP.
The above configuration works, even given that I have to declare the 10.1.3.30 static IP address as part of my GCE server setup. The problem is only that I can't get GCE to stop doing its own configuration of the interface, which overwrites the one above. On Azure, I can simply comment out the source reference to /var/run/network/interfaces.d in /etc/network/interfaces.
As for why I'd prefer this setup, it's an elegant method for setting up my internal network via my server configuration management software -- I just drop the above config into /etc/systemd/network/ and issue systemctl enable systemd-networkd && systemctl restart systemd-networkd, and it handles configuring the interface and setting the configuration to also happen on boot.
UPDATE #1:
I've filed https://issuetracker.google.com/issues/153513472 in GCP's issue tracker, hopefully they'll address. I'll update the issue accordingly when they do.
UPDATE #2:
After a round with paid GCP support, they've pointed me to another ongoing issue: https://issuetracker.google.com/issues/167371074
The support agent also suggested other users have simply put cron scripts in place to restart the network periodically.
I've adapted this suggestion for my specific network needs, with the following crontab entry:
* * * * * /usr/bin/test -z "$(/sbin/ip route | /bin/grep "10.1.1.0/24")" && /bin/systemctl restart systemd-networkd
Where '10.1.1.0/24' is a subnet that I know in my routing config should not be missing.
UPDATE #3:
Finally, a clean and satisfying answer! Google engineers informed me that the google-guest-agent.service which runs as part of their guest environment on Debian systems has a network daemon, which, when it re-configures any interfaces, calls dhclient directly. It would seem that dhclient called this way wipes out existing routes. Given that, an easy workaround is to create a dhclient exit-hook by adding a shell script in /etc/dhcp/dhclient-exit-hooks.d/.
In my case, I added the following script at /etc/dhcp/dhclient-exit-hooks.d/systemd-networkd:
case $reason in
BOUND|RENEW|REBIND|REBOOT)
systemctl restart systemd-networkd.service
;;
esac
This seems to solve the problem, including across reboots.
