How to permanently change sysctl settings on a GKE host node?

Question

We have a kubernetes cluster running in Google GKE. I want to permanently set another value for fs.aio-max-nr in sysctl, but it keeps changing back to default after running sudo reboot.

This is what I've tried:

• sysctl -w fs.aio-max-nr=1048576
• echo 'fs.aio-max-nr = 1048576' | sudo tee --append /etc/sysctl.d/99-gke-defaults.conf
• echo 'fs.aio-max-nr = 1048576' | sudo tee --append /etc/sysctl.d/00-sysctl.conf

Is it possible to change this permanently? And why isn't there a etc/sysctl.config but two sysctl files in sysctl.d/ folder?

Answer 1

I'd do this by deploying a DaemonSet on all the nodes on which you need this setting. The only drawback here is that the DaemonSet pod will need to run with elevated privileges. The container has access to /proc on the host, so then you just need to execute your sysctl commands in a script and then exit.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: sysctl
spec:
  template:
    spec:
      containers:
        - name: sysctl
          image: alpine
          command:
            - /bin/sh
            - -c
            - sysctl fs.aio-max-nr=1048576
          securityContext:
            privileged: true

There's also example here.

Answer 2

I ended up switching node image from Googles default image cos_containerd to ubuntu containerd. This made the sysctl changes permanent.