A DaemonSet for kubernetes works fine while in privileged mode, but fails even if all linux capabilties are added

Question

I have a Daemonset running in privileged mode in a kubernetes cluster. This is the YAML spec of the daemon set.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: my-daemon
spec:
  template:
    metadata:
      labels:
        app: my-daemon
    spec:
      hostNetwork: true
      serviceAccountName: my-sa-account
      containers:
      - name: my-daemon
        image: akhilerm/my-daemon:0.5
        imagePullPolicy: Always
        securityContext:
          privileged: true
...
...

Instead of using privileged:true, I am moving on to linux capabilties to give permissions to the DaemonSet. Therefore, I added all the linux capabilities to the container and removed privileged:true. This is the new YAML spec

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: my-daemon
spec:
  template:
    metadata:
      labels:
        app: my-daemon
    spec:
      hostNetwork: true
      serviceAccountName: my-sa-account
      containers:
      - name: my-daemon
        image: akhilerm/my-daemon:0.5
        imagePullPolicy: Always
        securityContext:
          capabilities:
            add: ["NET_BROADCAST", "NET_ADMIN", ..all CAPs..,"SYS_ADMIN"]
...
...

But when using with linux capabilities the daemon is not behaving as expected. In both cases the permission bitmap in /proc/1/status inside the container is same.

...
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000010000
SigIgn: 0000000000000004
SigCgt: 0000000000014002
CapInh: 0000003fffffffff
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
...

Is there any more fields or permissions that I need to set while using linux capabilities with a pod in kubernetes?

Answer 1

I am not 100% sure, but this issue may be in the missing capabilities: field.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: my-daemon
spec:
  template:
    metadata:
      labels:
        app: my-daemon
    spec:
      hostNetwork: true
      serviceAccountName: my-sa-account
      containers:
      - name: my-daemon
        image: akhilerm/my-daemon:0.5
        imagePullPolicy: Always
        securityContext:
          capabilities:
            add: ["NET_BROADCAST", "NET_ADMIN", ..all CAPs..,"SYS_ADMIN"]
...
...

or

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: my-daemon
spec:
  template:
    metadata:
      labels:
        app: my-daemon
    spec:
      hostNetwork: true
      serviceAccountName: my-sa-account
      containers:
      - name: my-daemon
        image: akhilerm/my-daemon:0.5
        imagePullPolicy: Always
        securityContext:
          capabilities:
            add:
            - NET_BROADCAST
            - NET_ADMIN
            - ...
            - SYS_ADMIN
...
...

Test it and let me know if there is any success. If not - I will try to dig deeper and provide you another solution. I found exactly this format in all the examples, so hope it will help. As a real example, I can provide you kubernetes-the-not-so-hard-way-with-ansible-ingress-with-traefik article with next explanations:

securityContext:
  capabilities:
    drop:
    - ALL
    add:
    - NET_BIND_SERVICE

Without this setting we won’t be able to bind Traefik on port 80 and 443 (which is basically true for all services that want to use ports < 1024). We can also use privileged: true but this makes the attack surface way bigger. So using Linux Capabilities give us fine-grained control over superuser permissions and reduce the permissions to a minimum.

Or 1 more from Kubernetes official security-context article, but this time configuration is for the Pod:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo-4
spec:
  containers:
  - name: sec-ctx-4
    image: gcr.io/google-samples/node-hello:1.0
    securityContext:
      capabilities:
        add: ["NET_ADMIN", "SYS_TIME"]

Hope this will help you.