How can i access a k8s worker node from a master node without ssh?

Question

I want to build a python script that accesses a kubernetes worker node, lists the network interfaces and captures the traffic of the choosen interface (with tcpdump for exemple) and then store the pcap file somewhere on the master node.

I want to know if it is possible to access a worker node from my master node without ssh? (maybe with a direct k8s-apiserver call ?) If ssh is the only way to access the worker node, how can i make a connection without entring the worker's password (for authentification).

Or maybe there is another way to do this ?

IME you can't directly access either the worker or master nodes, ssh or otherwise. This is especially true if you're using a managed Kubernetes (EKS, GKE, AKS, ...) and you're not on the operations team. — David Maze, May 28 '22 at 17:09
i'm assuming that i will lunch my script on the master node ... — Youva, May 28 '22 at 17:36
See kubectl debug docs, you have something like `kubectl debug node/mynode -it --image=busybox` to start a container on your nodes, that should let you access nodes filesystem, its pid namespace, ... — SYN, May 29 '22 at 06:00

SYN · Accepted Answer · 2022-05-29T10:34:24.253

One way to connect nodes, when SSH isn't an option, would be to start some privileged container, that would access your nodes filesystem, disabling pid namespace isolation.

Say I have some node "infra1"

$> kubectl get nodes
infra1     Ready         infra         728d   v1.21.6

I can get in using:

$ kubectl debug node/infra1 -it --image=busybox
Creating debugging pod node-debugger-infra1-s46g6 with container debugger on node infra1.
If you don't see a command prompt, try pressing enter.
/ #
/ # chroot /host
root@infra1:/# 
root@infra1:/# crictl ps | head -2
CONTAINER           IMAGE               CREATED             STATE               
NAME                       ATTEMPT             POD ID
a36863efe2a2e       3fb5cabb64693       4 minutes ago       Running

The /host being a "volume", sharing my host filesystem with that debug container. Using chroot, you're now working from your node runtime.

$ k get pods 
NAME                                      READY   STATUS    RESTARTS   AGE
node-debugger-infra1-g5nwg                0/1     Error     0          71s
node-debugger-infra1-s46g6                1/1     Running   0          55s

In practice, this is done creating a Pod, such as the following:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: hostaccess
  name: node-debugger-infra1-s46g6
  namespace: default
spec:
  containers:
  - image: busybox
    ...
    volumeMounts:
    - mountPath: /host
      name: host-root
      ...
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostIPC: true
  hostNetwork: true
  hostPID: true
  nodeName: infra1
  nodeSelector:
    kubernetes.io/os: linux
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - operator: Exists
  volumes:
  - hostPath:
      path: /
      type: ""
    name: host-root
  ...

Answering your follow-up question, regarding caliXXX interfaces. Those are specific to calico SDN, although the same remarks may apply to other implementations: there's no easy way to resolve pods IPs from those.

We can however inspect pods configuration, figuring out which interface they use:

# crictl pods
...
b693b9ff6487c       3 hours ago         Ready               controller-6b78bff7d9-5b7vr                 metallb-system          2                   (default)

# crictl inspectp b693b9ff6487c | jq '.info.cniResult'
{
  "Interfaces": {
    "cali488d4aeb0e6": {
      "IPConfigs": null,
      "Mac": "",
      "Sandbox": ""
    },
    "eth0": {
      "IPConfigs": [
        {
          "IP": "10.233.105.55",
          "Gateway": ""
        }
      ],
      "Mac": "",
      "Sandbox": ""
    },
    "lo": {
      "IPConfigs": [
        {
          "IP": "127.0.0.1",
          "Gateway": ""
        },
        {
          "IP": "::1",
          "Gateway": ""
        }
      ],
      "Mac": "00:00:00:00:00:00",
      "Sandbox": "/var/run/netns/cni-55322514-e37a-2913-022a-9f7488df8ca5"
    }
  },
  "DNS": [
    {},
    {}
  ],
  "Routes": null
}

Then, resolving the interface name for a given IP, we could so something like:

# MATCH_IP="10\.233\.105\.55"
# crictl pods | awk '/Ready/{print $1}' | while read pod
    do
        crictl inspectp $pod | grep $MATCH_IP >/dev/null 2>&1 || continue
        echo found pod $pod
        crictl inspectp $pod \
           | jq '.info.cniResult.Interfaces | with_entries(select(.key|match("cali"))) | to_entries[] | .key'
        break
    done
found pod b693b9ff6487c
"cali488d4aeb0e6"

thank you this is a good alternative :) i just have one more question: when i list network interfaces on a worker node, the pod's interface is named "cali@xxx" or something like that, so my question is how can i know this interface ip address without exploring the namespace of the pod located in /var/run/netns ? — Youva, May 29 '22 at 08:59
added a sample in my answer. although such problematics would have different answers depending on which SDN and container runtime you would use. In my sample, SDN matches (calico), although there's no guarantee my "crictl" commands would work for you — SYN, May 29 '22 at 10:36

How can i access a k8s worker node from a master node without ssh?

1 Answers1