7

I'm trying to debug a distroless POD by trying kubectl debug which uses ephemeral containers that are disabled by default in aws eks. I'm using aws eks 1.21

✗  kubectl debug -it opentelemetry-collector-agent-6hqvf --image=busybox --target=some-app

error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource")

So how to enable them for EKS?

sysadmin1138
  • 133,124
  • 18
  • 176
  • 300
DmitrySemenov
  • 835
  • 2
  • 15
  • 27
  • 1
    According to [the fine manual](https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/), they "are not suitable for production clusters" so it's likely EKS considers them a stability risk. If you need that level of control, consider using [EKS Anywhere](https://github.com/aws/eks-anywhere#readme) or your own distro for troubleshooting – mdaniel Oct 16 '21 at 19:57
  • There's a EKS roadmap issue tracking "feature gates" you'd need to allow ephemeral containers https://github.com/aws/containers-roadmap/issues/512 – russau Jun 16 '22 at 06:48

2 Answers2

3

The Kubernetes v1.23 release (not yet available on EKS) moved the ephemeral containers feature from alpha to beta.

When EKS adds support for Kubernetes v1.23, it's possible that this will include the ephemeral container API. It's not possible to say with confidence which way AWS will go here.

If EKS does add support for this API, make sure that you're applying appropriate restrictions. The ability to run an ephemeral container changes the attack surface for your workload and for the cluster it runs on.

Tim B
  • 81
  • 3
  • 2
    According to the [announcement](https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-kubernetes-1-23/) "Ephemeral containers are now beta in Kubernetes 1.23(in EKS), and are enabled by default." – Timur Bakeyev Sep 25 '22 at 08:50
1

AWS EKS 1.23 now includes the feature

✗  k exec -ti ebs-csi-node-gn5g6 -- bash   
Defaulted container "ebs-plugin" out of: ebs-plugin, node-driver-registrar, liveness-probe, debugger-sbn42 (ephem)
OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
command terminated with exit code 126

and via ephemeral containers:

✗  k debug -ti ebs-csi-node-gn5g6 --image=busybox --target=ebs-plugin
Targeting container "ebs-plugin". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-fzpnj.
If you don't see a command prompt, try pressing enter.
/ # ls
bin   dev   etc   home  proc  root  sys   tmp   usr   var
/ #
DmitrySemenov
  • 835
  • 2
  • 15
  • 27