3

I'd like to install a CI tool on GKE so I can run docker-in-docker via Sysbox.

In order to do that, I need to install the Sysbox container runtime.

Is it possible to do that in GKE?

Jeff Dean
  • 53
  • 6

4 Answers4

7

Deployment of pods with Sysbox is now officially supported, and does work on GKE.

The makers of Sysbox (Nestybox, where I am one of the founders) created a Kubernetes daemonset to make the installation easy and persistent across node recreations. The daemonset installs Sysbox on whichever Kubernetes nodes you wish. Sysbox is installed along-side the default runtime (typically the OCI runc), and you can select which pods get deployed with Sysbox and which with the default runtime.

The advantage of deploying pods with Sysbox is that they are strongly isolated via the Linux user-namespace (i.e., rootless) and are capable of running not just microservices, but system software such as systemd, Docker itself, even Kubernetes. This is very useful when using containers for CI/CD, dev environments, learning environments, running legacy apps, and more.

ctalledo
  • 341
  • 3
  • 5
2

The Sysbox runtime does not yet support integration with K8s yet (we are working on that).

The limitation arises from the fact that Sysbox always deploys containers with the Linux user-namespace (for improved isolation), so it requires K8s pods to use the Linux user-namespace too. The CRI-O runtime has initial support for this already, so we expect to have this working very soon. See this sysbox issue for details. It's likely containerd will also add this very soon.

To provision Sysbox on a GKE node, we are looking into a K8s native approach, such as a daemonSet, that would a user to easily get the runtime into the node and configure K8s with an option to use it. Once provisioned, user would select the runtime for your pod via a K8s RuntimeClass.

ctalledo
  • 341
  • 3
  • 5
  • The K8s native solution sounds great. We are looking for a similar solution for Azure Kubernetes Services (AKS). – Derek Feb 15 '21 at 12:58
1

There is now a preview version of Sysbox that works with Kubernetes.

This enables Kubernetes to deploy rootless pods that act like "light-weight VMs" and can run systemd, Docker, and even Kubernetes inside (without actually using VMs).

You can find it here: https://github.com/nestybox/sysbox-pods-preview

ctalledo
  • 341
  • 3
  • 5
0

GKE is a provider-managed Kubernetes solution where certain aspects are handled by your cloud provider. Things like nodes provisioning, automatic updates, security patching, access to certain storage solutions out of the box, etc.

Focusing specifically on Nodes. Kubernetes relies on Container Runtime to run it's containers. GKE has predefined images with specific Container Runtimes. You can see by below table:

OS Node images
Container-Optimized OS Container-Optimized OS with Containerd (cos_containerd), Container-Optimized OS with Docker (cos)
Ubuntu Ubuntu with Containerd (ubuntu_containerd), Ubuntu with Docker (ubuntu)
Windows Server Windows Server LTSC (windows_ltsc), Windows Server SAC (windows_sac)

-- Cloud.google.com: Kubernetes Engine: Docs: Concepts: Node images

As you can see the Container Runtimes are already predetermined. There are differences between the OS's which are prepared to run within GKE ecosystem. For example, COS does not have an inherent package manager (for that you should use CoreOS Toolbox).

Adding to that:

Node VM modifications

Modifications on the boot disk of a node VM do not persist across node re-creations. Nodes are re-created during manual upgrade, auto-upgrade, auto-repair, and auto-scaling. In addition, nodes are re-created when you enable a feature that requires node re-creation, such as GKE sandbox, intranode visibility, and shielded nodes.

To preserve modifications across node re-creation, use a DaemonSet.

It's not recommended to manage critical software provided by a node image, such as the kernel or container runtime (whether containerd or docker). Node images are tested extensively, and modifying critical software provided in the node image puts the node into an unknown and untestable state.

-- Cloud.google.com: Kubernetes Engine: Docs: Concepts: Node images: Modifications

Additional resources:

Dawid Kruk
  • 8,982
  • 2
  • 22
  • 45