Docker-in-Docker in AKS

Question

We have been tasked with setting up a container-based Jenkins deployment, and there is strong pressure to do this in AKS. Our Jenkins needs to be able to build other containers. Normally I'd handle this with a docker-in-docker approach by mounting /var/run/docker.sock & /usr/bin/docker into my running container.

I do not know if this is possible in AKS or not. Some forum posts on GitHub suggest that host-mounting is possible but broken in the latest AKS relase. My limited experimentation with a Helm chart was met with this error:

Error: release jenkins4 failed: Deployment.apps "jenkins" is invalid:
[spec.template.spec.initContainers[0].volumeMounts[0].name: Required 
value, spec.template.spec.initContainers[0].volumeMounts[0].name: Not 
found: ""]

The change I made was to update the volumeMounts: section of jenkins-master-deployment.yaml and include the following:

  -
  type: HostPath
  hostPath: /var/run/docker.sock
  mountPath: /var/run/docker.sock

Is what I'm trying to do even possible based on AKS security settings, or did I just mess up my chart?

If it's not possible to mount the docker socket into a container in AKS, that's fine, I just need a definitive answer.

Thanks,

Answer 1

Well, we did this a while back for VSTS (cloud TFS, now called Azure DevOps) build agents, so it should be possible. The way we did it is also with mounting the docker.sock

The relevant part for us was:

    ... container spec ...
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: docker-volume
  volumes:
  - name: docker-volume
    hostPath:
      path: /var/run/docker.sock

Answer 2

I have achieved the requirement using following manifests.

Our k8s manifest file carries this securityContext under pod definition.

securityContext:
 privileged: true

In our Dockerfile we were installing Docker-inside-Docker like this way

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install curl wget -y
RUN apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release -y
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg |  gpg --dearmor -o /etc/apt/keyrings/docker.gpg

RUN echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" |  tee /etc/apt/sources.list.d/docker.list > /dev/null

RUN apt-get update
RUN apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
# last two lines of Dockerfile
COPY ./agent_startup.sh .
RUN chmod +x /agent_startup.sh
CMD ["/usr/sbin/init"]
CMD ["./agent_startup.sh"]

Content of agent_startup.sh file

#!/bin/bash
echo "DOCKER STARTS HERE"
service --status-all 
service docker start
service docker start
docker version
docker ps
echo "DOCKER ENDS HERE"
sleep 100000

Sample k8s file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: build-agent
  labels:
    app: build-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: build-agent
  template:
    metadata:
      labels:
        app: build-agent
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: build-agent
          image: myecr-repo.azurecr.io/buildagent
          securityContext:
            privileged: true

When Dockerized agent pool was up, docker daemon was running inside docker container.

My Kubectl version

PS D:\Temp\temp> kubectl.exe version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.22.6
WARNING: version difference between client (1.25) and server (1.22) exceeds the supported minor version skew of +/-1

pod shell output:

root@**********-bcd967987-52wrv:/# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

**Disclaimer: Our kubernetes cluster version is 1.22 and base image is Ubuntu-18.04 and tested only to check if docker-inside-docker is running and not registered with Azure DevOps. You can modify startup script according to your need **