GitHub Actions Self-hosted Runner sibling container: permission denied... Docker daemon socket unix:///var/run/docker.sock

Question

I need to execute on GPU hardware so I have to create a self-hosted runner for github actions to execute my code. The self-hosted runner is hosted on my local machine (ubuntu 20.04).

I'm running the self hosted runner container locally with -v and binding the socks using: docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e GITHUB_OWNER=<xxx> -e GITHUB_REPOSITORY=<xxxx>-e GITHUB_PAT=<xxxx>

This local self-hosted runner executes successfully until I try to build the second "project" container I need for my project code. I get a permission issue with the docker sock when I try to build the container not run the container. I'm about 70% certain that with the -v binding when running the self-hosted runner locally this enables sibling containers versus Docker in Docker (which I've read isn't cool anymore).

Permission error:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied

I've tried building the project container with -v /var/run/docker.sock:/var/run/docker.sock in the docker build command but it doesn't like the -v and I've also tried the following approaches in the "project" docker container:

Approach 1.

useradd -m cnncontainer && \
usermod -aG sudo cnncontainer && \
echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
curl -sSL https://get.docker.com/ | sh
usermod -aG docker cnncontainer

Approach 2.

sudo groupadd docker && \
sudo usermod -aG docker "$USER" &&\
newgrp docker
docker run hello-world

Approach 3.

sudo usermod -aG docker $USER
sudo setfacl --modify user:$USER:rw /var/run/docker.sock
docker run hello-world

GitHub actions self-hosted runner Dockerfile:

FROM debian:buster
 #tensorflow/tensorflow:2.3.4-gpu - this image doesn't work either

ARG RUNNER_VERSION="2.298.2"

ENV GITHUB_PERSONAL_TOKEN ""
ENV GITHUB_OWNER ""
ENV GITHUB_REPOSITORY ""

RUN apt-get update \
    && apt-get install -y \
        curl \
        sudo \
        git \
        jq \
        tar \
        gnupg2 \
        apt-transport-https \
        ca-certificates  \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN useradd -m github && \
    usermod -aG sudo github && \
    echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

#setup docker runner 
RUN curl -sSL https://get.docker.com/ | sh
RUN usermod -aG docker github 

USER github
WORKDIR /home/github

#install github actions cli
RUN curl -O -L https://github.com/actions/runner/releases/download/v$RUNNER_VERSION/actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
RUN tar xzf ./actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
RUN sudo ./bin/installdependencies.sh

COPY --chown=github:github entrypoint.sh ./entrypoint.sh
RUN sudo chmod u+x ./entrypoint.sh

ENTRYPOINT ["/home/github/entrypoint.sh"]```

Self-hosted runner entrypoint.sh:

#!/bin/sh
registration_url="https://api.github.com/repos/${GITHUB_OWNER}/${GITHUB_REPOSITORY}/actions/runners/registration-token"
echo "Requesting registration URL at '${registration_url}'"

payload=$(curl -sX POST -H "Authorization: token ${GITHUB_PAT}" ${registration_url})
export RUNNER_TOKEN=$(echo $payload | jq .token --raw-output)

./config.sh \
    --name $(hostname) \
    --token ${RUNNER_TOKEN} \
    --url https://github.com/${GITHUB_OWNER}/${GITHUB_REPOSITORY} \
    --work ${RUNNER_WORKDIR} \
    --unattended \
    --replace

remove() {
    ./config.sh remove --unattended --token "${RUNNER_TOKEN}"
}

trap 'remove; exit 130' INT
trap 'remove; exit 143' TERM



./run.sh "$*" &  #changed from run.sh
### BEGIN
sudo systemctl start docker
sudo systemctl enable docker
export RUNNER_ALLOW_RUNASROOT=true
export AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
mkdir actions-runner
sudo mkdir /opt/hostedtoolcache
cd actions-runner

# Make /actions-runner/_work
mkdir _work

# Link /opt/hostedtoolcache as /actions-runner/_work/_tool
ln -s /opt/hostedtoolcache _work/_tool

### END

wait $!

Dockerfile I want to run in/with the self-hosted runner

FROM tensorflow/tensorflow:2.3.4-gpu

RUN mkdir -p /app
COPY . main.py /app/
WORKDIR /app
RUN sudo apt install -y make && sudo apt-get install python3-pip -y
RUN pip install -r requirements.txt

RUN  sudo usermod -aG docker $USER
RUN  sudo setfacl --modify user:$USER:rw /var/run/docker.sock
RUN  docker run hello-world

CMD [ "main.py" ]
ENTRYPOINT [ "python" ]