updates.jenkins.io: Temporary failure in name resolution while building jenkins docker image in docker in docker Jenkins agent running on K8S

Question

I'm using dynamically provisioned docker in docker agent as Kubernetes pod in Jenkins helm chart.

Docker in Docker agent uses 2 images, one with docker client, one with docker daemon. Their Dockerfile is below:

• dind-client

FROM jenkins/jnlp-agent-docker
USER root

COPY entrypoint.sh /entrypoint.sh
RUN chown jenkins:jenkins /entrypoint.sh
RUN chmod +x /entrypoint.sh

USER jenkins
ENTRYPOINT "/entrypoint.sh"

• entrypoint.sh busy waits for docker-daemon to become available:

#!/usr/bin/env bash

RETRIES=6

sleep_exp_backoff=1

for((i=0;i<RETRIES;i++)); do
    docker version
    dockerd_available=$?
    if [ $dockerd_available == 0 ]; then
        break 
    fi
    sleep ${sleep_exp_backoff}
    sleep_exp_backoff="$((sleep_exp_backoff * 2))"
done

exec /usr/local/bin/jenkins-agent "$@"

• dind-daemon

FROM docker:20.10.14-dind

• docker-agent is configured to run in Jenkins helm chart using below values in values.yaml:

dind:
    podName: docker-agent
    customJenkinsLabels: docker-agent
    image: [url]/docker-client
    tag: latest
    envVars:
     - name: DOCKER_HOST
       value: "tcp://localhost:2375"
    alwaysPullImage: true
    yamlTemplate:  |-  
     spec: 
         containers:
           - name: dind-daemon 
             image: [url]/docker-daemon:latest
             securityContext: 
               privileged: true
             env: 
               - name: DOCKER_TLS_CERTDIR
                 value: ""       

• I'm trying to build below image using the agent:

FROM docker.io/jenkins/jenkins:lts-jdk17

ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"

# Add Jenkins init files
COPY src/ /usr/share/jenkins/ref/

COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli --verbose --plugin-file /usr/share/jenkins/ref/plugins.txt

USER jenkins

• Below error pops during RUN jenkins-plugin-cli --verbose --plugin-file /usr/share/jenkins/ref/plugins.txt stage of docker build command inside the agent during jenkin build using the agent.

12:39:03  Retrieving update center information
12:39:03  Created cache at: /var/jenkins_home/.cache/jenkins-plugin-management-cli
12:39:03  Update center URL: https://updates.jenkins.io/update-center.json?version=2.346.1
12:39:03  Cache miss for: update-center-2.346.1
12:39:03  Unable to retrieve JSON from https://updates.jenkins.io/update-center.json?version=2.346.1: updates.jenkins.io: Temporary failure in name resolution
12:39:03  Unable to retrieve JSON from https://updates.jenkins.io/update-center.json?version=2.346.1: updates.jenkins.io
12:39:03  io.jenkins.tools.pluginmanager.impl.UpdateCenterInfoRetrievalException: Error getting update center json
12:39:03    at io.jenkins.tools.pluginmanager.impl.PluginManager.getJson(PluginManager.java:810)
12:39:03    at io.jenkins.tools.pluginmanager.impl.PluginManager.getUCJson(PluginManager.java:832)
12:39:03    at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:217)
12:39:03    at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:181)
12:39:03    at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:70)
12:39:03  Caused by: java.io.IOException: Unable to retrieve JSON from https://updates.jenkins.io/update-center.json?version=2.346.1: updates.jenkins.io
12:39:03    at io.jenkins.tools.pluginmanager.impl.PluginManager.getViaHttpWithResponseHandler(PluginManager.java:1344)
12:39:03    at io.jenkins.tools.pluginmanager.impl.PluginManager.getJson(PluginManager.java:796)
12:39:03    ... 4 more
12:39:03  Caused by: java.net.UnknownHostException: updates.jenkins.io
12:39:03    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
12:39:03    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
12:39:03    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1367)
12:39:03    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1301)
12:39:03    at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
12:39:03    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
12:39:03    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
12:39:03    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
12:39:03    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
12:39:03    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
12:39:03    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
12:39:03    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
12:39:03    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
12:39:03    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
12:39:03    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
12:39:03    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
12:39:03    at io.jenkins.tools.pluginmanager.impl.PluginManager.getViaHttpWithResponseHandler(PluginManager.java:1336)
12:39:03    ... 5 more
12:39:03  Error getting update center json
12:39:05  The command '/bin/sh -c jenkins-plugin-cli --verbose --plugin-file /usr/share/jenkins/ref/plugins.txt' returned a non-zero code: 1

• Jenkins pipeline runs docker build in sh step:

rc = sh(returnStatus: true, script: "docker build -t ${REGISTRY}/${REPO_NAME}/${NAME} .")

• It seems like the issue is dns issue. More specifically, failure to resolve updates.jenkins.io inside intermediate container which is running during specific docker build stage in docker-daemon container

So I ran the image in the same k8s cluster and tried to resolve updates.jenkins.io locally inside the container, it worked:

kubectl run --image=[...]  --privileged=true dind
kubectl exec -it pod/dind sh
/ # nslookup updates.jenkins.io
...

Non-authoritative answer:
Name:   updates.jenkins.io
Address: 52.202.51.185
...

• I then tried to replicate the failing use case (container running inside docker-daemon container and updates.jenkins.io not resolving inside it) and the resolution worked:

kubectl run --image=[...]  --privileged=true dind
kubectl exec -it pod/dind sh
docker run -it alpine sh
/ # nslookup updates.jenkins.io
...

Non-authoritative answer:
Name:   updates.jenkins.io
Address: 52.202.51.185
...

Not sure why then the dns issue pops during docker build.