Hosting Jenkins on Kubernetes while using docker.sock

Question

I'm trying to host a Jenkins image on GKE to run a build. Mostly, I've followed Google's tutorial for setting up Jenkins in Kubernetes. I've got a fairly basic set-up with one master node which runs the builds.

I also want to be able to use Docker inside of the Jenkins environment, and so I've gone into Jenkins' Global Tools Configuration and added a Docker instance. I've additionally mapped the docker.sock in my deployment file to bypass a "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" problem.

My current deployment looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: master
    spec:
      containers:
      - name: master
        image: jenkins/jenkins:2.95
        ports:
        - containerPort: 8080
        - containerPort: 50000
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 2
          failureThreshold: 5
        env:
        - name: JENKINS_OPTS
          valueFrom:
            secretKeyRef:
              name: jenkins
              key: options
        - name: JAVA_OPTS
          value: '-Xmx1400m'
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-home
        - mountPath: /var/run/docker.sock
          name: docker-socket
        securityContext:
          privileged: true
        resources:
          limits:
            cpu: 500m
            memory: 1500Mi
          requests:
            cpu: 500m
            memory: 1500Mi
      volumes:
      - name: jenkins-home
        gcePersistentDisk:
          pdName: jenkins-home
          fsType: ext4
          partition: 1
      - name: docker-socket
        hostPath:
          path: /var/run/docker.sock

Unfortunately, any builds fail with the following error:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: 
....
dial unix /var/run/docker.sock: connect: permission denied

Most Google searches involving this error seem don't seem to be Kubernetes-related.

What am I missing?

Update: To some extent, this works better if I use this configuration:

spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: master
    spec:
      securityContext:
        runAsUser: 0
      containers:

which basically runs Jenkins as root. Unfortunately, that has some consequences for the way that pipelines manage auth credentials. When I try to use "withRepository(repoName, credentialId)", the pipeline adds an entry to /var/jenkins_home/.dockercfg, but a later docker push step doesn't seem to be able to find those credentials (I think that later step ends up looking in /root/.dockercfg or /root/.docker/config.json).

Aha! If I use a different securityContext with "runAsUser 0", it works. — bcholmes, Dec 19 '17 at 18:05
I assume, If jenkins is running by Jenkin user then you can add Jenkin user to the docker group. then you do not need to use Root User. It would be more relevant and secure. — Suresh Vishnoi, Dec 19 '17 at 23:09
Yeah, sadly I'm not sure how to do that. Because this is hosted in a Kubernetes environment, it's not like I can SSH into a server and do a groupadd. But maybe that's just something I need to figure out. Thanks for the comment! — bcholmes, Dec 20 '17 at 15:17
I did not see update note at the first sight. I think you should make it bigger, it resolved the problem here, thank you. — Mustafa Güven, Jul 23 '19 at 14:33

score 3 · Answer 1 · answered Jan 05 '18 at 22:05

You might want to try running it as the 1000 user:

...
spec:
  ...
  securityContext:
    # Specify fsGroup for pod, so that the persistent volume is writable for the non-privileged uid/gid 1000
    runAsUser: 1000
    fsGroup: 1000
  ...

You may also find this Helm chart useful.

score 2 · Answer 2 · answered Jun 04 '18 at 11:25

I hope this helps someone who faced this problem and changing volume type to file of runAsUser to 1000 didn't work.

For me setting the runAsUser : 0 worked for me.

I am not sure if this is adviced as per this post. But you can try to set the DOCKE_HOST env variable and check if that works for you as well.

score 1 · Answer 3 · answered Dec 21 '17 at 16:32

1

volumes:  
- name: docker-sock-volume
          hostPath:
            path: /var/run/docker.sock
            type: File

Try to specify type: File

answered Dec 21 '17 at 16:32

Nicola Ben

10,615
8
41
65

Thanks for the suggestion. I hafta confess that I ran in to a variety of other problems with the approach of using Jenkins with Docker in a Kubernetes environment and bailed on that set-up, so I'm not really in a place to test this suggestion out, now. – bcholmes Dec 29 '17 at 16:27

Hosting Jenkins on Kubernetes while using docker.sock

3 Answers3