1

I have an issue with a gitlab-runner installed via the application panel on gitlab to a k3s rancher cluster. It was working fine then today gitlab was restarted and started giving out this error.

I found out where the gitlab certificate is and where to copy it but I do not have sudo on the POD :

bash-5.0$ cd gitlab-runner/
bash: cd: gitlab-runner/: Permission denied
bash-5.0$ pwd

If i do not install the Runner via the application panel from the Gitlab site it doesn't work properly (meaning I install it via helm with -f values but it will not work with Autodevops and it will not create a pod for each CI as expected)

Any workaround or solution would be greatly appreciated :)

Thank you in advance.

CodeM7
  • 93
  • 2
  • 12

2 Answers2

2

In the same boat here. Solved my issue copying my self signed certs to all my worker nodes at

user@worker01:~$ sudo cp gitlab.hostname.com.crt /usr/local/share/ca-certificates/
user@worker02:~$ sudo cp gitlab.hostname.com.crt /usr/local/share/ca-certificates/
user@worker03:~$ sudo cp gitlab.hostname.com.crt /usr/local/share/ca-certificates/

After this run the cert update. In my case ubuntu we can run the command below on all worker nodes

sudo update-ca-certificates --fresh

We may need to restarted all worker nodes, also a k3s-agent restart would have done the same.

sudo systemctl status k3s-agent.service
Tiago
  • 21
  • 3
  • yes the restart is key, but isnt sudo dpkg-reconfigure ca-certificates doing that ? also just grab some external service and grab a subdomains certificate so you can use it for your cluster your monitoring your gitlab etc etc https://www.youtube.com/watch?v=yR9aP1WfPCM&ab_channel=DanielUrra then just cron job to copy it over – CodeM7 Feb 09 '22 at 09:28
  • Yes thanks for pointing out actually we should not edit manually the "/etc/ssl/certs/ca-certificates.crt" as it will be updated dynamically and accordingly with the "sudo update-ca-certificates --fresh" – Tiago Feb 10 '22 at 14:53
  • And for SAN Subject Alternative Name I'm using let's encrypt configured on my Gitlalb CE omnibus something like this: GITLAB_OMNIBUS_CONFIG: | letsencrypt['auto_renew'] = true letsencrypt['alt_names'] = ['name1.hostname.com','name2.hostname.com','name3.hostname.com'] – Tiago Feb 10 '22 at 14:55
  • yes that works, my GitLab is also only internal cannot use that one :) – CodeM7 Feb 10 '22 at 15:04
1

After many tests and failures.

for the runner I am using a helm chart and create a secret with the certificate as secribed in the helm values.yaml

kubectl create secret generic gitlab-runner --from-file=10.0.0.2.nip.io.crt -n gitlab -apps

gitlab-ci.yaml

entrypoint: ["dockerd-entrypoint.sh"]
  command: ["--insecure-registry", "10.0.0.2.nip.io:5005"]

Self-signed cert:

apt-get install ca-certificates
cp cacert.pem /usr/share/ca-certificates
sudo dpkg-reconfigure ca-certificates

The correct way is to get it working was to set up an actual domain name and change GitLab's external_url to that so that Let's encrypt can get a renewed certificate since then it started working fine.

CodeM7
  • 93
  • 2
  • 12