helm: x509: certificate signed by unknown authority

Question

I'm using Kubernetes and I recently updated my admin certs used in the kubeconfig. However, after I did that, all the helm commands fail thus:

Error: Get https://cluster.mysite.com/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: x509: certificate signed by unknown authority

kubectl works as expected:

$ kubectl get nodes
NAME                                           STATUS    ROLES     AGE       VERSION
ip-10-1-0-34.eu-central-1.compute.internal     Ready     master    42d       v1.7.10+coreos.0
ip-10-1-1-51.eu-central-1.compute.internal     Ready     master    42d       v1.7.10+coreos.0
ip-10-1-10-120.eu-central-1.compute.internal   Ready     <none>    42d       v1.7.10+coreos.0
ip-10-1-10-135.eu-central-1.compute.internal   Ready     <none>    27d       v1.7.10+coreos.0
ip-10-1-11-71.eu-central-1.compute.internal    Ready     <none>    42d       v1.7.10+coreos.0
ip-10-1-12-199.eu-central-1.compute.internal   Ready     <none>    8d        v1.7.10+coreos.0
ip-10-1-2-110.eu-central-1.compute.internal    Ready     master    42d       v1.7.10+coreos.0

As far as I've been able to read, helm is supposed to use the same certificates as kubectl, which makes me curious as how how kubectl works, but helm doesn't?

This is a production cluster with internal releases handled through helm charts, so it being solved is imperative.

Any hints would be greatly appreciated.

I'm not that familiar with Helm, but does the assumption that "`helm` is supposed to use the same certificates as `kubectl`" come from the documentation? — Paweł Prażak, Jan 05 '18 at 19:55
But in any case, the the Helm -> Tiller communication fails or Tiller -> Kube API? — Paweł Prażak, Jan 05 '18 at 19:56
Thanks for your comment, Paweł! Yes, reading about this issue om GKE (I'm using AWS), this error means that "your local Kubernetes config file" must have the correct credentials". https://github.com/kubernetes/helm/blob/master/docs/install_faq.md — Helge Talvik Söderström, Jan 05 '18 at 19:59
Helm to Tiller, I'm pretty sure. When I tried to see if the certificates for the SA matched, I just found the CA, and it was the correct one. — Helge Talvik Söderström, Jan 05 '18 at 20:00
is it possible that you have two different `kubeconfig`s and `kubectl` is using one (e.g. a new one) and `helm` is using the other (e.g. an old one?) — Paweł Prażak, Jan 05 '18 at 20:05
I doubt it. I've been using the `$KUBECONFIG` env variable, which is read by both according to the docs. — Helge Talvik Söderström, Jan 05 '18 at 20:08
as a workaround you could try `insecure-skip-tls-verify: true` make helm working again till the root cause is fixed — Sebastian, Jan 05 '18 at 22:30
Where would I set that, Sebastian? As far as I know, there is no cli flag for it. — Helge Talvik Söderström, Jan 06 '18 at 13:44
@HelgeTalvikSöderström I put it in an answer as it is too much text — Sebastian, Jan 06 '18 at 20:18

Sebastian · Accepted Answer · 2018-01-07T23:30:41.060

25

As a workaround you can try to disable certificate verification. Helm uses the kube config file (by default ~/.kube/config). You can add insecure-skip-tls-verify: true for the cluster section:

clusters:
- cluster:
    server: https://cluster.mysite.com
    insecure-skip-tls-verify: true
  name: default

Did you already try to reinstall helm/tiller?

kubectl delete deployment tiller-deploy --namespace kube-system
helm init

Also check if you have configured an invalid certificate in the cluster configuration.

edited Jan 07 '18 at 23:30

answered Jan 06 '18 at 20:18

Sebastian

16,813
4
49
56

I haven't tried deleting the deployment, but if I do, what will happen to existing helm deployments? – Helge Talvik Söderström Jan 07 '18 at 11:14
After running `helm init`, I still get `x509: certificate signed by unknown authority`. – Helge Talvik Söderström Jan 07 '18 at 12:02
I found the issue. I marked your answer as the solution, but I'd like you to make a small edit to be the actual response - in the cluster configuration, I had pointed to a specific certificate for debugging purposes. I'm not sure why it worked with `kubectl` and not `helm`, but removing my certificate there and letting the `kubeconfig` file specified in `$KUBECONFIG` work instead solved my issue. Thanks for your help! – Helge Talvik Söderström Jan 07 '18 at 12:50
@HelgeTalvikSöderström Great that you solved the issue. I have edited my answer. – Sebastian Jan 07 '18 at 23:37
What is the risk of commenting out the `.clusters[*].cluster.certificate-authority-data` in kube config? – LexH Nov 01 '18 at 18:21

score 7 · Answer 2 · answered Oct 17 '20 at 03:10

In my case, I was running for a single self-manage and the config file was also container ca-file, so the following the above answer was throwing below error

Error: Kubernetes cluster unreachable: Get "https://XX.XX.85.154:6443/version?timeout=32s": x509: certificate is valid for 10.96.0.1, 172.31.25.161, not XX.XX.85.154

And my config was

- cluster:
    certificate-authority-data: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    server: https://54.176.85.154:6443
    insecure-skip-tls-verify: true

So I had to remove the certificate-authority-data.

- cluster:
    server: https://54.176.85.154:6443
    insecure-skip-tls-verify: true

brass monkey · Answer 3 · 2020-01-29T12:52:02.073

4

In my case the error was caused by an untrusted certificate from the Helm repository. Downloading the certificate and specifying it using the --ca-file option solved the issue (at least in Helm version 3).

helm repo add --ca-file /path/to/certificate.crt repoName https://example/repository

--ca-file string, verify certificates of HTTPS-enabled servers using this CA bundle

edited Jan 29 '20 at 12:52

answered Jan 29 '20 at 12:15

brass monkey

5,841
10
36
61

2

how did you download it ? – Abdennour TOUMI Dec 31 '20 at 06:47

score 4 · Answer 4 · edited Mar 30 '22 at 12:56

4

Use --insecure-skip-tls-verify to skip tls verification via command line

helm repo add stable --insecure-skip-tls-verify https://charts.helm.sh/stable

edited Mar 30 '22 at 12:56

Tobb

11,850
6
52
77

answered Mar 30 '22 at 12:51

nitu kiran

51
1

score 2 · Answer 5 · answered Nov 03 '21 at 01:28

Adding the line below the -cluster to /home/centos/.kube/config file fixed my issue

insecure-skip-tls-verify: true

fixed my issue.

my config file now looks like this.

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/centos/.minikube/ca.crt
    extensions:
    - extension:
        last-update: Tue, 02 Nov 2021 20:51:44 EDT
        provider: minikube.sigs.k8s.io
        version: v1.23.2
      name: cluster_info
    server: https://192.168.49.2:8443
    insecure-skip-tls-verify: true
  name: minikube
contexts:

score 1 · Answer 6 · answered Nov 24 '22 at 11:40

1

I encountered an edge case for this. You can also get this error if you have multiple kubeconfig files referenced in the KUBECONFIG variable, and more than one file has clusters with the same name.

answered Nov 24 '22 at 11:40

Audun Nes

124
5

score 0 · Answer 7 · answered May 31 '22 at 18:19

0

For my case, it was an old version of helm (v. 3.6.3 in my case) after I upgraded to helm v.3.9.0 brew upgrade helm everything worked again.

answered May 31 '22 at 18:19

Neal Sha

87
1
2

score -1 · Answer 8 · answered Feb 07 '20 at 12:33

Although adding repo with --ca-file did the thing, when it tried to download from that repo with the command posted under, I still got the x509: certificate signed by unknown authority

helm dependency update helm/myStuff
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "myRepo" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 18 charts
Downloading myService from repo https://myCharts.me/
Save error occurred:  could not download https://myCharts.me/stuff.tgz ...
x509: certificate signed by unknown authority
Deleting newly downloaded charts, restoring pre-update state

What I needed to do, apart from adding repo with --ca-file was to download the repository certificate and install it as Current User:

Place all certificates in the following store: Trusted Root Certification Authorities:

After installing the certificate I also needed to restart the computer. After restart, when you open the browser and paste the repo URL it should connect without giving a warning and trusting the site (this way you know you installed the certificate successfully).

You can go ahead and run the command, it should pick the certificate this time.

helm dependency update helm/myStuff
....
Saving 18 charts
Downloading service1 from repo https://myCharts.me/
Downloading service2 from repo https://myCharts.me/
....

helm: x509: certificate signed by unknown authority

8 Answers8

Linked