Getting "x509: certificate signed by unknown authority" by microk8s

Question

I would like to use microk8s with private registry, but pull image is not working (I'm using self-signed cert):

root@master-1:/var/snap/microk8s/common/var/lib/containerd# microk8s.ctr --debug images pull priv.repo:5000/busybox/hellomicrok8s:latest
DEBU[0000] fetching                                      image="priv.repo:5000/busybox/hellomicrok8s:latest"
DEBU[0000] resolving                                     host="priv.repo:5000"
DEBU[0000] do request                                    host="priv.repo:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.4 request.method=HEAD url="https://priv.repo:5000/v2/busybox/hellomicrok8s/manifests/latest"
ctr: failed to resolve reference "priv.repo:5000/busybox/hellomicrok8s:latest": failed to do request: Head "https://priv.repo:5000/v2/busybox/hellomicrok8s/manifests/latest": x509: certificate signed by unknown authority

here is my containerd-template.tom:

root@master-1:/var/snap/microk8s/common/var/lib/containerd# cat /var/snap/microk8s/current/args/containerd-template.toml
version = 2
oom_score = 0

[grpc]
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0

[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false

[cgroup]
  path = ""

[plugins."io.containerd.grpc.v1.cri"]

  stream_server_address = "127.0.0.1"
  stream_server_port = "0"
  enable_selinux = false
  sandbox_image = "k8s.gcr.io/pause:3.1"
  stats_collect_period = 10
  enable_tls_streaming = false
  max_container_log_line_size = 16384

  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "${SNAPSHOTTER}"
    no_pivot = false
    default_runtime_name = "${RUNTIME}"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v1"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime]
      runtime_type = "io.containerd.runc.v1"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime.options]
        BinaryName = "nvidia-container-runtime"

  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "${SNAP}/opt/cni/bin"
    conf_dir = "${SNAP_DATA}/args/cni-network"

  [plugins."io.containerd.grpc.v1.cri".registry]

    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io", ]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."priv.repo:5000"]
        endpoint = ["https://priv.repo:5000"]

I restarted microk8s via systemctl restart snap.microk8s.daemon-containerd.service && microk8s.stop && microk8s.start. Command docker login docker https://priv.repo:5000 is working and I can pull that image via docker pull priv.repo:5000/busybox/hellomicrok8s:latest. Do you know why it is not working?

Thanks in advance!

EDIT:

This is also set:

root@master-1:/var/snap/microk8s/common/var/lib/containerd# cat /etc/docker/daemon.json
{
    "insecure-registries" : ["priv.repo:5000"]
}

EDIT1:

This is working: microk8s.ctr --debug images pull -u ???:??? --skip-verify priv.repo:5000/busybox/hellomicrok8s:latest. How should I set --skip-verify, because when I create a pod via microk8s kubectl apply -f ... still getting x509: certificate signed by unknown authority.

I'm using ssl termination by haproxy. Should I mount them somewhere? — Zili, Jul 16 '20 at 15:57

score 5 · Accepted Answer · answered Jul 17 '20 at 08:57

5

I added my crt file to /etc/ssl/certs (on master node) and it started working.

BTW newly added rows in containerd-template.tom file are not needed for me.

answered Jul 17 '20 at 08:57

Zili

455
2
4
10

First I added it in master node but it didn't work. After I add it to worker to ssl directory now it's working. Thanks – Mahdi Karami Nov 03 '22 at 13:55

score 2 · Answer 2 · answered Feb 25 '21 at 10:01

2

I had the same issue and these commands below may fix this issue for others

openssl s_client -showcerts -connect <IP>:<PORT>< /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ca.crt
cp ca.crt /etc/ssl/certs
update-ca-certificates

answered Feb 25 '21 at 10:01

Hoàng Công Phước

31
2

score 1 · Answer 3 · answered Apr 16 '21 at 12:06

if you are using ubuntu microk8s cert-manager, you can fetch the certificate and install it like this:

Find the correct certificates name (you could have multiple)

microk8s kubectl get secrets -n cert-manager  --field-selector type=kubernetes.io/tls

if the correct name is e.g. dev-da

microk8s kubectl -n cert-manager get secrets dev-ca -o jsonpath='{.data.ca\.crt}' | base64 -d  > cert-manager-ca.crt
sudo cp cert-manager-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Hereafter, you can check with curl, if the certificate is installed correctly. And when genstart microk8s.

microk8s stop && microk8s start

Getting "x509: certificate signed by unknown authority" by microk8s

3 Answers3