3

I am finding all sorts of walkthroughs on how to add certificates to be used in the pods themselves, but I can't seem to find info on how to setup Kubernetes to allow a self-signed cert for pulling images from a Harbor instance running inside the cluster. I have the ca cert imported to the system's trusted certs (system is running Ubuntu 18.04), but I am guessing Kubernetes uses it's own trusted certs store somewhere, similar to how Java ignores the system's trusted certs and relies on keystore files?

Edit To be more specific, what I am trying to do is deploy a custom Docker image stored in my Harbor instance that is running in my Kubernetes cluster. I have the certs for Harbor imported into my host systems OS, and I can run:

docker login <url_to_harbor>
docker pull <url_to_harbor>/library/custom/image:latest

and it works fine from CLI, but if I try to create a deployment yaml like so:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom
  template:
    metadata:
      labels:
        app: custom
    spec:
      containers:
      - name: custom
        image: <url_to_harbor>/library/custom/image:latest
...

and run kubectl apply -f custom-deploy.yaml I get the following error when I get pods:

custom-deployment-6ff68947f6-8jj2p            0/1     ImagePullBackOff   0          13s

And if I get a description on the failed pod I see:

  Warning  Failed          18s                kubelet, node3     Failed to pull image "<url_to_harbor>/library/custom/image:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://<url_to_harbor>/v2/: x509: certificate signed by unknown authority
  Warning  Failed          18s                kubelet, node3     Error: ErrImagePull

I have restarted the host machine since I imported the CA certs. I added the certs by moving the root CA cert file into /usr/local/share/ca-certificates and then running sudo update-ca-certificates.

CodeChimp
  • 323
  • 1
  • 6
  • 16
  • 1
    Have you restarted the container engine after importing the ca cert into kubernetes nodes? What's the result of `docker pull ` directly inside one of your kubernetes nodes? – oldgiova Jun 07 '20 at 08:25
  • @rgio I added some addition details to the original post. – CodeChimp Jun 07 '20 at 14:49
  • 1
    You talked about a single host machine: actually you have to add ca cert to every kubernetes node: all workers and masters. What kind of kubernetes cluster do you have? – oldgiova Jun 07 '20 at 17:05
  • I have 4 servers, one as the master and 3 additional nodes. I only added the cert to the master. Let me try to add the certs to the other nodes and restart. – CodeChimp Jun 08 '20 at 10:59

2 Answers2

4

The method of importing CA cert is correct, but you have to do that on every cluster node: all masters and workers. Kubernetes simply relies on CAs installed on underlying operating system.

oldgiova
  • 436
  • 2
  • 5
1

If you're in a situation where you may not want to install the CA cert on every node, you can also run a DaemonSet which configures the ca-certificate. Note, this will only work if Kubernetes is configured with the containerd runtime.

apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-ca
  namespace: kube-system
data:
  ca.crt: |+
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: setup-script
  namespace: kube-system
data:
  setup.sh: |
    echo "$TRUSTED_CERT" > /usr/local/share/ca-certificates/ca.crt && update-ca-certificates && systemctl restart containerd
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: node-custom-setup
  labels:
    k8s-app: node-custom-setup
spec:
  selector:
    matchLabels:
      k8s-app: node-custom-setup
  template:
    metadata:
      labels:
        k8s-app: node-custom-setup
    spec:
      hostPID: true
      hostNetwork: true
      initContainers:
      - name: init-node
        command: ["nsenter"]
        args: ["--mount=/proc/1/ns/mnt", "--", "sh", "-c", "$(SETUP_SCRIPT)"]
        image: debian
        env:
        - name: TRUSTED_CERT
          valueFrom:
            configMapKeyRef:
              name: trusted-ca
              key: ca.crt
        - name: SETUP_SCRIPT
          valueFrom:
            configMapKeyRef:
              name: setup-script
              key: setup.sh
        securityContext:
          privileged: true
      containers:
      - name: wait
        image: k8s.gcr.io/pause:3.1

Source: http://hypernephelist.com/2021/03/23/kubernetes-containerd-certificate.html

These instructions were written prior to the implementation of a host registry directory in cri in containerd 1.5, so if running a new version, it may be possible to configure without restarts. See Registry Configuration - Introduction.

Finally, if you're running on a cloud managed cluster, they may provide a custom configuration for managing CA Certificates. E.g., AKS - Custom CA

NoUserException
  • 128
  • 4