2

I am running multinode k8s cluster on my workstation. I have created the VM's using multipass. K8s cluster is deployed using microk8s.

Microk8s provides private registry as a addon.

It is providing 32000 as node port for the registry.

I would like to connect to this cluster remotely and push docker images to this registry.

I have added 172.**.44.***:32000 as insecure registery in my docker settings in my remote pc.

Note: 172.**.44.*** is the IP of the cluster (something you get in kubectl cluster-info)

But I am unable to make it work

docker build -t 172.**.44.***:32000/myapp:v1 .
docker push 172.**.44.***:32000/myapp:v1
Get http://172.**.44.***:32000/v2/: dial tcp 172.**.44.***:32000: connect: no route to host

I didn't use microk8s to set up kubernetes cluster before, but I do have the feeling is, the ip of 172.xx.xx.xx is the wrong IP that you can't connect it from your local pc.

so what's the output of below commands:

  1. What's the nodes IP, include master and work nodes?
kubernetes get nodes -o wide
  1. What's the service setup?
kuberentes get services

can you make sure the service to that's private registry server's port is setup and can be connected.

  1. check your own PC's IP
# for windows
ipconfig

# for linux/macos
ifconfig

maybe there are many IPs in output, make sure you get the proper local IP for your PC.

For example, it is something like 10.xx.xx.xx, then you should use similar IPs to connect to that private registry server, you just need find it out

  1. check what network CNI you are using, weavenet, flannel, etc.

the IP of 172.xx.xx.xx are mostly used by these network CNI provider, it can be used in kubernetes cluster, but not your local host.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
piby180
  • 388
  • 1
  • 6
  • 18
  • Is the docker "remote pc" not the machine running the VMs? You will probably need to use a bridged network adapter on the VM to make it accessible via the network, or map ports via your hypervisor. `multipass` itself doesn't really provide that level of control over network setup. – Matt Jan 19 '20 at 05:11
  • please add the output of this two commands in order to troubleshoot your IP problem: `kubectl get services` and `kubectl get nodes -o wide` . – Will R.O.F. Jan 20 '20 at 16:15

1 Answers1

3

After you enable registry on the microk8s, run this script

kubectl get svc -n container-registry

you can see that microk8s has redirect registry service's port 32000 to 5000, then I use ingress to expose via https.

First, you have to enable ingress on microk8s:

microk8s.enable ingress

then, you have to create a tls sceret if you want to use https :

openssl genrsa -aes128 -out server.key 2048

openssl req -newkey rsa:2048 -nodes -keyout server.key -x509 -days 3650 -out server.crt

kubectl create secret tls registry-secret-tls --cert=server.crt --key=server.key -n container-registry

then use kubectl apply -f to create an ingress for revese proxy of registry service.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: registry
  namespace: container-registry
  annotations:
    nginx.ingress.kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "500m"
    nginx.ingress.kubernetes.io/proxy-pass-headers: "Location"
spec:
  tls:
  - hosts:
    - ingress.local
    secretName: registry-secret-tls
  rules:
  - host: ingress.local
    http:
      paths:
      - path: /
        backend:
          serviceName: registry
          servicePort: 5000

then, add 127.0.0.1 ingress.local to /etc/hosts file. At last, use buildah push docker images to ingress.local.

buildah push --tls-verify=false 44c92e82c220 docker://ingress.local/datacenter-school

This time, it looks everything is ok. But when I try list images in microk8s, I can't find the image that I just pushed.

microk8s.ctr images ls -q |grep datacenter-school

That's quiet weird!

zaoying
  • 31
  • 2