0

I had a kubernetes cluster running on a raspberry pi that had a simple server running on it. It's a small project i'm using to learn kubernetes.

After coming back to the project i realised it wasn't responding anymore. I Assumed this was because the https certificate needed renewing. But i cant figure out how to renew the certificate while pods aren't running.

I thought it would just be a case of reapplying the cluster-issuer but obviously that wouldn't work if some pods aren't running/being created.

trying to apply the cluser-issuer.yml gives me-

Error from server (InternalError): error when creating "cluster-issuer.yml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.100.140.135:443: connect: connection refused

i can also see this when getting cert-manager pods-

cert-manager   cert-manager-7fb78674d7-tnjf9              1/1     Running            4 (21h ago)       25h
cert-manager   cert-manager-cainjector-5dfc946d84-7wxrl   0/1     Error              271 (5m51s ago)   25h
cert-manager   cert-manager-webhook-8744b7588-zvmps       0/1     CrashLoopBackOff   270 (107s ago)    25h

output of kubectl logs cert-manager-cainjector-5dfc946d84-7wxrl -n cert-manager-

Error: error creating manager: Get "https://10.96.0.1:443/api?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

server-deployment.yml-

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: spoonobi/multi-server-arm
          ports:
            - containerPort: 8888

server-cluster-ip-service.yml-

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 8888
      targetPort: 8888

ingress-service.yml-

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: letsencrypt-prod # tell ingress to use https
    nginx.ingress.kubernetes.io/ssl-redirect: 'true' # redirect from http to https
spec:
  tls:
    - hosts:
        - ecmatrials.com
        - www.ecmatrials.com
      secretName: secret-ecmatrials-com
  rules:
    - host: ecmatrials.com
      http:
        paths:
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: server-cluster-ip-service
                port:
                  number: 8888
    - host: www.ecmatrials.com
      http:
        paths:
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: server-cluster-ip-service
                port:
                  number: 8888

cluster-issuer.yml-

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: redacted@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-redacted
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

service-nodeports.yml-

apiVersion: v1
kind: Service
metadata:
  name: server-nodeports
spec:
  type: NodePort
  selector:
    component: server
  ports:
    - name: http
      port: 80
      targetPort: 8888
      nodePort: 30602
    - name: https
      port: 443
      targetPort: 8888
      nodePort: 30824
  • Hi lloyd noone welcome to S.F. It's almost certainly due to a misconfigured CNI, as that 10.96.0.1 is a special `Service` IP that corresponds to the kubernetes API in every Namespace in the cluster. No one can _guess_ from your limited information what specifically is wrong with your cluster, but consider [editing your question](https://serverfault.com/posts/1119353/edit) to include the output of `kubectl get po -A` in order to see what the status is of the control plane and any CNI pods. Good luck! – mdaniel Jan 04 '23 at 04:19
  • hi @mdaniel, at some point when i was debugging and trying different things the coredns pod also stopped running. Found out this was due to old docker images still being referenced so deleted those. Then when i fixed that i managed to get the web hook and api-server etc running again after kubeadm reset and flushing iptables. So you were right about CNI i guess. Shame i had to go with the nuclear option. – lloyd noone Jan 06 '23 at 17:26

0 Answers0