Our AKS cluster was configured to auto-renew Let's Encrypt certificates through Ingress Cert-Manager annotation and this worked perfectly until we upgraded to AKS 1.20.7. This then stopped working and the certificates started to expire without them being renewed - I double-checked all changes to K8S and CertManager APIs and reviewed all YAMLs, but I'm not seeing anything obviously wrong. Would appreciate any pointers.
My understanding is that as long as I add the "cert-manager.io/cluster-issuer: letsencrypt-prod-p9v2" to my ingress - the whole renewal should happen automatically - this is not happening though.
> kubectl cert-manager version
util.Version{GitVersion:"v1.4.0", GitCommit:"5e2a6883c1202739902ac94b5f4884152b810925", GitTreeState:"clean", GoVersion:"go1.16.2", Compiler:"gc", Platform:"linux/amd64"}
AKS version: 1.20.7
cat shipit-ingress-p9v2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod-p9v2
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-body-size: 15m
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.org/client-max-body-size: 15m
generation: 4
name: shipit-ingress-p9v2
namespace: supplier
resourceVersion: "147087245"
uid: 6751dbff-83b1-48a1-a467-e75cc843ee79
spec:
rules:
- host: xxx.westeurope.cloudapp.azure.com
http:
paths:
- backend:
service:
name: planet9v2
port:
number: 8080
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- xxx.westeurope.cloudapp.azure.com
secretName: tls-secret-p9v2
status:
loadBalancer:
ingress:
- ip: 10.240.0.5
>>kubectl get clusterissuer -o yaml letsencrypt-prod-p9v2
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
annotations:
creationTimestamp: "2020-05-29T13:31:10Z"
generation: 2
name: letsencrypt-prod-p9v2
resourceVersion: "25493731"
uid: 0e0e46f5-4cdf-42ea-a022-2dfe9ed56ad8
spec:
acme:
email: xxx
http01: {}
privateKeySecretRef:
name: letsencrypt-prod
server: https://acme-v02.api.letsencrypt.org/directory
status:
acme:
uri: https://acme-v02.api.letsencrypt.org/acme/acct/76984529
conditions:
- lastTransitionTime: "2020-05-29T13:31:11Z"
message: The ACME account was registered with the ACME server
reason: ACMEAccountRegistered
status: "True"
type: Ready
>>kubectl cert-manager inspect secret tls-secret-p9v2
...
Debugging:
Trusted by this computer: no: x509: certificate has expired or is not yet valid: current time 2021-08-24T07:03:32Z is after 2021-08-22T06:40:20Z
CRL Status: No CRL endpoints set
OCSP Status: Cannot check OCSP: error reading OCSP response: ocsp: error from server: unauthorized
kubectl describe secret tls-secret-p9v2
Name: tls-secret-p9v2
Namespace: supplier
Labels: certmanager.k8s.io/certificate-name=tls-secret-p9v2
Annotations: certmanager.k8s.io/alt-names: shipit-dev-p9v2.westeurope.cloudapp.azure.com
certmanager.k8s.io/common-name: shipit-dev-p9v2.westeurope.cloudapp.azure.com
certmanager.k8s.io/ip-sans:
certmanager.k8s.io/issuer-kind: ClusterIssuer
certmanager.k8s.io/issuer-name: letsencrypt-prod-p9v2
Type: kubernetes.io/tls
Data
====
tls.key: 1679 bytes
ca.crt: 0 bytes
tls.crt: 5672 bytes
kubectl get order
NAME STATE AGE
tls-secret-p9v2-4123722043 valid 24d
[(⎈ |shipit-k8s-dev:supplier)]$ k describe order tls-secret-p9v2-4123722043
Name: tls-secret-p9v2-4123722043
Namespace: supplier
Labels: acme.cert-manager.io/certificate-name=tls-secret-p9v2
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Order
Metadata:
Creation Timestamp: 2021-07-31T04:12:42Z
Generation: 4
Managed Fields:
API Version: certmanager.k8s.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:acme.cert-manager.io/certificate-name:
f:ownerReferences:
.:
k:{"uid":"a1dec741-0fe7-42be-99d2-176c3d4cdf38"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:config:
f:csr:
f:dnsNames:
f:issuerRef:
.:
f:kind:
f:name:
f:status:
.:
f:certificate:
f:challenges:
f:finalizeURL:
f:state:
f:url:
Manager: jetstack-cert-manager
Operation: Update
Time: 2021-07-31T04:13:09Z
Owner References:
API Version: certmanager.k8s.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: tls-secret-p9v2
UID: a1dec741-0fe7-42be-99d2-176c3d4cdf38
Resource Version: 143545958
UID: a646985b-6d44-4c99-bb39-ceb6c4919047
Spec:
Config:
Domains:
shipit-dev-p9v2.westeurope.cloudapp.azure.com
http01:
Ingress Class: nginx
Csr: MIIC3zCCAccCAQAwTzEVMBMGA1UEChMMY2VydC1tYW5hZ2VyMTYwNAYDVQQDEy1zaGlwaXQtZGV2LXA5djIud2VzdGV1cm9wZS5jbG91ZGFwcC5henVyZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCdqF08foRx7qVCU4YpVLHxodqMJp1h10l0s89MUVK7C2IWwHdQ5w2BjUB12gT6T6NK9ZhJEzzYtLk18wFAojKUOFjuwF5Kklh+Qe6rFiZNNJ2+uDN/WhCLylbsXjHzQ+N3XMZ0jhGv+72XQyeK/X8jurMmVk5dSZbYP0ysk7w7gSFjjpeN2EIpYcnp2rCjTU+ksfeJ04DDm84hN9snMpGKspIhTFBphCQQgScPO9Fx+S5NVG/ScoM0CLSYiQVB0oPYUaw84O/lNC7kq/UWERli2pNy9Gnxdw2g37nFTj2uvPGGbPE1WBTFtdzkFWMepaw1l25X1//Nsap3zuZY0C3jAgMBAAGgSzBJBgkqhkiG9w0BCQ4xPDA6MDgGA1UdEQQxMC+CLXNoaXBpdC1kZXYtcDl2Mi53ZXN0ZXVyb3BlLmNsb3VkYXBwLmF6dXJlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAWEqfGuYcgf2ujby+K9MK+9q/r0cajo4q0JM6ZkBQQGb88b/nwxa7sr4n7hnlpKhXdLPp5QoeBMr3UM7Nwc7PrYxOws9v51mq3ekUOARgO4/4eJw4agFf8KKQLjtkFr2Q3OFJp6GuYKDCo2+Z1jqs76v7ReKlBoVhMtxOkjykQJheFQzg7ezGshE5trXh3NL/FaaThp1vP+qp8nDnq1YXkvOyaoc7u4X2sl831FTPcv3tsQJJzrOPlZPUJcgCC9cZiCTwqdttaJFRobTEGSk+pzc54C6eRQv9muto8D29Eg2G9f9xDSJULT6WbZWL6gzbJ/5pu3ep+V+cB43f5H+Sqg==
Dns Names:
shipit-dev-p9v2.westeurope.cloudapp.azure.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod-p9v2
Status:
Certificate: LS0tLS1CRUdJTiBDRVJUSUZJ.....
Challenges:
Authz URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/17660284180
Config:
http01:
Ingress Class: nginx
Dns Name: shipit-dev-p9v2.westeurope.cloudapp.azure.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod-p9v2
Key: AxP1pv5I087QVyKXIkGyT5pqlD4Aa-UYmJHAOgzHPu4.mIcOL5pBlkZJSpSUslpjJTC_hFunxNRCEA82VcfFAHE
Token: AxP1pv5I087QVyKXIkGyT5pqlD4Aa-UYmJHAOgzHPu4
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/17660284180/Sh057Q
Wildcard: false
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/75003870/13444902230
State: valid
URL: https://acme-v02.api.letsencrypt.org/acme/order/75003870/13444902230
Events: <none>
