I have a k8s cluster with 6 nodes and ingress controllers spawned over all of them. But 3 of 6 pods are not working properly.
kubectl -n ingress-nginx get pods
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-mjxvv 0/1 Completed 0 147d
ingress-nginx-admission-patch-jz84x 0/1 Completed 0 147d
nginx-ingress-controller-4xrsq 1/1 Running 0 240d
nginx-ingress-controller-dkwst 0/1 CrashLoopBackOff 8 18m
nginx-ingress-controller-ft7jq 1/1 Running 0 240d
nginx-ingress-controller-hwrr9 0/1 CrashLoopBackOff 12 37m
nginx-ingress-controller-rk4b2 0/1 CrashLoopBackOff 8 18m
nginx-ingress-controller-tg6pm 1/1 Running 1 249d
Log output isn't helping because the pod not even start
kubectl -n ingress-nginx logs nginx-ingress-controller-dkwst
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: nginx-0.43.0-rancher1
Build: git-7eabf144d
Repository: https://github.com/rancher/ingress-nginx.git
nginx version: nginx/1.19.6
-------------------------------------------------------------------------------
unknown flag: --controller-class
Usage of :
unknown flag: --controller-class
--add_dir_header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files
--annotations-prefix string Prefix of the Ingress annotations specific to the NGINX controller. (default "nginx.ingress.kubernetes.io")
--apiserver-host string Address of the Kubernetes API server.
Takes the form "protocol://address:port". If not specified, it is assumed the
program runs inside a Kubernetes cluster and local discovery is attempted.
--certificate-authority string Path to a cert file for the certificate authority. This certificate is used
only when the flag --apiserver-host is specified.
--configmap string Name of the ConfigMap containing custom global configurations for the controller.
--default-backend-service string Service used to serve HTTP requests not matching any known server name (catch-all).
Takes the form "namespace/name". The controller configures NGINX to forward
requests to the first port of this Service.
--default-server-port int Port to use for exposing the default server (catch-all). (default 8181)
--default-ssl-certificate string Secret containing a SSL certificate to be used by the default HTTPS server (catch-all).
Takes the form "namespace/name".
--disable-catch-all Disable support for catch-all Ingresses
--election-id string Election id to use for Ingress status updates. (default "ingress-controller-leader")
--enable-metrics Enables the collection of NGINX metrics (default true)
--enable-ssl-chain-completion Autocomplete SSL certificate chains with missing intermediate CA certificates.
Certificates uploaded to Kubernetes must have the "Authority Information Access" X.509 v3
extension for this to succeed.
--enable-ssl-passthrough Enable SSL Passthrough.
--health-check-path string URL path of the health check endpoint.
Configured inside the NGINX status server. All requests received on the port
defined by the healthz-port parameter are forwarded internally to this path. (default "/healthz")
--health-check-timeout int Time limit, in seconds, for a probe to health-check-path to succeed. (default 10)
--healthz-port int Port to use for the healthz endpoint. (default 10254)
--http-port int Port to use for servicing HTTP traffic. (default 80)
--https-port int Port to use for servicing HTTPS traffic. (default 443)
--ingress-class string Name of the ingress class this controller satisfies.
The class of an Ingress object is set using the field IngressClassName in Kubernetes clusters version v1.18.0 or higher or the annotation "kubernetes.io/ingress.class" (deprecated).
If this parameter is not set, or set to the default value of "nginx", it will handle ingresses with either an empty or "nginx" class name.
--kubeconfig string Path to a kubeconfig file containing authorization and API server information.
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--maxmind-edition-ids string Maxmind edition ids to download GeoLite2 Databases. (default "GeoLite2-City,GeoLite2-ASN")
--maxmind-license-key string Maxmind license key to download GeoLite2 Databases.
https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases
--maxmind-mirror string Maxmind mirror url (example: http://geoip.local/databases
--metrics-per-host Export metrics per-host (default true)
--monitor-max-batch-size int Max batch size of NGINX metrics (default 10000)
--one_output If true, only write logs to their native severity level (vs also writing to each lower severity level
--profiler-port int Port to use for expose the ingress controller Go profiler when it is enabled. (default 10245)
--profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
--publish-service string Service fronting the Ingress controller.
Takes the form "namespace/name". When used together with update-status, the
controller mirrors the address of this service's endpoints to the load-balancer
status of all Ingress objects it satisfies.
--publish-status-address string Customized address to set as the load-balancer status of Ingress objects this controller satisfies.
Requires the update-status parameter.
--report-node-internal-ip-address Set the load-balancer status of Ingress objects to internal Node addresses instead of external.
Requires the update-status parameter.
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--ssl-passthrough-proxy-port int Port to use internally for SSL Passthrough. (default 442)
--status-port int Port to use for the lua HTTP endpoint configuration. (default 10246)
--status-update-interval int Time interval in seconds in which the status should check if an update is required. Default is 60 seconds (default 60)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--stream-port int Port to use for the lua TCP/UDP endpoint configuration. (default 10247)
--sync-period duration Period at which the controller forces the repopulation of its local object stores. Disabled by default.
--sync-rate-limit float32 Define the sync frequency upper limit (default 0.3)
--tcp-services-configmap string Name of the ConfigMap containing the definition of the TCP services to expose.
The key in the map indicates the external port to be used. The value is a
reference to a Service in the form "namespace/name:port", where "port" can
either be a port number or name. TCP ports 80 and 443 are reserved by the
controller for servicing HTTP traffic.
--udp-services-configmap string Name of the ConfigMap containing the definition of the UDP services to expose.
The key in the map indicates the external port to be used. The value is a
reference to a Service in the form "namespace/name:port", where "port" can
either be a port name or number.
--update-status Update the load-balancer status of Ingress objects this controller satisfies.
Requires setting the publish-service parameter to a valid Service reference. (default true)
--update-status-on-shutdown Update the load-balancer status of Ingress objects when the controller shuts down.
Requires the update-status parameter. (default true)
-v, --v Level number for the log level verbosity
--validating-webhook string The address to start an admission controller on to validate incoming ingresses.
Takes the form "<host>:port". If not provided, no admission controller is started.
--validating-webhook-certificate string The path of the validating webhook certificate PEM.
--validating-webhook-key string The path of the validating webhook key PEM.
--version Show release information about the NGINX Ingress controller and exit.
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
--watch-namespace string Namespace the controller watches for updates to Kubernetes objects.
This includes Ingresses, Services and all configuration resources. All
namespaces are watched if this parameter is left empty.
I've been searching through all day and most of the answers are related to exit code 255, in this case the exit code is 2
kubectl -n ingress-nginx describe pod nginx-ingress-controller-hwrr9
Name: nginx-ingress-controller-hwrr9
Namespace: ingress-nginx
Priority: 0
Node: leon/172.30.36.17
Start Time: Fri, 14 Oct 2022 14:05:58 -0300
Labels: app=ingress-nginx
app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
controller-revision-hash=6d48c5f9b4
pod-template-generation=2
Annotations: cni.projectcalico.org/podIP: 10.42.1.154/32
cni.projectcalico.org/podIPs: 10.42.1.154/32
Status: Running
IP: 10.42.1.154
IPs:
IP: 10.42.1.154
Controlled By: DaemonSet/nginx-ingress-controller
Containers:
controller:
Container ID: docker://b0e5c09c46c88af20d4cf88ef9d5061a3ded5121999e1a2614e034d125659fb2
Image: rancher/nginx-ingress-controller:nginx-0.43.0-rancher1
Image ID: docker-pullable://rancher/nginx-ingress-controller@sha256:677fb1a51ceb290f503fd44e6e27cf020813cf36d6822eea0ec8edff73002595
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader-nginx
--controller-class=k8s.io/ingress-nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Fri, 14 Oct 2022 14:27:32 -0300
Finished: Fri, 14 Oct 2022 14:27:32 -0300
Ready: False
Restart Count: 9
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: nginx-ingress-controller-hwrr9 (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-sfjvg (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
ingress-nginx-token-sfjvg:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-token-sfjvg
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: :NoExecute op=Exists
:NoSchedule op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 21m default-scheduler Successfully assigned ingress-nginx/nginx-ingress-controller-hwrr9 to leon
Warning FailedCreatePodSandBox 21m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6160500e9c31a4e4cad778e1bd65d0d4f7140a274ad7846c9e288db09bedde5a" network for pod "nginx-ingress-controller-hwrr9": networkPlugin cni failed to set up pod "nginx-ingress-controller-hwrr9_ingress-nginx" network: unable to create chain CNI-HOSTPORT-SETMARK: failed to list iptables chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status -1:
Warning FailedCreatePodSandBox 21m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c3ef2e5f7f3bd6b4396b3c4dbc07b7427ff6fe4c49454c828bb0f15e87850a14" network for pod "nginx-ingress-controller-hwrr9": networkPlugin cni failed to set up pod "nginx-ingress-controller-hwrr9_ingress-nginx" network: unable to create chain CNI-HOSTPORT-SETMARK: failed to list iptables chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status -1:
Normal SandboxChanged 21m (x3 over 21m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 21m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d446cfcbdc7aeaf57a582d85528b79254b2cd31fcb8c8ea4130c4ec190dc987a" network for pod "nginx-ingress-controller-hwrr9": networkPlugin cni failed to set up pod "nginx-ingress-controller-hwrr9_ingress-nginx" network: unable to create chain CNI-HOSTPORT-SETMARK: failed to list iptables chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status -1:
Normal Pulled 20m (x4 over 21m) kubelet Container image "rancher/nginx-ingress-controller:nginx-0.43.0-rancher1" already present on machine
Normal Created 20m (x4 over 21m) kubelet Created container controller
Normal Started 20m (x4 over 21m) kubelet Started container controller
Warning BackOff 4m57s (x76 over 21m) kubelet Back-off restarting failed container
Versions
k8s: 1.20.5
ingress: rancher/nginx-ingress-controller:nginx-0.43.0-rancher1
