Kubernetes Ingress (Specific APP) 504 Gateway Time-Out with 60 seconds

Question

We have an environment with k8s + Rancher 2 (3 nodes) and an external nginx that only forwards connections to the k8s cluster according to this documentation: https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/

In a specific application running in this environment, when we perform a POST (since this POST takes around 3 to 4 minutes to complete), it is interrupted with the message "504 Gateway Time-Out" after 60 seconds. I've tried to apply specific notes to change the timeout as below, but to no avail:

Ingress of application:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: api-loteamento-spring-hml
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/server-snippet: "keepalive_timeout 3600s;client_body_timeout 3600s;client_header_timeout 3600s;"
  labels:
    run: api-loteamento-spring-hml
spec:
  rules:
  - host: hml-api-loteamento-sp.gruposfa.bla.bla
    http:
      paths:
      - backend:
          serviceName: api-loteamento-spring-hml
          servicePort: 80

I have also tried to create a Global ConfigMap with the parameters as below, also without success:

[rancher@srv-rcnode01 ssl]$ kubectl get pods -n ingress-nginx
NAME                                    READY   STATUS    RESTARTS   AGE
default-http-backend-67cf578fc4-lcz82   1/1     Running   1          38d
nginx-ingress-controller-7jcng          1/1     Running   11         225d
nginx-ingress-controller-8zxbf          1/1     Running   8          225d
nginx-ingress-controller-l527g          1/1     Running   8          225d

[rancher@srv-rcnode01 ssl]$ kubectl get pod nginx-ingress-controller-8zxbf -n ingress-nginx -o yaml |grep configmap
    - --configmap=$(POD_NAMESPACE)/nginx-configuration
    - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
    - --udp-services-configmap=$(POD_NAMESPACE)/udp-services

[rancher@srv-rcnode01 ~]$ cat global-configmap.yaml
apiVersion: v1
data:
  client-body-timeout: "360"
  client-header-timeout: "360"
  proxy-connect-timeout: "360"
  proxy-read-timeout: "360"
  proxy-send-timeout: "360"
kind: ConfigMap
metadata:
  name: nginx-configuration

And apply:

kubectl apply -f global-configmap.yaml

Accessing the ingress pods and checking the nginx.conf, I see that annotations are created according to the parameters set inside the application block:

[rancher@srv-rcnode01 ~]$ kubectl -n ingress-nginx exec --stdin --tty nginx-ingress-controller-8zxbf -- /bin/bash

And view nginx.conf

keepalive_timeout 3600s;client_body_timeout 3600s;client_header_timeout 3600s;

# Custom headers to proxied server
            
            
            proxy_connect_timeout                   3600s;
            proxy_send_timeout                      3600s;
            proxy_read_timeout                      3600s;

What I noticed at the beginning of the nginx.conf file in the "server" configuration block is that it has default 60-second timeout values:

# Custom headers to proxied server
            
            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

My question in this case is whether these values may be interfering with this problem, and how can I change these values in the k8s?

Has anyone gone through this situation or something and can give me a north?

Thanks!!

Its your local env or cloud? How did you deploy Ingress? Did you enforce ingress to use new configuration? Did you try to add specific path in your ingress `spec.rules.http.paths.path` according to minial requirements from [docs](https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource) — PjoterS, Sep 18 '20 at 09:55
Hi. The environment is local. Yes, I applied the changes on the ingress, but no success. — Angelo Braga, Sep 18 '20 at 11:37

score 10 · Answer 1 · answered Sep 29 '20 at 12:29

What you want to achieve was mentioned in Nginx Documentation in Custom Configuration. It recommend to use ConfigMap.

$ cat configmap.yaml
apiVersion: v1
data:
  proxy-connect-timeout: "10"
  proxy-read-timeout: "120"
  proxy-send-timeout: "120"
kind: ConfigMap
metadata:
  name: ingress-nginx-controller

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/custom-configuration/configmap.yaml \
    | kubectl apply -f -

If the Configmap it is updated, NGINX will be reloaded with the new configuration.

After that, in Ingress controller pod, you should see entries like:

8 controller.go:137] Configuration changes detected, backend reload required.
8 controller.go:153] Backend successfully reloaded.

However, please keep in mind that higher timeout values are not recommended for Nginx. This information can be found in Nginx Ingress - proxy-connect-timeout:

Sets the timeout for establishing a connection with a proxied server. It should be noted that this timeout cannot usually exceed 75 seconds.

Additional information:

1. Sometimes, when Nginx Ingress cannot load new configuration, you can find log like below:

controller.go:149"] - Unexpected failure reloading the backend": Invalid PID number "" in "/tmp/nginx/pid"

To fix it, you just need to restart Ingress pod.

2. If you dont like default settings, you can create a config template in Go language.

score 1 · Answer 2 · answered Jun 23 '22 at 19:38

1

To update only the ingress, and not the entire controller, use the following.
Source: Advanced Configuration with Annotations

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-loteamento-spring-hml
  annotations:
    nginx.org/proxy-connect-timeout: 3600s
    nginx.org/proxy-read-timeout: 3600s
    nginx.org/proxy-send-timeout: 3600s
  labels:
    run: api-loteamento-spring-hml
spec:
  ingressClassName: yourIngressClass
  rules:
  - host: hml-api-loteamento-sp.gruposfa.bla.bla
    http:
      paths:
        - path: /
          pathType: ImplementationSpecific
          backend:
            service:
              name: api-loteamento-spring-hml
              port:
                number: 80

answered Jun 23 '22 at 19:38

John C

1,761
2
20
30

1

It don't work for me. I guess you use the nginx-ingress controller, but k8s default's ingress is ingress-nginx, it is different. – miemengniao Dec 19 '22 at 08:40
@miemengniao correct, that is why I pointed to the documentation for NGINX ingress. You can still do the same thing with the default k8s ingress, you just have to find out what the [correct annotations](https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#custom-timeouts) are for theirs. – John C Mar 16 '23 at 07:18

score -1 · Answer 3 · answered Nov 30 '22 at 12:50

I was facing the same problem. proxy timeout values were not set as per defined values. So after changing metadata.annotations my issue resolved.

Previous Values -

nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

Working Values -

nginx.org/proxy-connect-timeout: 3600s
nginx.org/proxy-read-timeout: 3600s
nginx.org/proxy-send-timeout: 3600s

Kubernetes Ingress (Specific APP) 504 Gateway Time-Out with 60 seconds

3 Answers3

Linked