I have one api, which will run about 4 minutes. I have deployed it on kubernetes with ingress-nginx. All api work normally except the long-run api, it always return 504 Gateaway as below:
I have check in stackoverflow, and try some solution, none of it work for me.
any help is welcome.
Kubernetes Ingress (Specific APP) 504 Gateway Time-Out with 60 seconds
I have changed ingress config as below:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ .Values.releaseName }}-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/server-snippet: "keepalive_timeout 3600s; grpc_read_timeout 3600s; grpc_send_timeout 3600s;client_body_timeout 3600s;"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "3601"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3601"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3601"
# nginx.org/proxy-connect-timeout: 3600s
# nginx.org/proxy-read-timeout: 3600s
# nginx.org/proxy-send-timeout: 3600s
nginx.ingress.kubernetes.io/proxy-body-size: "100M"
nginx.ingress.kubernetes.io/proxy-next-upstream: "error non_idempotent http_502 http_503 http_504"
nginx.ingress.kubernetes.io/retry-non-idempotent: "true"
nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "1"
# nginx.ingress.kubernetes.io/configuration-snippet: |-
# location /HouseKeeping/Health/Healthz {
# deny all;
# return 403;
# }
# location /InternalApi/ {
# deny all;
# return 403;
# }
nginx.ingress.kubernetes.io/server-snippets: |
http {
client_max_body_size 100m;
}
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_cache_bypass $http_upgrade;
proxy-connect-timeout: 3602;
proxy-read-timeout: 3602;
proxy-send-timeout: 3602;
}
spec:
rules:
- host: {{ .Values.apiDomain }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ .Values.releaseName }}-web-clusterip-srv
port:
number: 80
I also change the configMap for ingress-nginx-controller to add below config:
apiVersion: v1
data:
allow-snippet-annotations: "true"
proxy-connect-timeout: "300"
proxy-read-timeout: "300"
proxy-send-timeout: "300"
kind: ConfigMap
I also used command to get ingress-nginx's conf, it seems okay
kubectl -n ingress-nginx exec ingress-nginx-controller-6cc65c646d-ljmrm cat /etc/nginx/nginx.conf | tee nginx.test-ingress-export.conf
# Custom headers to proxied server
proxy_connect_timeout 3601s;
proxy_send_timeout 3601s;
proxy_read_timeout 3601s;
it still timeout in 120 seconds.
