How to configure Ingress request timeouts on GKE

Question

I currently have an Ingress configured on GKE (k8s 1.2) to forward requests towards my application's pods. I have a request which can take a long time (30 seconds) and timeout from my application (504). I observe that when doing so the response that i receive is not my own 504 but a 502 from what looks like the Google Loadbalancer after 60 seconds.

I have played around with different status codes and durations, exactly after 30 seconds i start receiving this weird behaviour regardless of statuscode emitted.

Anybody have a clue how i can fix this? Is there a way to reconfigure this behaviour?

score 31 · Answer 1 · edited Jun 23 '23 at 11:36

31

Beginning with 1.11.3-gke.18, it is possible to configure timeout settings in kubernetes directly.

First add a backendConfig:

# For GKE < 1.16.8-gke.3 use "cloud.google.com/v1beta1" below
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-bsc-backendconfig
spec:
  timeoutSec: 40

Then add an annotation in Service to use this backendConfig:

apiVersion: v1
kind: Service
metadata:
  name: my-bsc-service
  labels:
    purpose: bsc-config-demo
  annotations:
   # For GKE < 1.16.8-gke.3 use "beta.cloud.google.com/backend-config" below 
   cloud.google.com/backend-config: '{"default": "my-bsc-backendconfig"}'
spec:
  type: NodePort
  selector:
    purpose: bsc-config-demo
  ports:
    - port: 80
      protocol: TCP
      targetPort: 8080

And voila, your ingress load balancer now has a timeout of 40 seconds instead of the default 30 seconds.

See https://cloud.google.com/kubernetes-engine/docs/how-to/configure-backend-service#creating_a_backendconfig

edited Jun 23 '23 at 11:36

Greg Dubicki

5,983
3
55
68

answered Dec 21 '18 at 13:34

Zhe Li

1,103
1
14
20

I don't understand: where does the `backendConfig` go? I mean, is it in a separate file? – pietro909 Oct 20 '20 at 21:16
2

@pietro909 it can be in its own yaml file or in a yaml file together with other resources. See https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/ – Zhe Li Oct 22 '20 at 14:51
it's also possible to associate a backend-config with all ports on the service with an annotation like this: cloud.google.com/backend-config: '{"default": "my-backendconfig"}' (see https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#associating_backendconfig_with_your_ingress) – Toby 1 Kenobi Aug 05 '21 at 06:45

score 20 · Accepted Answer · edited Mar 28 '18 at 09:18

20

When creating an ingress on GKE the default setup is that a GLBC HTTP load balancer will be created with the backends that you supplied. Default it is configured at a 30 second timeout for your application to handle the request.

If you need a longer timeout you have to edit this manually after setup in the backends of your HTTP Load balancer in the google cloud console.

edited Mar 28 '18 at 09:18

Kat

1,604
17
24

answered Mar 24 '16 at 13:10

Mark van Straten

9,287
3
38
57

2

Yes the controller tries to pick sensible defaults. Timeout > 30s is not something that's been asked for, so I just dropped it from beta. If it's something you need often I can push it through as an annotation on the Ingress itself. – Prashanth B Mar 24 '16 at 18:10
11

@PrashanthB We do have a need to configure the timeout and would appreciate it if it were configurable via annotation – user1568967 Oct 19 '16 at 18:34

How to configure Ingress request timeouts on GKE

2 Answers2

Linked