Istio ingress gateway randomly returns 507 response

Question

Ingress gateway is retrying if the upstream returns 502. Most of the time it is working as expected. Sometimes gateway returns 507 "exceeded request buffer limit while retrying upstream" without retrying.

From the logs I can see this: app 502 -> istio-proxy sidecar 502 -> ingress gateway 507 -> client. Unable to find other errors in logs related to ingress-gateway.

Requests are 1-30Mb in size. Any ideas where to look for the issue?

VirtualService:

...
      retries:
        attempts: 4
        retryOn: 502,retriable-status-codes,connect-failure
        retryRemoteLocalities: true
...

EnvoyFilter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: buffer-limit
spec:
  configPatches:
    - applyTo: LISTENER
      listenerMatch: 0.0.0.0_8080
      patch:
        operation: MERGE
        value:
          per_connection_buffer_limit_bytes: 100000000

is that `ListenerMatch` correctly used? https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-ListenerMatch. Also, it should have been `listenerMatch`, wich small `l`. — suren, Mar 08 '21 at 13:07
It didn't help. I started to doubt if my `EnvoyFilter` is correct. Is it possible to see the Envoy configuration with the applied Istio transformation? — Jonas, Mar 08 '21 at 14:31
yes, you can access the sidecar and check `/etc/istio/proxy/envoy-rev0.json`. also, istioctl has a proxy-check sub-command to check it. — suren, Mar 08 '21 at 14:58
I am unable to find any changes to istio-ingressgateway `/etc/istio/proxy/envoy-rev0.json`. As I understand `EnvoyFilter` should alter these configurations? — Jonas, Mar 09 '21 at 12:51
Ideally it should, but with `EnvoyFilters` sometimes things work, sometimes not. I should open an issue with istio team on github. Sometimes EF examples in their docs do not work as expected. — suren, Mar 09 '21 at 12:58

score 0 · Accepted Answer · answered Apr 01 '21 at 07:51

Solution was to use correct Envoy filter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  namespace: istio-system
  name: buffer-limit
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: LISTENER
      patch:
        operation: MERGE
        value:
          per_connection_buffer_limit_bytes: 100000000

The filter is working on v1.8.4

To check if the filter is applied:

istioctl proxy-config listeners <istio-ingressgateway-pod> -o json -n istio-system

Istio ingress gateway randomly returns 507 response

1 Answers1