an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/0/01/0000000010 while reading upstream

Question

I am using Google Kubernetes Engine, got this error from stackdriver logs

2020/05/14 07:31:19 [warn] 9#9: *658 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/0/01/0000000010 while reading upstream, client: 155.201.35.107, server: , request: "POST /graphql?key=AIzaSyCkP6djTqVJ5E9lUWcOlWYxpJw7zJuU9dQ HTTP/1.1", upstream: "http://127.0.0.1:4000/graphql?key=AIzaSyCkP6djTqVJ5E9lUWcOlWYxpJw7zJuU9dQ", host:...

I had read this post: an upstream response is buffered to a temporary file

But it's nginx related, not GKE. I know GKE maybe use Nginx as a proxy server.

I also found this doc: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-max-temp-file-size

But I don't know does GKE support it. I don’t know the cause of this problem and can’t reproduce it. How should I solve this issue? Thanks in advance.

UPDATE:

I am doing the API load testing using jmeter, here is the error from jmeter:

As you can see, the API should work fine and I have handled each exception inside isLogin API. So I don't think this error comes from my application. I guess it's related to infrastructure.

Answer 1

But I don't know does GKE support it?

The ingress-gce documentation does not mention this feature, but I do know you can use NGINX Ingress with GKE and use it's annotations to change the proxy-max-temp-file-size.

• First deploy Nginx Ingress for GCE-GKE basically running:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/cloud/deploy.yaml

• wait until Nginx Ingress Controller is assigned an external IP:

$ kubectl get svc ingress-nginx-controller -n ingress-nginx
NAME                       TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.21.3.21   XX.XX.XXX.XXX   80:31269/TCP,443:30522/TCP   84m

• Configure your Ingress Resource to use NGINX Ingress Controller by adding:
  • annotations: kubernetes.io/ingress.class: nginx, otherwise it will use the default gce ingress class.
• And add the proxy-max-temp-file-size set to zero to deny the creation of temporary files, here is a model for your Ingress Object:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <INGRESS_NAME>
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
spec:
  rules:
  - host: <FQDN>
    http:
      paths:
      -  backend:
           serviceName: <SERVICE_NAME>
           servicePort: <PORT>

The ingress.class: nginx is the key to assign the Ingress object to Nginx Ingress. Note that GCE and NGINX ingresses can coexist, you are allowed to have multiple ingress objects being handled by both ingresses.

Let me know in the comments if you have any question.