0

i have relatively specific problem that doesn't seem to have a specific solution! The situation looks like this: A Spring-Boot Backend is running in a container, in a pod, in a daemonset, in Kubernetes. In the same pod there is a second container which has an nginx web-server, serving an Angular (Ionic) web-app. There is also an nginx ingress controller running in the cluster for load balancing, etc.

These two can send and receive from eachother for the most part without any issues. There are, however, certain operations that require an EventSource - SSEEmitter connnection to be build up between client and backend. This is where the problems begin.

If there is no message sent from the SSEEmitter for around 50s, the connection times out with a 504 Gateway Time-Out message, and any future info being pushed from the server is not received.

All research points to a timeout issue in nginx. I have tried, as most places suggest, to increase the proxy_read_timeout value in both the ingress-service, and the nginx server. I have noticed intermittent changes, but nothing that corresponds to the values i have entered. I have also tried to send more explicit headers with the SSEEmitter to have it recognized as an eventstream, but also with no success.

I would really like to avoid sending a "heartbeat" from the backend to keep the connection alive, but can't seem to find a stable solution for the problem.

To further complicate matters, if I test everything in a local Minikube environment, setting the proxy_read_timeout sometimes works! In the live Cluster there are never any changes.

Has anyone else dealt with this and has a stable solution? Any help would be greatly appreciatied!! Here are the relevant nginx configs..

INGRESS:
apiVersion: v1
kind: Service
metadata:
  name: my-ingress
  namespace: test-ns
spec:
  ports:
    - name: other-thing
      port: 50080
      protocol: TCP
      targetPort: 50080
    - name: ionic
      port: 80
      protocol: TCP
      targetPort: 80
    - name: spring
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    k8s-app: amb-spring
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/cors-allow-headers: Content-Type
    nginx.ingress.kubernetes.io/cors-allow-methods: POST, GET, OPTIONS
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "5s" 
# ----> I also tried setting a really low value to force a timeout...
    nginx.ingress.kubernetes.io/eventsource: "true"
  name: ingress
  namespace: test-afm
spec:
  tls:
    - hosts:
        - iot-dev.local
      secretName: some-cert
  rules:
    - host: iot-dev.local
      http:
        paths:
          - backend:
              service:
                name: service-dev
                port:
                  number: 80
            path: /
            pathType: Prefix
          - backend:
              service:
                name: service-dev
                port:
                  number: 8080
            path: /spring
            pathType: Prefix

NGINX.CONF

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    sendfile        on;
    tcp_nopush      on;
    keepalive_timeout  620s;

    ## Logging ##
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
#    error_log syslog:"local7",tag=nginx_error;
#     error_log syslog:server=localhost;
#     access_log syslog:server=localhost combined;

    ## MIME Types ##
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    ## Caching ##
    proxy_cache_path        /var/cache/nginx levels=1:2 keys_zone=one:8m max_size=3000m inactive=600m;
    proxy_temp_path         /var/tmp;

    ## Security Headers ##
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header X-Forwarded-Proto "$http_x_forwarded_proto" always;

    server {
        listen 80;
        listen 443;

        root /usr/src/app/www;
        index index.html index.htm;

        location / {
            expires -1;
            add_header Pragma "no-cache";
            add_header Cache-Control "no-store, no-cache, must-revalidate, post-check=0, pre-check=0";
            add_header Access-Control-Allow-Origin * always;
            add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range,Authorization,Accept,Content-Type,Access-Control-Allow-Credentials,Access-Control-Allow-Origin' always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'Origin, Access-Control-Allow-Origin, Content-Type,Accept, Authorization, Origin, Accept, X-Requested-With,Access-Control-Request-Method, Access-Control-Request-Headers' always;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            try_files $uri $uri/ /index.html;
        }
    }
}

Rodgersmg
  • 1
  • 1
  • While Stack Overflow does permit certain questions about Kubernetes, we require that they (like all questions asked here) be specifically related to programming. This question does not appear to be specifically related to programming, which makes it off-topic here. You might be able to ask questions like this one on [sf] or [DevOps](https://devops.stackexchange.com/). – Turing85 May 19 '23 at 13:28
  • Hi @Turing85, thanks for your reply. I'll try my luck over on the other forums too. – Rodgersmg May 19 '23 at 13:33

0 Answers0