i have relatively specific problem that doesn't seem to have a specific solution! The situation looks like this: A Spring-Boot Backend is running in a container, in a pod, in a daemonset, in Kubernetes. In the same pod there is a second container which has an nginx web-server, serving an Angular (Ionic) web-app. There is also an nginx ingress controller running in the cluster for load balancing, etc.
These two can send and receive from eachother for the most part without any issues. There are, however, certain operations that require an EventSource - SSEEmitter connnection to be build up between client and backend. This is where the problems begin.
If there is no message sent from the SSEEmitter for around 50s, the connection times out with a 504 Gateway Time-Out message, and any future info being pushed from the server is not received.
All research points to a timeout issue in nginx. I have tried, as most places suggest, to increase the proxy_read_timeout value in both the ingress-service, and the nginx server. I have noticed intermittent changes, but nothing that corresponds to the values i have entered. I have also tried to send more explicit headers with the SSEEmitter to have it recognized as an eventstream, but also with no success.
I would really like to avoid sending a "heartbeat" from the backend to keep the connection alive, but can't seem to find a stable solution for the problem.
To further complicate matters, if I test everything in a local Minikube environment, setting the proxy_read_timeout sometimes works! In the live Cluster there are never any changes.
Has anyone else dealt with this and has a stable solution? Any help would be greatly appreciatied!! Here are the relevant nginx configs..
INGRESS:
apiVersion: v1
kind: Service
metadata:
name: my-ingress
namespace: test-ns
spec:
ports:
- name: other-thing
port: 50080
protocol: TCP
targetPort: 50080
- name: ionic
port: 80
protocol: TCP
targetPort: 80
- name: spring
port: 8080
protocol: TCP
targetPort: 8080
selector:
k8s-app: amb-spring
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/cors-allow-headers: Content-Type
nginx.ingress.kubernetes.io/cors-allow-methods: POST, GET, OPTIONS
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/proxy-read-timeout: "5s"
# ----> I also tried setting a really low value to force a timeout...
nginx.ingress.kubernetes.io/eventsource: "true"
name: ingress
namespace: test-afm
spec:
tls:
- hosts:
- iot-dev.local
secretName: some-cert
rules:
- host: iot-dev.local
http:
paths:
- backend:
service:
name: service-dev
port:
number: 80
path: /
pathType: Prefix
- backend:
service:
name: service-dev
port:
number: 8080
path: /spring
pathType: Prefix
NGINX.CONF
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
sendfile on;
tcp_nopush on;
keepalive_timeout 620s;
## Logging ##
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
# error_log syslog:"local7",tag=nginx_error;
# error_log syslog:server=localhost;
# access_log syslog:server=localhost combined;
## MIME Types ##
include /etc/nginx/mime.types;
default_type application/octet-stream;
## Caching ##
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=one:8m max_size=3000m inactive=600m;
proxy_temp_path /var/tmp;
## Security Headers ##
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
add_header X-Forwarded-Proto "$http_x_forwarded_proto" always;
server {
listen 80;
listen 443;
root /usr/src/app/www;
index index.html index.htm;
location / {
expires -1;
add_header Pragma "no-cache";
add_header Cache-Control "no-store, no-cache, must-revalidate, post-check=0, pre-check=0";
add_header Access-Control-Allow-Origin * always;
add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range,Authorization,Accept,Content-Type,Access-Control-Allow-Credentials,Access-Control-Allow-Origin' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Origin, Access-Control-Allow-Origin, Content-Type,Accept, Authorization, Origin, Accept, X-Requested-With,Access-Control-Request-Method, Access-Control-Request-Headers' always;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
try_files $uri $uri/ /index.html;
}
}
}