Modified the helm chart of alerta to spin it up on an istio- enabled GKE cluster.
The alerta pod and its sidecar are created OK
▶ k get pods | grep alerta
alerta-758bc87dcf-tp5nv 2/2 Running 0 22m
When I try to access the url that my virtual service is pointing to
I get the following error
upstream connect error or disconnect/reset before headers. reset reason: connection termination
▶ k get vs alerta-virtual-service -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
annotations:
helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
creationTimestamp: "2020-04-23T14:45:04Z"
generation: 1
name: alerta-virtual-service
namespace: mynamespace
resourceVersion: "46844125"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/mynamespace/virtualservices/alerta-virtual-service
uid: 2a3caa13-3900-4da1-a3a1-9f07322b52b0
spec:
gateways:
- mynamespace/istio-ingress-gateway
hosts:
- alerta.myurl.com
http:
- appendHeaders:
x-request-start: t=%START_TIME(%s.%3f)%
match:
- uri:
prefix: /
route:
- destination:
host: alerta
port:
number: 80
timeout: 60s
and here is the service
▶ k get svc alerta -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
creationTimestamp: "2020-04-23T14:45:04Z"
labels:
app: alerta
chart: alerta-0.1.0
heritage: Tiller
release: alerta
name: alerta
namespace: mynamespace
resourceVersion: "46844120"
selfLink: /api/v1/namespaces/mynamespace/services/alerta
uid: 4d4a3c73-ee42-49e3-a4cb-8c51536a0508
spec:
clusterIP: 10.8.58.228
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: alerta
release: alerta
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
However, when I exec via another pod into the cluster and try to reach the alerta svc endpoint:
/ # curl -IL http://alerta
curl: (56) Recv failure: Connection reset by peer
/ # nc -zv -w 3 alerta 80
alerta (10.8.58.228:80) open
although as it is evident the port is open
any suggestion?
could it be that the chaining of the 2 proxies is creating issues? nginx behind envoy?
The container logs seem normal
2020-04-23 15:34:40,272 DEBG 'nginx' stdout output:
ip=\- [\23/Apr/2020:15:34:40 +0000] "\GET / HTTP/1.1" \200 \994 "\-" "\kube-probe/1.15+"
/web | /index.html | > GET / HTTP/1.1
edit: Here is a verbose curl with host header explicitly set
/ # curl -v -H "host: alerta.myurl.com" http://alerta:80
* Rebuilt URL to: http://alerta:80/
* Trying 10.8.58.228...
* TCP_NODELAY set
* Connected to alerta (10.8.58.228) port 80 (#0)
> GET / HTTP/1.1
> host: alerta.myurl.com
> User-Agent: curl/7.57.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
The nginx config file used by the app/pod is the following FWIW
worker_processes 4;
pid /tmp/nginx.pid;
daemon off;
error_log /dev/stderr info;
events {
worker_connections 1024;
}
http {
client_body_temp_path /tmp/client_body;
fastcgi_temp_path /tmp/fastcgi_temp;
proxy_temp_path /tmp/proxy_temp;
scgi_temp_path /tmp/scgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
include /etc/nginx/mime.types;
gzip on;
gzip_disable "msie6";
log_format main 'ip=\$http_x_real_ip [\$time_local] '
'"\$request" \$status \$body_bytes_sent "\$http_referer" '
'"\$http_user_agent"' ;
log_format scripts '$document_root | $uri | > $request';
default_type application/octet-stream;
server {
listen 8080 default_server;
access_log /dev/stdout main;
access_log /dev/stdout scripts;
location ~ /api {
include /etc/nginx/uwsgi_params;
uwsgi_pass unix:/tmp/uwsgi.sock;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
root /web;
index index.html;
location / {
try_files $uri $uri/ /index.html;
}
}
}
edit 2: Trying to get the istio authentication policy
✔ 18h55m ⍉
▶ kubectl get peerauthentication.security.istio.io
No resources found.
✔ 18h55m
▶ kubectl get peerauthentication.security.istio.io/default -o yaml
Error from server (NotFound): peerauthentications.security.istio.io "default" not found
edit 3: when performing curl to the service from within the istio proxy container
▶ k exec -it alerta-758bc87dcf-jzjgj -c istio-proxy bash
istio-proxy@alerta-758bc87dcf-jzjgj:/$ curl -v http://alerta:80
* Rebuilt URL to: http://alerta:80/
* Trying 10.8.58.228...
* Connected to alerta (10.8.58.228) port 80 (#0)
> GET / HTTP/1.1
> Host: alerta
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
