4
  • Istio: 1.3 (also tried 1.1 before update to 1.3)
  • K8s: 1.16.2
  • Cloud provider: DigitalOcean

I have a cluster setup with Istio. I have enabled grafana/kiali and also installed kibana and RabbitMQ management UI and for all of those I have gateways and virtual services configured (all in istio-system namespace) along with HTTPS using SDS and cert-manager and all works fine. It means I can access these resources in the browser over HTTPS with a sub domain.

Then I deployed a microservice (part of a real application) and created Service, VirtualService and Gateway resources for it (for now it is the only one service and gateway except rabbitmq which uses different sub domain and differend port). And it is located in default namespace.

$ kubectl get gateway

NAME             AGE
gateway-rabbit   131m
tg-gateway       45m

$ kubectl get po

NAME                           READY     STATUS    RESTARTS   AGE
rabbit-rabbitmq-0              2/2       Running   2          134m
tg-app-auth-79c578b94f-mqsz9   2/2       Running   0          46m

If I try to connect to my service with port forwarding I can get a success response from localhost:8000/api/me (also healthz, readyz both return 200 and pod has 0 restarts) so it is working fine.

kubectl port-forward $(kubectl get pod --selector="app=tg-app-auth" --output jsonpath='{.items[0].metadata.name}') 8000:8000

But I can't access it neither via HTTP nor HTTPS. I get 404 using HTTP and the following response using HTTPS:

*   Trying MYIP...
* TCP_NODELAY set
* Connected to example.com (MYIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443 
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443

Here are my yaml files:

Gateway:

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:
  name: tg-gateway
  namespace: default

spec:
  selector:
    istio: ingressgateway

  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - www.example.com
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - www.example.com
    tls:
      mode: SIMPLE
      serverCertificate: sds
      privateKey: sds
      credentialName: tg-certificate

Service:

apiVersion: v1

kind: Service

metadata:
  name: tg-app-auth
  namespace: default
  labels:
    app: tg-app-auth

spec:
  selector:
    app: tg-app-auth

  ports:
  - name: http
    port: 8000

VirtualService

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:
  name: tg-app-auth-vs
  namespace: default

spec:
  hosts:
  - www.example.com

  gateways:
  - tg-gateway

  http:
  - match:
    - port: 443
    - uri:
        prefix: /api/auth
    rewrite:
      uri: /api
    route:
    - destination:
        host: tg-app-auth
        port:
          number: 8000

---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: tg-app-auth-dr
  namespace: default
spec:
  host: tg-app-auth
  trafficPolicy:
    tls:
      mode: DISABLE

I tried to remove all the HTTPS and TLS details and configure it with HTTP only but still can not get any response. I read all the issues on github but nothing helps and it seems like I have a very silly mistake. All these configurations are pretty much the same as I have for grafana/kibana/kiali/rabbit and all of them works fine.

UPD: Tried to get response with and it also works fine but I can't get response from LB IP or domain

kubectl exec $(kubectl get pod --selector app=tg-app-auth --output jsonpath='{.items[0].metadata.name}') -c istio-proxy -- curl -v http://$(kubectl get endpoints tg-app-auth -o jsonpath='{.subsets[0].addresses[0].ip}'):8000/api/me

$ kubectl get endpoints tg-app-auth

NAME          ENDPOINTS          AGE
tg-app-auth   10.244.0.37:8000   22h

UPD

All statuses are OK. There are a lot more with different ports but I copied 80/443 only.

$ istioctl authn tls-check <pod_name>

cert-manager-webhook.istio-system.svc.cluster.local:443 
istio-galley.istio-system.svc.cluster.local:443
istio-ingressgateway.istio-system.svc.cluster.local:80
istio-ingressgateway.istio-system.svc.cluster.local:443
istio-sidecar-injector.istio-system.svc.cluster.local:443
kubernetes.default.svc.cluster.local:443

$ kubectl get ingress --all-namespaces
No resources found.

$ kubectl get gateways --all-namespaces

default        gateway-rabbit                    3d2h
default        tg-gateway                        17h
istio-system   gateway-grafana                   3d2h
istio-system   gateway-kiali                     3d2h
istio-system   istio-autogenerated-k8s-ingress   3d2h
logging        gateway-kibana                    3d2h
Max
  • 404
  • 2
  • 17
  • 39
  • Can You try to make gateway,vs,sv and destination rule in istio-namespace like with kibana,rabbitmq? Would like to know if that works then or we have to look somewhere else,for me yamls look ok,i dont see any errors here. – Jakub Nov 14 '19 at 19:41
  • Thank you for the response! The situation is next: if we move everything as it is (changing namespace only) the result is the same, if we change HTTPS port from 443 to 31400 (non-standard that is presented in istio gateway/values.yml configuration) it starts working! On HTTP I always get 404 (redirect to HTTPS not working and changing port from 80 to 31400 also not working). I moved everything back from istio-system to default but keep 31400 port instead of 443 and it also behaves the same way as for istio-system. Too weird. What's next should we try? – Max Nov 14 '19 at 20:28
  • Can You tell me why You use `tls: mode: SIMPLE serverCertificate: sds privateKey: sds credentialName: tg-certificate` and then `trafficPolicy: tls: mode: DISABLE`,its like you create tls and then disable it in destination rule, maybe that's the issue? Can you check if istio ingress-gateway is listening to port 443? – Jakub Nov 15 '19 at 06:33
  • Oh, it was one of my experiments trying to make it work. Anyway we have the same behaviour with or without this destination rule (as well as enabled/disabled trafficPolicy). Yes, istio-ingressgateway is listening on 443 (80:31380/TCP,443:31390/TCP,31400:31400/TCP etc.) – Max Nov 15 '19 at 07:30
  • 1.You use nodeport or loadbalancer? 2.it's kubeadm right? 3. Did you export the host and port like [there](https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/#determining-the-ingress-ip-and-ports), when u echo host and port everything is ok? 4.Did You enable sidecar injection on default namespace? 5. Did You create secret in default namespace? `The secret must be named istio-ingressgateway-certs in the istio-system namespace to align with the configuration of the Istio default ingress gateway used in this task.` – Jakub Nov 15 '19 at 08:34
  • 1. LoadBalancer, 2. Probably not, I setup cluster on DigitalOcean with thier Kubernetes feature (not manually on the server). 3. Yes, I see the same LB IP for host/secure host and 80 for the port, 4. Yes, I did and it works ok. 5. That is for the file mount approach as I see but I use cert-manager and sds and all their secrets are located in istio-system namespace (and once again, they works fine for all dev stack as well as for default ns but with port other then 443 (and for some reason redirect from http is not working)) – Max Nov 15 '19 at 08:46
  • And yes, I have the secret itself in istio-system ns named "tg-certificate" and I use it in gateway as "credentialName: tg-certificate". And if I comment this line out it stops working even with 31400 port that means secret and certificate seems to be ok – Max Nov 15 '19 at 08:51
  • Could You check if You dont have doubled ports somewhere? `kubectl get gateways -A`, `kubectl get ingress -A`, `istioctl authn tls-check `. – Jakub Nov 15 '19 at 10:47
  • For some reason chat is not working here :( I've updated my question with the results of the commands above. – Max Nov 15 '19 at 13:57
  • Did You see that [github issue](https://github.com/libressl-portable/portable/issues/369)? There are some ways people fix it that libressl error You have, for example `I update with brew, libressl, openssl and curl and reboot my machine. It's now working.` – Jakub Nov 18 '19 at 08:18
  • @jt97 Don't think this is the case actually. I suppose you were right about some sort of conflict since I get 404 with plain HTTP for port 80. It seems like standard ports 80/443 forward to somewhere else. But I don't know how to debug it :( – Max Nov 18 '19 at 11:16
  • Check if something is working on this ports, for example [netstat](https://en.wikipedia.org/wiki/Netstat) or [tcpdump](https://danielmiessler.com/study/tcpdump/),by the way, there is new feature in istio 1.4, called [istioctl](https://istio.io/docs/setup/getting-started/) analyze, worth to try install to install it and check your kubernetes cluster with `istioctl x analyze -k`, maybe it will find something. – Jakub Nov 18 '19 at 12:11

1 Answers1

1

Issue was really simple and silly. I had enabled global.k8sIngress.enabled = true in Istio values.yml. After changing it to false all starts working.

Max
  • 404
  • 2
  • 17
  • 39
  • I have a similar problem - http/80 is working ok, but https/443 is not - do you know why changing this to false worked? What does it do? – Paul J Mar 11 '20 at 17:35