4

The backend uses Springboot to provide a WebSocket connection and sets the maximum idle time to 3 minutes. The program runs well in local. After 3 minutes of idle, the connection will be disconnected as scheduled. It can also be accessed normally by node port service when deployed in Kubernetes.

But when I inject sidecar into this backend pod, there is a problem. The connection doesn't work properly, often breaks, and is completely irregular. Sometimes when the frontend and backend are sending messages, it is interrupted suddenly. Sometimes it is interrupted after about 2 minutes idle. And sometimes the connection can only last for tens of seconds.

When the connection is interrupted, the backend will throw java.io.EOFException, and the frontend will receive the on close event.

This phenomenon will occur as long as a sidecar is injected into the pod(Even if I use node port service to access the pod). Also, I did a test, I used Nginx to transfer the request to port 31380 of istio-ingressgateway, and configured the gateway vs and dr as follows. But the result is the same.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: msapi
    version: product
  name: msapi
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      run: msapi
  template:
    metadata:
      labels:
        run: msapi
    spec:            
      containers:
      - env:
        - name: JAVA_OPTS
          valueFrom:
            configMapKeyRef:
              key: jvm.options
              name: test-config
        image: test/msapi:1.0.0
        imagePullPolicy: Always
        name: msapi
        ports:
        - containerPort: 9000
          protocol: TCP   
---
apiVersion: v1
kind: Service
metadata:
  name: msapi
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 9000
  selector:
    run: msapi
  type: ClusterIP
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-test
  namespace: test
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*.ingress.xxx.com'
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: msapi
  namespace: test
spec:
  gateways:
  - ingress-test
  hosts:
  - msapi.ingress.xxx.com
  http:
  - match:
    - headers:
        cookie:
          regex: ^(.*?; ?)?(version=pre)(;.*)?$
    route:
    - destination:
        host: msapi
        subset: pre
  - route:
    - destination:
        host: msapi
        subset: product
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: msapi
  namespace: test
spec:
  host: msapi
  subsets:
  - labels:
      version: product
    name: product
  - labels:
      version: pre
    name: pre
Deep Parsania
  • 253
  • 2
  • 12
Li Yongsheng
  • 43
  • 1
  • 5
  • What K8s and istio versions You have? Which platform/infrastructure do You have? You can install [istioctl 1.4.0](https://istio.io/docs/ops/diagnostic-tools/istioctl/) and try to run istioctl x analyze -k to see if it can find any problems within istio mesh. You could try use MTLS [permisive mode](https://istio.io/docs/tasks/security/authentication/mtls-migration/) and check with that so we would be sure it's not policy problem. – Jakub Dec 02 '19 at 14:58
  • The k8s version is 1.13,and istio version is 1.2.4.The k8s is built on a private cloud.Do I need to upgrade the istio to 1.4.0 – Li Yongsheng Dec 03 '19 at 01:05
  • No, istioctl is command line tool, it's independent of istio. Link about how to install is provided above, just two lines. Let me know if that found something, Could You please try the other suggestion with mtls and let me know if problem still exist? – Jakub Dec 03 '19 at 06:19
  • I tried both of your suggestions.No relevant problems was found to run istioctl x analyze -k.And mtls did not work,the problem still exists. – Li Yongsheng Dec 03 '19 at 07:37
  • I even upgraded istio to 1.4.0,but the problem still exits. – Li Yongsheng Dec 03 '19 at 07:39
  • I have found a github issue where Someone had same issue and was able to fix it,check out this [answer](https://github.com/istio/istio/issues/9152#issuecomment-427564282). Let me know if that help You. – Jakub Dec 03 '19 at 11:47
  • It works.But I wonder what's the cause of the problem,which configation works.So I have tried many times with different configurations.Finally, I found that only need to add conf of "websocketUpgrade: true". – Li Yongsheng Dec 04 '19 at 07:43
  • 1
    The respository of istio on github misled me a lot.[In this issue](https://github.com/istio/istio/issues/14740) ,the contributor of istio says that websocketUpgrade was removed some time ago, and is no longer needed. – Li Yongsheng Dec 04 '19 at 08:00
  • Happy to hear that it works now, it happens, there are a lot false informations and You have to check everything by yourself. Would You like me to make an answer so if someone had same issue, he will find answer here or You want to do it by yourself? – Jakub Dec 04 '19 at 08:25
  • 1
    Of course you can answer.And thank you for your help. – Li Yongsheng Dec 04 '19 at 08:40

1 Answers1

4

The problem here was websocketUpgrade, one line but important one.

As I could find on github there

Support for websockets is enabled by default in Istio from version 1.0: https://godoc.org/istio.io/api/networking/v1alpha3#HTTPRoute

And OP provided another one there

websocketUpgrade was removed some time ago, and is no longer needed.

So it should work without adding it to virtual service.

HOWEVER

As showed on github issue and confirmed by OP You still have to add it.

I found that only need to add conf of "websocketUpgrade: true".

So if you have same issue, you should try add weboscketUpgrade to your virtual service yaml.

If that doesn't work, there is another idea on github how to fix this.

Jakub
  • 8,189
  • 1
  • 17
  • 31