-1

I'm quite new to istio and experimenting Istio circuit breaker concepts, for that I have configured the istio virtual service, destination rules and nginx pods.

My virtual service have two different nginx pods, I configured 50-50 traffic distribution, For testing the circuit breaker I scaled down my service B to zero, thought of expectation should route the traffic always to Service A. But It's not working a expected

I'm getting no healthy upstream error, whenever traffic routed to service B

Kindly find the configuration files for the same

Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-1
  labels:
    app: nginx-deployment-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      version: "1"
  template:
    metadata:
      labels:
        app: nginx
        version: "1"
    spec:
      containers:
      - name: nginx-1
        image: nginx:latest
        ports:
        - containerPort: 80

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-2
  labels:
    app: nginx-deployment-2
spec:
  replicas: 0
  selector:
    matchLabels:
      app: nginx
      version: "2"
  template:
    metadata:
      labels:
        app: nginx
        version: "2"
    spec:
      containers:
      - name: nginx-2
        image: nginx:1.14.2
        ports:
        - containerPort: 80

VirtualService.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-virtual-service
spec:
  hosts:
  - "*"
  gateways:
    - nginx-gateway
  http:
    - route:
      - destination:
          host: nginx-service
          subset: v1
          port: 
            number: 80
        weight: 50
      - destination:
          host: nginx-service
          subset: v2
          port: 
            number: 80
        weight: 50
      # retries:
      #   attempts: 3
      #   perTryTimeout: 1s
      #   retryOn: 5xx
      # timeout: 3s
      
**service.yaml**

    apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx-service
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http-web
  selector:
    app: nginx

destinationrule.yml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-load-balancer
spec:
  host: nginx-service
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
      tcp:
        maxConnections: 1
    outlierDetection:
        baseEjectionTime: 60s
        consecutive5xxErrors: 1
        interval: 2s
        maxEjectionPercent: 100
  subsets:
  - name: v1
    labels:
      version: "1"
  - name: v2
    labels:
      version: "2"

I'm getting the below error while routing the traffic to service B, Since I scaled down that pod to zero for testing, Not sure circuit breaker will trip and block the service B request and send all traffic only to Service A

enter image description here

Anyone pls advise on this ?

Debugger
  • 690
  • 1
  • 18
  • 41
  • 1
    Can you check if the below article is useful in resolving the issue. https://developers.redhat.com/articles/2022/12/23/troubleshooting-no-healthy-upstream-errors-istio-service-mesh# – Kiran Kotturi May 17 '23 at 10:52
  • Have you gone through the above referenced article. Also,since you got the "no healthy upstream" error while routing the traffic to service B,you need to update the "VirtualService" yaml file to distribute all requests to v1 (Service A). – Kiran Kotturi May 18 '23 at 10:56
  • 1
    Thanks for sharing @KiranKotturi, I too suspect Virtual Service weightage is the problem, But it works in this tutorial without changing the traffic percentage https://www.youtube.com/watch?v=hn6YfgXMRz8 So with weightage we couldn't able to achieve this flow ? – Debugger May 18 '23 at 11:45
  • Based upon the above comments, I have posted this as an answer for greater visibility for the community.If the answer was useful, please mark the answer as accepted or upvote if the answer has some useful information. – Kiran Kotturi May 19 '23 at 07:43

1 Answers1

0

As per the information provided, you are working on Istio circuit breaker concept and in order to send all the traffic to Service A you have scaled down the Service B pod to zero for testing.But you are getting “no healthy upstream” error while routing the traffic to service B.

Option 1:

As per the article written by Stelios Kousouris to identify a problem related to Istio configuration, you can use Istio's Kiali console to visualize the network state and pinpoint where issues are occurring.Kiali allows you to "play back" network behavior in a timely manner.

Once the network state is visualized,you need to update the "VirtualService" yaml file to distribute all requests to v1 (Service A) only as the initial distribution is 50-50 for both Service A and Service B in "VirtualService" yaml file.

Option 2:

Alternately, If possible , update the network policy to below in “destinationrule” yaml file as you want to use a Load Balancer in your deployment and maintain the Outlier detection values as same.

trafficPolicy:
    loadBalancer:
      simple: RANDOM

In this case there is no need to change the weightage distribution in the “VirtualService" yaml file.

Hope the above information is useful to you.

Kiran Kotturi
  • 492
  • 6