Istio: How do I exclude unhealthy destination from a VirtualService?

Question

I'm trying to configure load balancing and failover for external services. Each HTTP endpoint for the service needs its own specific headers.

I created a virtual service with two destinations:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: test-external
spec:
  hosts:
  - test-external.com
  http:
  - headers:
      request:
        set:
          test: "true"
    route:
    - destination:
        host: "201.returnco.de"
      weight: 50
      headers:
        request:
          set:
            Host: "201.returnco.de"
            api-key: "xxxxxxxxxx"
    - destination:
        host: "501.returnco.de"
      weight: 50
      headers:
        request:
          set:
            Host: "501.returnco.de"
            api-key: "yyyyyyyyyy"
    retries: {}

The hosts 201.returnco.de and 501.returnco.de are external services, so I created a service entry for them.

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: test-external
spec:
  hosts:
  - test-external.com
  - 201.returnco.de
  - 501.returnco.de
  location: MESH_EXTERNAL
  ports:
  - name: http
    number: 80
    protocol: HTTP
  resolution: DNS

What I want is to route requests only to 201.returnco.de. The requests should not be routed to a host which returns 5xx status code. In this case, 501.returnco.de always returns 5xx status code, so it is considered unhealthy.

How should I configure the mesh?

I tried making the following destination rule, but this doesn't work.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: test-external
spec:
  host: "*.returnco.de"
  trafficPolicy:
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      consecutiveGatewayErrors: 1
      interval: 15s
      maxEjectionPercent: 100

The mesh considers 201.returnco.de and 501.returnco.de as two separate services. After the unhealthy endpoint for the host 501.returnco.de is evicted, Istio proxy returns 503 error for requests because there are no healthy endpoints.

Configuring multiple endpoints for a single service is not ideal because I need to set different headers for each endpoint.

score 0 · Answer 1 · answered Aug 17 '23 at 06:39

0

From your configs and description, it is understood that you are creating multiple endpoints for the same service. In this scenario if you use “*.returnco.de” in your DestinationRule for OutlierDetection, when you receive 5xx errors for the endpoint 501.returnco.de as you mentioned it will evict the pods. Since 201.returnco.de is another endpoint of the same service which inturn relies on the same pods you are getting 503 errors because the pods are already evicted. In order to prevent the pods from getting evicted you can directly mention 201.returnco.de in your DestinationRule as we already know 501.returnco.de returns the 5xx errors and will evict the pods, below is the modified DestinationRule for your reference

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: test-external
spec:
  host: 201.returnco.de
  trafficPolicy:
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      consecutiveGatewayErrors: 1
      interval: 15s
      maxEjectionPercent: 100

Also if you don’t want your requests to be routed to 501.returnco.de you can give less weight such as 0 to 501.returnco.de rule and some high weight such as 100 to 201.returnco.de rule or use the explicit deny option of Istio.

answered Aug 17 '23 at 06:39

Kranthiveer Dontineni

167
8

`501.returnco.de` is just an example to emulate the failure on a service. In practice I don't know when or which destination will return 5xx errors, so I cannot simply hardcode the configuration. – hylowaker Aug 18 '23 at 05:49
@hylowaker I gave this solution because of unique use case you mentioned in the description. In general if you don't want to have multiple services you should omit outlier detection configuration for the endpoints, as both the endpoints exist on the same pod it will evict the pods as a result you will continuously get the 5xx errors. – Kranthiveer Dontineni Aug 21 '23 at 06:04
@hylowaker have you gone through my reply.., revert back if you are still facing some issues. – Kranthiveer Dontineni Aug 23 '23 at 03:10
I didn't find a solution with Istio-alone so I implemented my own Envoy Filter for this. – hylowaker Aug 28 '23 at 05:24
@hylowaker can you provide your solution here so that it will be helpful for remaining community members. – Kranthiveer Dontineni Aug 28 '23 at 06:18

Istio: How do I exclude unhealthy destination from a VirtualService?

1 Answers1