0

I am using Istio 1.1.6, Prometheus 2.18.1, in conjunction with Seldon 1.1.0 (for Machine Learning); when Prometheus is attempting to scrape the endpoints, we have the following settings for the Istio ConfigMap:

outboundTrafficPolicy:
  mode: ALLOW_ANY

This should, as per the documentation, allow Prometheus to scrape any endpoint which does not have an associated Service associated with it (via VirtualServices). However, I consistently get the following alert blasting the access logs:

2020-05-27 12:56:43 
[2020-05-27T16:56:42.448Z] "- - -" 0 - "-" 247 0 0 - "-" "-" "-" "-" "10.51.105.4:6001" PassthroughCluster 10.51.105.4:35708 10.51.105.4:6001 10.51.106.2:51760 -
Show context
2020-05-27 12:56:43 
[2020-05-27T16:56:42.448Z] "- - -" 0 UO "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.51.105.4:6001 10.51.105.4:35708 -

Now, when I bash-exec into a container inside the pod, the destination, localhost:6001 is reachable, and is formatted for scraping by Prometheus.

In addition, the DestinationRule has no short-circuit limitations:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  generation: 1
  name: destination-pod
  ownerReferences:
  - apiVersion: machinelearning.seldon.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: SeldonDeployment
    name: destination
spec:
  host: destination-pod
  subsets:
  - labels:
      version: 41949a6
    name: pod

What might be the reason that Istio is appearing to block this?

Eric Meadows
  • 887
  • 1
  • 11
  • 19
  • What you mean by `allow Prometheus to scrape any endpoint which does not have an associated Service associated with it (via VirtualServices)`? Istio has an outboundTrafficPolicy, that configures the sidecar handling of external services, that is, those services that are not defined in Istio’s internal service registry. If this option is set to ALLOW_ANY, the Istio proxy lets calls to unknown services pass through. But it´s connected with service entry, not virtual services, 2 different things. Take a look at this [example](https://istiobyexample.dev/monitoring-egress-traffic/). – Jakub Jun 01 '20 at 12:24
  • It's an Istio & Kubeflow 1.x problem - used istioctl for 1.5.4, with a backported patch to make it work. Kubeflow's integration with Istio and other tools caused the issue. – Eric Meadows Jun 09 '20 at 16:01

0 Answers0