0

Setup: I have a small aws k8s cluster where the infrastructure components are deployed as helm charts. One of those components is the awesome Prometheus chart(see here). The network communication between the components should be configured with the Istio Service Mesh.

Problem: Prometheus finds the single endpoint for the kube-prometheus-alertmanager k8s service, and then keeps trying to communicate with it based on the IP address of the endpoint and not referencing it as "kube-prometheus-alertmanager..svc.cluster.local" host.

Since istio sidecar on the same node does not really know the IPs, but has the outbound routes defined for the kube-prometheus-alertmanager..svc.cluster.local host, it does not find the corresponding route and keeps telling to the prometheus container that since corresponding route can be found it can only provide the 404 HTTP response. Hence on the istio-proxy sidecar container of the prometheus POD we are seeing the lines like:

[2021-01-22T15:08:54.492Z] "POST /api/v2/alerts HTTP/1.1" 404 NR "-" 0 0 0 - "-" "Prometheus/2.24.0" "62c4ab73-1111-2222-b7d0-9aea26621ec0" "<ENDPOINT_IP>:9093" "-" - - <ENDPOINT_IP>:9093 <kube-prometheus-prometheus_IP>:37178 - -

where the 404 NR "-" clearly denotes the issue the side-car container has with routing the request.

Wish: Would be happy to hear about the best strategies for keeping the Prometheus using the Endpoints, but make it friends with the Istio.

Thanks in advance!

user15060367
  • 1
  • 1
  • why would you deploy prometheus separately? you can enable it on istio. Naturally, if you deploy it separetely, you would have to integrate it to istio. – suren May 10 '21 at 15:07

1 Answers1

1

I just had the same issue, where all internal traffic to the Alertmanager Pod IP got 404.

Solution: in kube-prometheus-stack values.yml

set

alertmanager.alertmanagerSpec.listenLocal=True

Not sure why this is, but it works.

Marcel Hofgesang
  • 951
  • 1
  • 15
  • 36
Simon
  • 13
  • 3