1

I'm trying to scrape the AKS kube-proxy metrics with Prometheus without success, the reason is that some of the metrics are only exposed to localhost as the following example:

tcp LISTEN 0  16384  127.0.0.1:10249 *:*

AKS version: 1.16.9

To validate this I also tried to add this to the kube-proxy DeamonSet :

--metrics-bind-address=0.0.0.0

After the pods restart I can see the metrics, but after some minutes Azure will bring back the version to the managed one, removing the metrics bind address.

Tommy Jimmy Emiliano
  • 59
  • 7
  • Any reason you are trying to do this yourself as Azure Monitor for Containers does a pretty good job already. https://learn.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview – Hassan Raza Sep 21 '20 at 13:10
  • Yes, we are using Prometheus and Grafana to scrape the metrics! :) I know about azure monitor but we want to use the other stack since we have other application in another cluster in another Cloud. And because of this we want to have the same configuration for all the cluster we have. – Tommy Jimmy Emiliano Sep 21 '20 at 14:16
  • see if this help: https://learn.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-prometheus-integration – Hassan Raza Sep 21 '20 at 14:31
  • So is not possible to scrape those services with prometheus like a normal Kubernetes Cluster? – Tommy Jimmy Emiliano Sep 21 '20 at 15:29
  • for the managed service on AKS it is supported the way as pointed in the above article but if not then as of now i do not see support for it. – Hassan Raza Sep 21 '20 at 22:24

1 Answers1

1

There is the following thread: "kube-proxy cannot be scraped for metrics #1395" at AKS/issues on Github.

As of Sep 2020, the issue is not resolved on AKS 1.16.9, 1.16.10, 1.17.7...

the metrics are only locally available, it seems it listens to localhost only and can not be accesses from outside. Prometheus can't access it.

The discussion continued at "Prometheus scrape metrics for kube-proxy #1695". The latest post in that thread is your's, so I assume you are totally aware of how exactly scraping works on AKS as of now :-D (compared to Azure Monitor).

UPD. Just for the sake of "Search engine optimization" (the answer in that github tread) :

palma21 commented 2 days ago

This was not possible to be changed on the fleet for security concerns on clusters with 1.15 and prior. We'll take a look at this now that 1.15 has been removed.

Nick
  • 1,882
  • 11
  • 16
  • I know exactly how this works :) But no one can explain whether or not this will be updated or not. Maybe this won't never released, just need to understand if there's another way to get those infos with Prometheus. If you don't know then you are free to not answer :D – Tommy Jimmy Emiliano Sep 22 '20 at 08:39
  • @TommyJimmyEmiliano, and you eager to scrape it only with "external" Prometheus. Am I right? – Nick Sep 22 '20 at 08:42
  • I have my Prometheus server already set up, and all my applications are already exposing metrics which are eventually visualized in Grafana. Since we have other clusters in different Cloud we choose to use Prometheus and have a common architecture for the monitoring. Now I can also use Azure Monitor to scrape the metrics in AKS but you can see my problem I guess. Also don't really understand why those metrics are not exposed at all. – Tommy Jimmy Emiliano Sep 22 '20 at 09:12
  • https://prometheus.io/docs/prometheus/latest/federation/ I assume you have been considering something like that? "why those metrics are not exposed at all" <-- it's a managed solution (plus they have their own ecosystem) and I don't work for Microsoft. – Nick Sep 22 '20 at 13:49
  • We already have the Prometheus federation but this doesn't solve my problem :) it's a managed solution <-- Yes I know that, but those are metrics I don't see the point to close that port to get the metrics tbh. Also, the Kube proxy is deployed in Kubernetes, in the Kube-system namespace, so I really don't understand why it doesn't allow to get the metrics from another solution but Azure Monitor. – Tommy Jimmy Emiliano Sep 23 '20 at 09:27