1

I was looking into Kubernetes Heapster and Metrics-server for getting metrics from the running pods. But the issue is, I need some custom metrics which might vary from pod to pod, and apparently Heapster only provides cpu and memory related metrics. Is there any tool already out there, which would provide me the functionality I want, or do I need to build one from scratch?

Bitswazsky
  • 4,242
  • 3
  • 29
  • 58
  • If you're using Java then you can use the micrometer library for pushing metrics to Prometheus. Is this the sort of thing you have in mind? – Ryan Dawson Aug 23 '18 at 08:58
  • Hi, all of our micro-services are java components, so that should be fine. I'm not familiar with Prometheus at all, so I'd like to know if it supports any arbitrary metric (to be analyzed later)? – Bitswazsky Aug 23 '18 at 09:28
  • You could look at adding custom metrics to the spring boot actuator and shipping to prometheus like in https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/ I realise you said heapster in your question so see https://brancz.com/2018/01/05/prometheus-vs-heapster-vs-kubernetes-metrics-apis/ on how it relates to prometheus – Ryan Dawson Aug 23 '18 at 09:55
  • You now have labels for custom metrics with Kubernetes 1.12 (Sept. 2018): https://stackoverflow.com/a/52565900/6309 – VonC Sep 29 '18 at 07:02

2 Answers2

1

What you're looking for is application & infrastructure specific metrics. For this, the TICK stack could be helpful! Specifically Telegraf can be set up to gather detailed infrastructure metrics like Memory- and CPU pressure or even the resources used by individual docker containers, network and IO metrics etc... But it can also scrape Prometheus metrics from pods. These metrics are then shipped to influxdb and visualized using either chronograph or grafana.

enzian
  • 733
  • 5
  • 13
  • just so you know - there are many alternatives to the TICK stack - a bit of googling should reveal them just fine ;-) That is if you should be looking for something else/different/more! – enzian Aug 24 '18 at 09:15
1

Not sure if this is still open.
I would classify metrics into 3 types.
Events or Logs - System and Applications events which are sent to logs. These are non-deterministic.

Metrics - CPU and Memory utilization on the node the app is hosted. This is deterministic and are collected periodically.

APM - Applicaton Performance Monitoring metrics - these are application level metrics like requests received vs failed vs responded etc.

Not all the platforms do everything. ELK for instance does both the Metrics and Log Monitoring and does not do APM. Some of these tools have plugins into collect daemons which collect perfmon metrics of the node.

APM is a completely different area as it requires developer tool to provider metrics as Springboot does Actuator, Nodejs does AppMetrics etc. This carries the request level data. Statsd is an open source library which application can consume to provide APM metrics too Statsd agents installed in the node.

AWS offers CloudWatch agents for log shipping and sink and Xray for distributed tracing which can be used for APM.

Chandra Ganapathy
  • 11
  • 2