3

We use Prometheus to scrape Spring Boot 2.0.0 metrics and then persist them in InfluxDB. We then use Grafana to visualize them from InfluxDB.

Our micrometer dependencies are

  • micrometer-core
  • micrometer-registry-prometheus

I want to be able to show a latency metric for our REST APIs.

From our Prometheus scraper I can see these metrics are generated for HTTP requests.

  • http_server_requests_seconds_count
  • http_server_requests_seconds_sum
  • http_server_requests_seconds_max

I understand from the micrometer documentation, https://micrometer.io/docs/concepts#_client_side, that latency can be done by combining 2 of the above generated metrics: totalTime / count.

However our data source is InfluxDB which does not support combining measurements, https://docs.influxdata.com/influxdb/v1.7/troubleshooting/frequently-asked-questions/#how-do-i-query-data-across-measurements, so I am unable to implement that function in InfluxDB.

Do I need to provide my own implementation of this latency metric in the Spring Boot component or is their an easier way that I can achieve this?

Vy Do
  • 46,709
  • 59
  • 215
  • 313
Donal Hurley
  • 159
  • 2
  • 9
  • If you collect data with Prometheus - why don't you store & process it with Prometheus? Vice versa, if you intend to store & process your data in Influx - why don't you send it to Influx directly? – Yuri G Dec 17 '18 at 15:11
  • It is the common pattern we use for gathering metrics from microservices in our cluster. We use prometheus to scrape the metrics from services and export the data directly to influxdb. We could implement as you describe but then would not be following our common pattern. – Donal Hurley Dec 18 '18 at 15:17
  • I get the fact you use it that way right from your initial text, there was no need to repeat it. I was asking - WHY did you do it? See, in fact, you'd unnecessarily overengineered your solution (and to add insult to injury - you're trying to combine products with two significantly different approaches) - and that's the only how & why you ever stumbled upon a problem like this. – Yuri G Dec 18 '18 at 15:42

1 Answers1

0

You essentially can join your measurements in Kapacitor, another component of Influxdata TICK stack.

It's going to be pretty simple with JoinNode, possibly followed by Eval to calculate what you want right in place. There's tons of examples around it in documentation.

Although the problem is different there: you'd unnecessarily overingeneered your solution, and moreover - you're trying to combine two products that has the same purpose, but uses different approach to it. How smart is that?

You're already scraping things with Prometheus? Fine! Stay with it, do the math there, it's simple. And Grafana works with Prometheus too, right out of the box!

You wanna have your data in Influx (I can understand that, it's certainly more advanced)? Fine! Micrometer can send it right to Influx out of the box - and in at least two ways!

I, personally, don't see any reason to do what you suppose to do, can you share one?

Yuri G
  • 1,206
  • 1
  • 9
  • 13
  • Thanks for the suggestions. We were using a Prometheus server to pull metrics from different services and then decorate them with extra information specific to the environment like role, ip, instance. etc. and forward them to InfluxDB. But as you have highlighted they are 2 different products each with a different focus. Our choice is to change the way we send to InfluxDB or work around it for the moment. – Donal Hurley Dec 19 '18 at 14:20
  • You can just simply join your points in in Kapacitor, with no further aggregations/calculations & stream it back to Influx as one measurement, which you'd be querying from Grafana. Aside of Kapacitor provision (though the installation by itself must be pretty easy) - that seems to the least painful way, if you're adamant to keep your current workflow. – Yuri G Dec 19 '18 at 19:31