I have five related questions around the configuration of Prometheus for aggregated metric reception.
There are some invariants in my system:
- Python server code that needs to be instrumented
- Prometheus service that needs to get aggregated metrics for alerts
Everything in the middle is currently being designed. I've done some reading on how this is supposed to work but have gone quite in circles.
Push Gateway expressly does not support aggregation:
The Pushgateway is explicitly not an aggregator or distributed counter but rather a metrics cache. It does not have statsd-like semantics. The metrics pushed are exactly the same as you would present for scraping in a permanently running program. If you need distributed counting, you could either use the actual statsd in combination with the Prometheus statsd exporter, or have a look at Weavework's aggregation gateway.
But then (1) why would its official Python client support Histogram, which
allows for aggregatable calculation of quantiles
?
If I go the statsd route, then statsd_exporter says that
Since the StatsD exporter uses the same line protocol as StatsD itself, you can also configure your applications to send StatsD metrics directly to the exporter. In that case, you don't need to run a StatsD server anymore.
(2) Does this mean that Prometheus does in fact support aggregation in statsd_exporter alone?
I would want to use the most popular Python bindings for statsd, simply called statsd in pypi. But it claims that
statsd is a friendly front-end to Graphite. This is a Python client for the statsd daemon.
(3) If I only used statsd_exporter, could I get away with skipping the Graphite dependency?
Finally: the docs for statsd_exporter also say that
We recommend this only as an intermediate solution and recommend switching to native Prometheus instrumentation in the long term.
The "native Prometheus implementation" link for Python brings me back to prometheus_client, but it immediately seems like that's not a client - it's tightly coupled to a server implementation, as seen here:
Metrics are usually exposed over HTTP, to be read by the Prometheus server.
The easiest way to do this is via
start_http_server, which will start a HTTP server in a daemon thread on the given port.
(4) Does that mean that the "native" instrumentation requires that you attach an instrumentation server to code that gets queried by Prometheus? (5) If this were to use Histogram, would it support aggregation?
