0

We can see Flink uses the Pushgateway instead of Prometheus's usual pull model for general metrics collection when exposing Flink Metrics to an external system such as Prometheus.

@Override
public void report() {
    try {
        pushGateway.push(CollectorRegistry.defaultRegistry, jobName);
    } catch (Exception e) {
        log.warn("Failed to push metrics to PushGateway with jobName {}.", jobName, e);
    }
}

https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java

however from the Prometheus's official document below it states that "Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs" , obviously Flink Streaming job is not short-lived jobs, so why Flink uses the Pushgateway instead of Prometheus's usual pull model for general metrics collection?

https://prometheus.io/docs/introduction/overview/

Michael Doubez
  • 5,937
  • 25
  • 39
YuFeng Shen
  • 1,475
  • 1
  • 17
  • 41

1 Answers1

3

Flink offers both the PrometheusPushGatewayReporter and the generally more appropriate pull-based PrometheusReporter. Prometheus has become quite popular with Flink users, and there was interest in the community in supporting both types of connection.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • David,may I know which flink metrics are suitable for PrometheusReporter and others metrics are more suitable for PrometheusPushGatewayReporter ? – YuFeng Shen Aug 22 '18 at 09:10
  • 1
    I believe the motivating use case involved running Flink in yarn-cluster mode, in which case it's difficult to point prometheus to the appropriate job manager. Pushing is much easier. I don't see it being a question of which metrics are better suited for push vs pull, but rather about the characteristics of the job or the deployment model. – David Anderson Aug 22 '18 at 09:37
  • Can you kindly help to check this issue ?https://stackoverflow.com/questions/51962849/how-to-export-flink-task-or-back-pressure-related-metrics-to-prometheus – YuFeng Shen Aug 22 '18 at 10:35