1

I have read an article about multiple pushgateway instances may cause data confusion. But I don't understand how it happen.

"When the same indicator data is pushed from different Pushgateway instances, there may be a time stamp synchronization issue or value discrepancy, which may cause data inconsistency when aggregating indicator data on the Prometheus server."

The time stamp of data series of prometheus collected is depended on prometheus collection time. How dose this have to do with pushgateway?

Pushgateway just push data to prometheus. I don't see the relation between push time stamp and collect time stamp.

I just curious why multiple pushgateway can cause lots of problems.

Of course, my final goal is to determine whether monitoring metrics collected from one server can be uploaded to different Pushgateway instances? Or better to ensure one server metrics be uploaded to some pushgateway instances.

esther lee
  • 33
  • 4

1 Answers1

1

Pushgateway just push data to Prometheus.

First of it doesn't push anything. You push your metrics to pushgateway. Later Prometheus scrapes them through usual scraping job. Those events are not synchronized in any way.

I don't see the relation between push time stamp and collect time stamp.

If you don't specify timestamp when pushing, there is no relation. Prometheus will attach timestamp of scrape to the value. Official readme:

If you push metrics at time t1, you might be tempted to believe that Prometheus will scrape them with that same timestamp t1. Instead, what Prometheus attaches as a timestamp is the time when it scrapes the Pushgateway.

Consider following example:
Imagine current value of metric : 300, after some time you push value 100, and later 200. Since your pushes and scrapes are not synchronized you can encounter timeline like this:

             push to p1, value 100
             |       push to p2, value 200
             v       v
-----------------------------------------------
^                         ^        ^               
previous scrape from p1   |        scrape p1       
                          scrape from p2

Values as seen by Prometheus:
 300 -------------------  | 200 -- | 100 -----

my final goal is to determine whether monitoring metrics collected from one server can be uploaded to different Pushgateway instances?

It is better to push same metrics to same pushgateway. It is even better to use single Pushgatway. If you considering this for redundancy you can use two instances with reverse proxy enforcing mirroring for pushing.

The best solution is of course as always not to use Pushgateway if possible.

markalex
  • 8,623
  • 2
  • 7
  • 32