0

Telegraf.conf :

[[outputs.prometheus_client]]
  listen = ":9237"
  path = "/metrics"
  export_timestamp = true



[[inputs.prometheus]]
  urls = [
    "http://<server_link>:8082/metrics"
  ]
  metric_version = 2

  [inputs.prometheus.tags]
    instance="<>"
    job="sidekiq"

Error from logs - Error writing to outputs.http: when writing to [https://metrics.obsmon.p2.pks.dell.com/api/prom/push] received status code: 400. body: failed pushing to ingester: user=<>: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (**err-mimir-sample-duplicate-timestamp**). The affected sample has timestamp 2023-08-17T21:30:50Z and is from series {__name__="sidekiq_jobs_queue_duration_seconds_bucket", do_env="", external_dependencies="no", feature_category="geo_replication", host="", instance="<>:8082", job="sidekiq", le="+Inf", queue="default", urgency="low", url="http://<instance>:8082/metrics", worker="Geo::Secondary::RegistryConsistencyWorker"}

System Info- Telegraf 1.25, Linux server

  • I have shut down prometheus and Telegraf is the one scraping and pushing the metrics to RemoteReadWrite

  • This issue is occuring only with metrics_version=2 and where there is histogram data like buckets , sum , count metrices

Sample prometheus output - gitlab_sql_duration_seconds_bucket{db_config_name="main",endpoint_id="Wikis::GitGarbageCollectWorker",feature_category="gitaly",le="+Inf",queue="default"} 1159 gitlab_sql_duration_seconds_bucket{db_config_name="main",endpoint_id="Wikis::GitGarbageCollectWorker",feature_category="gitaly",le="0.05",queue="default"} 1159 gitlab_sql_duration_seconds_bucket{db_config_name="main",endpoint_id="Wikis::GitGarbageCollectWorker",feature_category="gitaly",le="0.1",queue="default"} 1159 gitlab_sql_duration_seconds_bucket{db_config_name="main",endpoint_id="Wikis::GitGarbageCollectWorker",feature_category="gitaly",le="0.25",queue="default"} 1159 gitlab_sql_duration_seconds_count{db_config_name="ci",endpoint_id="Geo::EventWorker",feature_category="geo_replication",queue="default"} 234593

Since telegraf is the only one pushing the metrices to Remote readwrite , no errors should have come

pragati
  • 1

0 Answers0