6

I am writing a Prometheus exporter which has to rad different CSV files. Each of them contains one entire day of data from the past (the goal is to have the exporter read a new CSV file each day. One CSV file is uploaded to the server each day, containing the data of the previous day.

IN the CSV file, I have the same metrics every 5 mns. e.g :

Date;Time;data
23.03.20;23:55:00;1
23.03.20;23:50:00;50
23.03.20;23:45:00;3

I struggle to add theses data in Prometheus properly.

class CSVCollector(object):
  def collect(self):
    # We list all the min files in the current directory
    list_min = glob.glob("min*.csv")
    metric = GaugeMetricFamily(
                'day_tests_seconds',
                'kw', labels=["jobname"])
    for min in list_min :
      with open(min) as csv_file:
        csv_reader = csv.reader(csv_file, delimiter=';')
        line_count = 0
        for row in csv_reader:
            if line_count == 1:
                correct_date_format = row[0][:6] + "20" + row[0][6:]
                datetime_row = correct_date_format + ';' + row[1]
                timestamp = int(time.mktime(datetime.datetime.strptime(datetime_row, "%d.%m.%Y;%H:%M:%S").timetuple()))
                metric.add_metric(str(line_count), int(row[4]), timestamp)
            line_count += 1
    yield metric   
     


if __name__ == '__main__':
  # Usage: json_exporter.py port endpoint
  start_http_server(int(sys.argv[1]))
  REGISTRY.register(CSVCollector())
  while True: time.sleep(1)

Prometheus just read the first line, add it as a metric and read exactly the same again each time he scrapes the exporter. What am I doing wrong ? I feel like this data should be a Jauge, as it goes up and down, but Prometheus seems that he doesn't want different data from the same Jauge in one scrape?

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
surfict
  • 63
  • 1
  • 7

3 Answers3

4

The page introduces a way to import history data to Prometheus.

The .txt file is OpenMetrics format, and an example import command is: tsdb import rrd_exported_data.txt /var/lib/prometheus/ --max-samples-in-mem=10000.

Sample file rrd_exported_data.txt ([metric]{[labels]} [number value] [timestamp ms]):

collectd_df_complex{host="myserver.fqdn.com",df="var-log",dimension="free"} 5.8093906125e+10 1582226100000
collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.client_req"} 2.3021666667e+01 1582226100000
collectd_df_complex{host="myserver.fqdn.com",df="var-log",dimension="free"} 5.8093906125e+10 1582226100000
collectd_load{host="myserver.fqdn.com",type="midterm"} 0.0155 1582226100000

Maybe you can execute the command in your python code.

Muz
  • 699
  • 1
  • 11
  • 25
3

Here is another PR (docs) just been merged. Now prometheus supports backfilling out-of-box:

promtool tsdb create-blocks-from openmetrics <input file> [<output directory>]
kvaps
  • 2,589
  • 1
  • 21
  • 20
-1

Prometheus doesn't support data importing (aka push model) - it supports only data scraping (aka pull model). See this article for details. Additionally to this, Prometheus doesn't support storing historical data (aka backfilling).

If you need importing historical CSV data to Prometheus-like system, then take a look at VictoriaMetrics - it supports importing CSV data and it supports data backfilling.

valyala
  • 11,669
  • 1
  • 59
  • 62