0

I'm trying to backfill Prometheus with old data. My input data is:

$ cat /etc/prometheus/backfill.txt
# HELP mymetric My metric.
# TYPE mymetric counter
mymetric{code="200",service="user"} 123 1665824331000
mymetric{code="500",service="user"} 456 1665824331000
# EOF

Following the docs, I then run the command which doesn't output any error:

$ promtool  tsdb create-blocks-from openmetrics /etc/prometheus/backfill.txt /tmp/backfill_data
BLOCK ULID                  MIN TIME          MAX TIME          DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01GFNDV65PR5Z6MS5J82C6SAH4  1665824331000000  1665824331000001  1ms          2            2            2            787

And then copy the created folders to Prometheus' data folder:

$ cp -r /tmp/backfill_data/* /prometheus/data

But then nothing happens: this metric is not listed in Prometheus. It seems however to be correctly stored in TSDB:

$ promtool tsdb analyze /prometheus/data
Block ID: 01GFNDV65PR5Z6MS5J82C6SAH4
Duration: 1ms
Series: 2
Label names: 3
Postings (unique label pairs): 4
Postings entries (total label pairs): 6

Label pairs most involved in churning:
2 service=user
2 __name__=mymetric
1 code=500
1 code=200

Label names most involved in churning:
2 __name__
2 code
2 service

Most common label pairs:
2 __name__=mymetric
2 service=user
1 code=200
1 code=500

Label names with highest cumulative label value length:
8 __name__
6 code
4 service

Highest cardinality labels:
2 code
1 __name__
1 service

Highest cardinality metric names:
2 mymetric

I tried to restart my Prometheus container but that didn't change anything.

Edit: I start Prometheus using the command arguments --no-scrape.adjust-timestamps --log.level=debug --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.retention.time=2y --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles --web.config.file=/etc/prometheus/web.yml

Bruno Pérel
  • 131
  • 1
  • 1
  • 5

1 Answers1

2

You are creating the backfill-file with a wrong time format, it just takes a normal unix timestamp, but yours is in milliseconds.

so, change it to:

# HELP mymetric My metric.
# TYPE mymetric counter
mymetric{code="200",service="user"} 123 1665824331
mymetric{code="500",service="user"} 456 1665824331
# EOF

and it should work.

Thoro
  • 43
  • 8