How do I delete a time series from Prometheus v2, specifically a series of alerts

Question

We are getting to grips with alerting so from time to time need to clear out old alerts which we did by calling the HTTP API, to remove the pseudo time series where the alerts were stored, e.g.:

DELETE https://prometheus/api/v1/series?match[]={__name__="ALERTS"}

We have recently upgraded our Prometheus server from 1.8 to 2.2.1.

Calling this endpoint now gives

{
    "status": "error",
    "errorType": "internal",
    "error": "not implemented"
}

I have done some research and found a solution in various locations, which I will summarise in an answer below in case it's of use to my fellow StackOverflowers

Spangen · Accepted Answer · 2018-04-16T15:25:47.957

26

Firstly the admin API is not enabled by default in Prometheus 2. This must be made active by starting the server with the option

--web.enable-admin-api

There is a new endpoint in v2 at

https://prometheus/api/v2/admin/tsdb/delete_series

This takes a POST specifying the search criteria, e.g. for a time series with the name ALERTS where the alert name is MyTestAlert, post the following application/json to the delete_series endpoint, from the tool of choice (Tested with Postman 6 on Mac)

{
    "matchers": [{
        "type": "EQ",
        "name": "__name__",
        "value": "ALERTS"
    },
    {
        "type": "EQ",
        "name": "alertname",
        "value": "MyTestAlert"
    }]
}

For completeness and to free the disk space where the alerts were persisted, POST an empty payload to

https://prometheus/api/v2/admin/tsdb/clean_tombstones

Answer aggregated from:

edited Apr 16 '18 at 15:25

answered Apr 16 '18 at 14:09

Spangen

4,420
5
37
42

Using `curl` might be easier... Robust Perception has an example, https://www.robustperception.io/deleting-time-series-from-prometheus – Franklin Piat Sep 27 '18 at 15:24
Cheers, that example didn't exist back in April. Don't see the advantage of curl over any other http posting utility though. – Spangen Sep 27 '18 at 16:03
1

I get error response with "message": "unknown value \"EQ\" for enum prometheus.LabelMatcher_Type" and code 3 – user669789 Feb 25 '20 at 18:23
I deleted series and cleaned up tombstones, but I can still see the metric names in the drop-down in the Console page. There is no data for deleted metrics, but they are still there for some reason. Reload via webAPI, kill -HUP did not help. Anybody knows how to wipe the out? – mac13k Jul 21 '20 at 14:18
1

Use `/api/v1/admin/tsdb/delete_series` and `/api/v1/admin/tsdb/clean_tombstones` (v1 instead of mentions v2). See https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series – Maxim Suslov Feb 08 '23 at 10:26

score 4 · Answer 2 · answered Jan 31 '19 at 14:06

4

Curl variant:

# delete alerts series
curl -g -XPOST 'http://prometheus:9090/api/v2/admin/tsdb/delete_series?match[]=ALERTS'

# delete data from disk
curl -XPOST http://prometheus:9090/api/v2/admin/tsdb/clean_tombstones

answered Jan 31 '19 at 14:06

cybergrind

734
7
12

recent versions seem to have changed the URLs to ```/v1/``` see here: https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series – Markus Mar 01 '23 at 16:52

How do I delete a time series from Prometheus v2, specifically a series of alerts

2 Answers2