How to set default retention policy and duration for InfluxDB via configuration

Question

I am using the official InfluxDB docker image. I want to set the retention policy to be 14 days by default.

There are various ENV variables that I can set to change the config for InfluxDB, such as INFLUXDB_RETENTION_POLICY. This expects the name of a retention policy such as "default" to be used as the default retention policy.

The problem is that this default policy has a duration of 7 days. I need to set it to 14 days.

The documentation is rather poor. I cannot find any ENV variable to adjust the default duration. I could also set the INFLUXDB_RETENTION_POLICY variable to a different name of a different retention policy, but I don't see how I can create that retention policy through configuration.

Is anyone aware of: 1) a way to change the default duration for retention via configuration or 2) a way to create a retention policy through configuration

Answer 1

Unfortunately there is no way to set the default retention policy via the configuration. The reason for this is that typically retention policy duration is defined during database creation.

CREATE DATABASE <database_name>
[WITH [DURATION <duration>] [REPLICATION <n>]
      [SHARD DURATION <duration>] [NAME <retention-policy-name>]]

If users were allowed to set a default retention duration via the configuration, the results of the command

CREATE DATABASE mydb 

would vary from instance to instance. While this isn't necessarily problematic, it isn't ideal either.

The problem is that this default policy has a duration of 7 days. I need to set it to 14 days.

The default retention policy in InfluxDB should be infinite.

> CREATE DATABASE mydb
> SHOW RETENTION POLICIES ON mydb
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 0s       168h0m0s           1        true

Here we see that the duration of the retention policy is 0s which is an alias for infinite and the shard group duration is 168h0m0s which is 7 days.

I think the main point of confusion here is relatively common--and mostly due to retention policies being poorly named. In InfluxDB an Database is a container for Retention Policies and a Retention Policy is a container for the actual time series data. That is to say a Retention Policy isn't so much of a policy as it as a container that has a policy for all data it contains.

My recommendation would be to always be fully explicit when creating a database in InfluxDB. Doing so will always guarantee that your database will have the correct retention policy duration. So for creating a database with a 14 day retention policy you'd issue the command

CREATE DATABASE mydb WITH DURATION 14d

Answer 2

To answer the question for those coming from google with an existing database (this was my situation), there are three ways to set the retention policy:

• On a new database, with database create (as above)
• On create of retention policy, by adding DEFAULT at the end of the create statement
• By altering an existing retention policy to be the default for the db (see below)

Creating a database

CREATE DATABASE "NOAA_water_database" WITH DURATION 3d REPLICATION 1 SHARD DURATION 1h NAME "liquid"

Creating a policy

CREATE RETENTION POLICY "one_day_only" ON "NOAA_water_database" DURATION 1d REPLICATION 1

Updating an existing policy

ALTER RETENTION POLICY "what_is_time" ON "NOAA_water_database" DURATION 3w SHARD DURATION 2h DEFAULT

So for an existing database and retention policy which you wish to make the default, the simplest solution is to use alter retention policy.

One point to note on adding/updating a policy on an existing db - data prior to the policy expiration will be immediately dropped, so you will lose all older data.

Answer 3

In fact, the mentioned environment variable to change influxdb retention policy doesn't work such as DOCKER_INFLUXDB_INIT_RETENTION mentioned in the hub.docker. That's why this issue is open since 2018 up to now.

Nevertheless, I use .iql file to set a retention policy that should be located in the directory /docker-entrypoint-initdb.d/ inside the container with the following content:

setretention.iql:

CREATE RETENTION POLICY "two_hours" ON "metrics" DURATION 2h REPLICATION 1 SHARD DURATION 1h DEFAULT

Actually, I create a new retention policy named two_years on metrics database with 2h duration and other stuff as mentioned. Therefore, its docker command would be as follows:

docker run --rm -d --name influxdb \
    -e INFLUXDB_REPORTING_DISABLED=true \
    -e INFLUXDB_DB=metrics \
    -e INFLUXDB_USER=retention_test \
    -e INFLUXDB_USER_PASSWORD=testing123! \
    -v ./setretention.iql:/docker-entrypointinitdb.d/setretention.iql \
    influxdb:1.7.11

To check that:

$ docker exec -it influxdb influx
> show retention policies on metrics

name      duration shardGroupDuration replicaN default
----      -------- ------------------ -------- -------
autogen   0s       168h0m0s           1        false
two_hours 2h0m0s   1h0m0s             1        true

---

[NOTE]:

• Bear in mind, follow this table for choosing the right values for related retention policy duration as well as shard group duration.

---

[UPDATE Apr 2022]

Apparently, due to this approved pull request, you can use the mentioned .iql query directly as an environment from influxdb 1.9 onward.

Answer 4

No one here really answered the question. The answer is that you can't create a retention policy with configuration files or env variables. So, in order to automate the solution you will need to create a "K8s Job" that import your commands into the influx CLI.

For example:

• First create this command file and wrapped it to "db_init.txt" file:

  CREATE DATABASE example CREATE RETENTION POLICY twoweeks ON example DURATION 14d REPLICATION 1 DEFAULT

• Second, create a bash script to upload this file directly to Influx CLI:

  #!/usr/bin/env bash

  influx -import -path=/tmp/db-init.txt

Wrap it up in a image and upload it as a job - and you got it automated.

Hope i have manage to help a little.

Answer 5

There is no way to set a retention policy for all databases influxDB via config. Most probably because retention policy is a container that contains your actual compressed time-series data. And it's a property of each individual database. Changing the retention policy for one database will not change the retention policy for any other.

If a retention policy is deleted then all the data it contains will be deleted. But altering the retention policy will not delete any data.

So here is a bash script to automate this task of altering retention policy

for db in $(influx -username $INFLUX_USERNAME -password $INFLUX_PASSWORD -execute="SHOW DATABASES"| grep <pattern_to_list_required_db>); do 
influx -username $INFLUX_USERNAME -password $INFLUX_PASSWORD -execute="ALTER RETENTION POLICY \"autogen\" ON \"$db\" DURATION 14d"
influx -username $INFLUX_USERNAME -password $INFLUX_PASSWORD -database="$db" -execute="SHOW RETENTION POLICIES"
done