table manager does not purge chunk in loki 2.4

Question

I wanted to have a deletion of chunk data older than 31 days so I made such config

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: "/var/lib/loki/boltdb-shipper-active"
    cache_location: "/var/lib/loki/boltdb-shipper-cache"
    cache_ttl: 24h 
    shared_store: filesystem
  filesystem:
    directory: /var/lib/loki/chunks

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 31d

but actually the deletion never happen.

did I forget something in the configuration ?

loki version: 2.4.1

score 7 · Answer 1 · answered Jan 18 '22 at 05:13

Retention in Grafana Loki is achieved either through the Table Manager or the Compactor. In your case, boltdb-shipper is used for index and chunks. Table manager isn't a great choice when it comes to local storage.

The Compactor retention will become the default and have a long-term support. It supports more granular retention policies on per tenant and per stream use cases.

Please kindly check the Guide here

compactor:
  retention_enabled: true
  retention_delete_delay: 2h <amount of hours to delay the retention period>
  retention_delete_worker_count: 150
limits_config:
  retention_period: [30d]

score 6 · Accepted Answer · answered Dec 27 '21 at 11:46

6

I'm not sure it's your case, but starting on the Loki 2.4.0, the single binary no longer runs a table-manager, this impacts anyone in the following scenarios:

Running a single binary Loki with any index type other than boltdb-shipper or boltdb
Relying on retention with the configs retention_deletes_enabled and retention_period

See more info and how to resolve this issue, in the Loki Upgrade Guide, here.

answered Dec 27 '21 at 11:46

Marcelo Ávila de Oliveira

19,950
3
39
50

Thanks for this docs, but my index is using `boltdb-shipper` and I can see the table-manager running because there is such events each minute `caller=table_manager.go:171 msg="uploading tables"` however I added the config as described but nothing happened. – Baptiste Mille-Mathias Dec 27 '21 at 13:34
1

indeed I had to adapt the config as pointed, and also I had to set `retention_delete_delay` and `delete_request_cancel_period` to a short value to have the deletion happen quickly after restarting loki and now this is working. thanks – Baptiste Mille-Mathias Dec 27 '21 at 14:19
@BaptisteMille-Mathias Could you share your configuration? I did tests but I can't get the logs to be deleted. – Ricardo Cardona Ramirez May 06 '22 at 12:54

score 1 · Answer 3 · answered Jun 02 '22 at 07:37

1

I had the same issue with Loki version 2.4.1. The chunks from the /data/loki/chunks directory were not deleted by the compactor. Tried all config variables I could find on the internet and the issue turned out to be some bug in this version of Loki. Upgrading to 2.5.0 resolved the issue for me and now I see that chunks are being removed and after the retention period is reached the volume doesn't grow any more.

answered Jun 02 '22 at 07:37

Atanas Stoyanov

91
8

how did you upgraded could you share steps and sample loki-config.yml file – shubham sachan May 27 '23 at 10:13
Have the same issue for my loki 2.4.1 chunks are not getting deleted – shubham sachan May 27 '23 at 10:14
@shubhamsachan we are running Loki in a Kubernetes cluster and I only upgraded the helm chart version that we are using, which also bumped the Loki version from 2.4.1 to 2.5.0. I would recommend upgrading, which should be the fastest and easiest solution. – Atanas Stoyanov Jun 15 '23 at 08:20

table manager does not purge chunk in loki 2.4

3 Answers3