33

Is is possible to invalidate or clear a pipeline cache with Gitlab CI after a pipeline completes?

My .gitlab-ci.yml file has the following global cache definition

cache:
  key: "%CI_PIPELINE_ID%"
  paths:
    - './msvc/Project1`/bin/Debug'
    - './msvc/Project2`/bin/Debug'
    - './msvc/Project3`/bin/Debug'

The cache-key value specifies that each pipeline should maintain it's own cache, which is working fine, but the cache file continues to exist after the pipeline completes. With hundreds of pipelines being run, the size starts to add up and manually deleting the cache folder on our machine isn't a great solution.

I tried adding a cleanup job at the end of the pipeline

cleanup:
  stage: cleanup
  script:
  - rm -rf './msvc/Project1/bin'
  - rm -rf './msvc/Project2/bin'
  - rm -rf './msvc/Project3/bin'
  when: always

which deletes the local files, but won't delete them from the cache.

Am I missing something here?

Currently running Gitlab-EE 10.3.3

Dan
  • 1,805
  • 2
  • 18
  • 21
  • 3
    in your case I would rather use artifacts than cache. artifacts can be passed from one job to the other within a pipeline and they are not persistent between pipelines. you can deactivate cache alltogether – w.stoettinger Apr 26 '18 at 15:11
  • Yes, you are absolutely correct. That's the conclusion we finally came to as well. – Dan Apr 26 '18 at 15:54

3 Answers3

33

Artifacts are the solution as mentioned in the comments. However there is an option to clear caches in the Pipelines page as shown in the image below.

enter image description here

Merhawi Fissehaye
  • 2,482
  • 2
  • 25
  • 39
  • 2
    Not sure, if this was removed. However, at current non-commercial gitlab the button does not exist. – Jay-Pi Jan 23 '21 at 18:37
  • I can still find it in my projects although I am using the free version. I can find it even in new projects that I create. – Merhawi Fissehaye Jan 24 '21 at 04:59
  • 2
    Fissehay You need to be maintainer to clear the cache. Thats annoying, because a developer can push to `.gitlab-ci.yml` doing the same. – Jay-Pi Jan 28 '21 at 22:07
  • 2
    Note that the Clear Runner Caches does not actually delete the cache data. You have to [delete it manually](https://docs.gitlab.com/ee/ci/caching/#clear-the-cache-manually) – Bart Swennenhuis Oct 12 '21 at 16:54
20

It's not a perfect solution, but we ended up creating a cleanup job at the end of our .gitlab-ci.yaml file that deletes the cache directory from the filesystem.

This way, each pipeline gets its own unique cache, without cluttering up the file system over time.

cleanup_job:
  stage: cleanup
  script:
    - echo "Cleaning up"
    - rm -rf "%CACHE_PATH%/%CI_PIPELINE_ID%"
  when: always

where

CACHE_PATH: "C:/gitlab-runner/cache/group/project/repo/"
Dan
  • 1,805
  • 2
  • 18
  • 21
2

Quote from @ayufan one of the project masters of Gitlab:

The cache is stored in /home/gitlab-runner/cache or as docker container docker ps | grep -cache-.

There fore you can try to delete the directory and purge all returned docker containers.

secustor
  • 3,001
  • 2
  • 14
  • 20
  • 3
    I'm aware where the caches are stored. My goal is to avoid having to manually go and periodically delete them from our build server. – Dan Jan 27 '18 at 17:38
  • 1
    If it is the manual task which is bothering you, you can use Cronjobs or [Scheduled Pipelines](https://docs.gitlab.com/ce/user/project/pipelines/schedules.html). But directly deleting cache within the same runtime is not possible at the moment. – secustor Jan 27 '18 at 18:30
  • 3
    The problem with deleting it manually is that I don't know from terminal which pipeline cache should be cleared or persisted. It seems to me that the cache should detect if a file has been removed by the running script, and update the cache – Dan Jan 27 '18 at 21:26