How the gitlab-ci cache is working on docker runner? what is /cache directory? what is cache_dir?

Question

1. How the gitlab-ci cache is working on docker runner?

2. What is /cache directory?

3. What is cache_dir?

4. Where and how files matching the "paths" in "cache" gitlab-ci.yml are stored?

Answer 1

Volume mounted to /cache directory is created automatically on gitlab-runner installation and managed by cache_dir setting

more about cache_dir:

• https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section

• https://gitlab.com/gitlab-org/gitlab-runner/blob/main/docs/executors/docker.md#the-builds-and-cache-storage

If you modify the /cache storage path, you also need to make sure to mark this
directory as persistent by defining it in volumes = ["/my/cache/"] under the
[runners.docker] section in config.toml.

---

TLDR

/cache dir is different from cache config in gitlab-ci.yml

1. /cache dir in job container is where the cached files are stored
2. files matching to cache config in gitlab-ci.yml are copied to /cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/<cache-key>-<cache-number> at the end of job
3. The "Clear Runner Caches" button in your project "Pipelines" page schedules TO NOT extract /cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/<cache-key>-<cache-number>/cache.zip to dir specified in cache config in gitlab-ci.yml (Instead of "removing content of /cache folder as I though at first)

P.S.

There is container named gitlab-runner-cache created on machine with gitlab-runner (https://gitlab.com/gitlab-org/gitlab-runner/blob/af343971874198a1923352107409583b78e8aa80/executors/docker/executor_docker.go#L382)

(Seems like) This container is used to create anonymous volume where /cache data is stored. After the anonymous volume is created this container is stopped.

The job containers (meaning container were your tests typically run) mounts this anonymous volume

---

Proofs

HAVING gitlab-ci.yml

image: srghma/docker-nixos-with-git-crypt

cache:
  key: "test00000" # to reset cache - change this key OR clear cache in project settings page
  paths:
    - .mycache # gitlab allows only cache dirs that are relative to project root OR /cache (created automatically)

testtest:
  script:
    - nix-env -i tree

    - tree --dirsfirst -L 4 /cache
    - ls -al ./.mycache || true

    - echo "test" > /cache/test
    - mkdir -p ./.mycache
    - echo "test" > ./.mycache/test

    - tree --dirsfirst -L 4 /cache
    - ls -al ./.mycache || true

Output:

1. on first run

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 675caa7 feat: cache update
From https://gitlab.com/srghma/myproject
   675caa7..3d1e223  nix        -> origin/nix
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-11...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache

0 directories, 0 files
$ ls -al ./.mycache || true
$ echo "test" > /cache/test
ls: ./.mycache: No such file or directory
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
`-- test

0 directories, 1 file
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
Creating cache test00000-11...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

2. on second run

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 3d1e223 feat: cache update
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-11...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
$ echo "test" > /cache/test
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
Creating cache test00000-11...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

3. after you clear cache by clicking on "Clear Runner Caches" in your project "Pipelines" page

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 3d1e223 feat: cache update
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-12...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
ls: ./.mycache: No such file or directory
$ echo "test" > /cache/test
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:45 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:45 ..
-rw-r--r--    1 root     root             5 Feb 24 11:45 test
Creating cache test00000-12...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

Answer 2

There are two caches when using Docker GitLab Runner actually, one for GitLab CI/CD (cache: statement in GitLab CI/CD YAML config file) and one for Docker internal data (volumes).

In the Runner configuration examples below you will see only statements relevant to the caching, nothing more. You need to add other statements like executor = "docker" in the [[runners]] section for example. See documentation for more details.

Both GitLab CI/CD cache and Docker internal cache cache_dir paths are accessed from the host running Docker. If your Docker is running on different host than your GitLab Runner, you have to ensure that the cache_dir paths (/cache_cicd or /cache_docker in examples below) are existing paths on the host running Docker. Moreover, the GitLab CI/CD cache cache_dir have to be either a Docker volume or bind-mounted in volumes array in Runner config – see examples below.

---

If you want to work just with GitLab CI/CD cache without Docker internal cache, you need to ensure that the cache_dir path is mounted inside Docker – i.e. listed in volumes array:

[[runners]]
  cache_dir = "/cache_cicd"
  [runners.docker]
    disable_cache = true
    volumes = ["cache_cicd:/cache_cicd"]

⚠️ With disable_cache = true no Docker volumes (like "/cache" – see usage below) is mounted. Bind-mounts are mounted, though.

---

Docker internal cache is independent to GitLab CI/CD cache_dir config and uses the Docker volumes by default (the volumes are named like runner-<hash>-project-<id>-concurrent-<number>-<hash>):

[[runners]]
  cache_dir = "/cache_cicd"
  [runners.docker]
    disable_cache = false
    volumes = ["/cache_cicd:/cache_cicd", "/some_volume"]

Here the /some_volume Docker volume (in my case it was named like runner-<hash>-project-<id>-concurrent-<number>-cache-<hash>) is reused by subsequent runs of same Docker Runner (see naming – even when Docker is being run from single host by multiple runners, each runner uses only it's volumes).

You can store the Docker internal cache in a specific folder too:

[[runners]]
  cache_dir = "/cache_cicd"
  [runners.docker]
    disable_cache = false
    cache_dir = "/cache_docker"
    volumes = ["/cache_cicd:/cache_cicd", "/some_volume"]

In this case the /some_volume volume is bind-mounted from subfolder of /cache_docker with unique folder name (in my case it was runner-<hash>-project-<id>-concurrent-<number>/<hash>/). Again, because of runner-specific naming, the volumes are visible only from same runner even when shared Docker host is used.

The Docker internal cache_dir should not be listed in volumes array (unless the cache_dir is same for both caches – see below).

And, finally, you can even have the Docker internal cache stored in the same path as the GitLab CI/CD cache:

[[runners]]
  cache_dir = "/cache_cicd"
  [runners.docker]
    disable_cache = false
    cache_dir = "/cache_cicd"
    volumes = ["/cache_cicd:/cache_cicd", "/some_volume"]

---

⚠️ If you do not specify the GitLab CI/CD cache_dir, the default is /cache, meaning that in the following default configuration:

[[runners]]
  [runners.docker]
    disable_cache = false
    volumes = ["/cache"]

the GitLab CI/CD cache will be stored in the /cache Docker volume, which will be visible only within the same runner – see above.

---

⛔ This will be broken:

[[runners]]
  [runners.docker]
    disable_cache = false
    volumes = []

This means that the GitLab CI/CD cache is stored in the non-existing location (/cache is neither a volume nor bind-mount), so it will not be saved and restored.