1

I am trying to use Cache Task in Azure Pipelines for the Docker setup. According to the documentation I need to set below parameters:

  • Key (Required)
  • Path (Required)
  • RestoreKeys (Optional)
 - task: Cache@2
      inputs:
        key: 'docker | "$(Agent.OS)" | cache'
        path: '$(Pipeline.Workspace)/docker'

Unfortunately, the post-job for Cache task always failing with this error. Any suggestions?

Starting: Cache
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.1
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - docker       [string]
 - "Windows_NT" [string]
 - cache        [string]
Resolved to: docker|"Windows_NT"|cache
ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session xxxx
Getting a pipeline cache artifact with one of the following fingerprints:
Fingerprint: `docker|"Windows_NT"|cache`
There is a cache miss.
tar: could not chdir to 'D:\a\1\docker'

ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session xxxx
##[error]Process returned non-zero exit code: 1
Finishing: Cache
  

Update: After making the changes in creating the direction based on the suggested answer the cache has been hit but the size of it is 0.0MB. Do we need to take care of copy ourselves?

Starting: Cache
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.1
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - docker       [string]
 - "Windows_NT" [string]
 - cache        [string]
Resolved to: docker|"Windows_NT"|cache
ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session xxxxxx
Getting a pipeline cache artifact with one of the following fingerprints:
Fingerprint: `docker|"Windows_NT"|cache`
There is a cache hit: `docker|"Windows_NT"|cache`
Used scope: 3;xxxx;refs/heads/master;xxxx
Entry found at fingerprint: `docker|"Windows_NT"|cache`

7-Zip 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21


Extracting archive: 
Expected size to be downloaded: 0.0 MB
**Downloaded 0.0 MB out of 0.0 MB (214%).
Downloaded 0.0 MB out of 0.0 MB (214%).**

Download statistics:
Total Content: 0.0 MB
Physical Content Downloaded: 0.0 MB
Compression Saved: 0.0 MB
Local Caching Saved: 0.0 MB
Chunks Downloaded: 3
Nodes Downloaded: 0

--
Path = 
Type = tar
Code Page = UTF-8

Everything is Ok




  
Dheeraj Palagiri
  • 1,829
  • 3
  • 23
  • 46
  • 1
    Does this dir exists ? like is it ever created in your pipeline or cloned from repo? `$(Pipeline.Workspace)/docker`. Please share more of your pipeline and folder structure. – The Fool Aug 26 '21 at 16:28
  • The folder does not exist, I thought the CacheTask would do that. Do we need to write copy commands too into that folder or CacheTask will copy the docker images to that folder.@TheFool – Dheeraj Palagiri Aug 28 '21 at 09:33

2 Answers2

2

I got the same issue. After creating the cache path folder before cache task, error is resolved.

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: 'New-Item -ItemType directory -Path $(Pipeline.Workspace)/docker'

As mentioned, still the cache itself didn't work as expected. I modified the cache folder, cache key and cache path to different values, since cache is immutable. And Cache key and restoreKeys are set to same value.

pool:
  vmImage: windows-2019

variables:
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/testcache1/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

steps:

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: 'New-Item -ItemType directory -Path $(MAVEN_CACHE_FOLDER)'

- task: Cache@2
  inputs:
    key: mykeyazureunique
    restoreKeys: mykeyazureunique
    path: $(MAVEN_CACHE_FOLDER)
  displayName: Cache Maven local repo

- task: MavenAuthenticate@0
  displayName: Authenticate Maven to Artifacts feed
  inputs:
    artifactsFeeds: artifacts-maven
    #mavenServiceConnections: serviceConnection1, serviceConnection2 # Optional

- task: Maven@3
  displayName: Maven deploy into Artifact feed
  inputs:
    mavenPomFile: 'pom.xml'
    goals: 'clean install'
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
    publishJUnitResults: false
    javaHomeOption: 'JDKVersion'
    mavenVersionOption: 'Default'
    mavenAuthenticateFeed: false
    effectivePomSkip: false
    sonarQubeRunAnalysis: false

Note: Cache will be set only if the job is successful.

If the cache is saved successfully, then you will see below message in the Post-job:Cache

Content upload statistics:
Total Content: 41.3 MB
Physical Content Uploaded: 17.9 MB
Logical Content Uploaded: 20.7 MB
Compression Saved: 2.8 MB
Deduplication Saved: 20.7 MB
Number of Chunks Uploaded: 265
Total Number of Chunks: 793

Now the cache is set properly, we have to make sure Cache location is picked up while execution. First thing, verify that cache is restored properly. Below log will be displayed if restore is done

There is a cache hit: `mykeyazureunique`
Extracting archive: 
Expected size to be downloaded: 20.7 MB
Downloaded 0.0 MB out of 20.7 MB (0%).
Downloaded 20.7 MB out of 20.7 MB (100%).
Downloaded 20.7 MB out of 20.7 MB (100%).

Then Cache location has to be communicated to target runner. In my case, I have used Maven. So I have set cache location in the Maven_opts.

MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'
mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
Lavanya
  • 21
  • 2
1

I could reproduce the same issue when the docker folder is not created before the cache task.

enter image description here

You need to create the folder before the cache task or directly use the existing folder.

Here is an example:

pool:
  vmImage: windows-latest

steps:
- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: 'New-Item -ItemType directory -Path $(Pipeline.Workspace)/docker'
- task: Cache@2
  inputs:
        key: 'docker | "$(Agent.OS)" | cache'
        path: '$(Pipeline.Workspace)/docker'
Kevin Lu-MSFT
  • 20,786
  • 3
  • 19
  • 28
  • Thank you, that's very useful to know. I was under the impression CopyTask will take care of it and it might be mostly a permissions issue. Now, I can see the cache has been hit but the size of it is 0.0MB. Do we need to do the copy ourselves too? Please see the update for more information. @kevin-lu-msft – Dheeraj Palagiri Aug 28 '21 at 09:36
  • Based on my test, I did not do the copy operation and it can cache the valid image – Kevin Lu-MSFT Aug 30 '21 at 07:27
  • Are you using self hosted agent? – Dheeraj Palagiri Aug 31 '21 at 07:43
  • No. I use Microsoft-hosted agent. Do you mean that you are using self-hosted agent? – Kevin Lu-MSFT Aug 31 '21 at 07:44
  • No, I have read somewhere that caching only works for self hosted agents. I am using Microsoft hosted agent. When you said, "I did not do the copy operation and it can cache a valid image" So your consecutive builds using cached image even with out copy task? how does it know what to cache and where to copy if we did not specify? – Dheeraj Palagiri Aug 31 '21 at 08:05