Cache NodeJS tool in DevOps

Question

Is it possible to use Cache with the NodeTool task on order to skip downloading a specific version of tools and use a version shared across multiple pipelines? Right now NodeTool task occasionally fails to download the tool which causes intermittent errors. I'm a little unclear on how the Cache task works to allow another task to conditionally run the first time, and also where/how to get the cache to restore the Node JS tool to the appropriate path consistent with what NodeTool would do. This is a pipeline hosted in DevOps.

Note I'm not asking about catching NPM packages. Just a version of NodeJS tool supplied by NodeTool task.

Most currently supported versions of Node are pre-cached on the agent. What version of Node are you using? The NodeTool task doesn't have any cache support and the Cache task doesn't normally want to operate outside of the job's workfolder. — jessehouwing, Feb 27 '23 at 17:10
Using v14, one day the download was failing for several jobs, but it appeared that the task continued in this case and silently used v18 instead. Unfortunately this legacy project uses alot of older NPM packages that don't work with v18. So I assume this is because did v18 was already be present as you described it "found" it in the absence of the failed v14 download. So my hope was to cache v14 to avoid this intermittent problem. — AaronLS, Mar 04 '23 at 21:53
If download fails, the task *should* fail if you still have the execution logs of those runs, submit them in an issue to the Microsoft/azure-pipelines-tasks repo. Is your UseNode task configured to not fail the pipeline by any chance? https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/UseNodeV1/installer.ts#L207 — jessehouwing, Mar 05 '23 at 09:26
Continue on Error is not enabled. I tend to look for work arounds that don't rely on MS to fix, hence this question. They auto close or even explicitly close bugs that don't generate a ton of noise from community. I have found several bugs while refining pipelines in the last couple months and found others with same issues. I don't waste my time with that nonsense anymore. I've even seen where staff respond to confirm the bug, but still close it: https://github.com/microsoft/azure-pipelines-tasks/issues/7952 They don't even update docs so everyone else gets to stumble on these known issues. — AaronLS, Mar 05 '23 at 19:11
Yeah, I've seen that happen as well, but asking on StackOverflow won't really help as we can't provide a proper solution here. Instead of relying on the cache task you could simply zip up the Node folder in the tool cache and extract that to your agent in a future run. No need to use the cache task at all, just a few simple lines of script. — jessehouwing, Mar 05 '23 at 19:30
That issue you reference though, was closed by the author of the issue, not the Pipelines team. The Replace Existing Files was actually fixed: https://github.com/microsoft/azure-pipelines-tasks/commit/fe9eb5e00e1797c9123f46263a189309aa4a7426. The best place to cause a fix to happen is still the official repo, optionally WITH a pull request. Though many changes aren't accepted, because it would constitute a breaking change, even if it fixes a bug. Super irritating, but inexpectedly changing 1000s of pipelines globally is also an issue. — jessehouwing, Mar 05 '23 at 19:36
@jessehouwing "Yeah, I've seen that happen as well, but asking on StackOverflow won't really help as we can't provide a proper solution here. " I'm not asking stackoverflow to fix the bug with the NodeTool download issue. I'm asking about using the Cache task to cache the tool. The bug and the question are unrelated regardless of the former making the latter more important. Even if the NodeTool was behaving properly where it failed the task on a download error, it would still stand that it'd be appropriate to cache the tool and eliminate that download, especially on multistage builds. — AaronLS, Mar 05 '23 at 21:23

jessehouwing · Answer 1 · 2023-03-05T23:05:24.053

You can't use the Cache task, because it will only cache things within the working directory of the job.

But it's relatively easy to re-populate the hosted tools cache at the start of the job.

The agent keeps a toolcache folder under which each tools installer puts the contents and then updates the path. The path to the tools cache is stored in a predefined variable: Agent.ToolsDirectory.

This folder uses a predefined structure:

{cache root}
    {tool name}
        {semantic version}
            {platform}
                 {tool files}

In the case of Node 16.19.1 this ends up here: $(Agent.ToolsDirectory)/node/16.19.1/x64

If you zip up the folder $(Agent.ToolsDirectory)/node/16.19.1/ and store it somewhere (Azure DevOps Universal Artifacts comes to mind), you can restore the folder at the start of the job. The UseNode task will detect the presence of the restored files and it will automatically use it.

Note: In case of a linux or mac agent, make sure you restore the executable bits

You can run a pipeline once to create the zip and upload it to universal artifacts. You can then use this artefact from any other workflow that needs to restore this specific tool.

Note: The thing I can't understand is how the pipeline would default to a different Node version. The UseNode task is setup to fail the pipeline when it can't give you the version of node you want.

Something like:

name: Backup Node

pool:
  vmImage: ubuntu-latest

steps:
- task: UseNode@1
  inputs:
    version: 14.x

- script: |
    nodeversion=$(node --version)
    echo "##vso[task.setvariable variable=nodeversion]${nodeversion:1}"

- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Agent.ToolsDirectory)/node/$(nodeversion)/'
    includeRootFolder: true
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/node.zip)'
    replaceExistingArchive: true

- task: UniversalPackages@0
  inputs:
    command: 'publish'
    publishDirectory: '$(Build.ArtifactStagingDirectory)/node.zip)'
    feedsToUsePublish: 'internal'
    vstsFeedPublish: '6484ebc3-af16-4af9-aa66-6b3398db7214/72f9ef5d-b5b8-445b-87b9-90b7a3fa4305'
    vstsFeedPackagePublish: 'node'
    versionOption: 'custom'
    versionPublish: '$(nodeversion)'

And then to restore:

- task: UniversalPackages@0
  inputs:
    command: 'download'
    downloadDirectory: '$(Agent.ToolsDirectory)/node/'
    feedsToUse: 'internal'
    vstsFeed: '6484ebc3-af16-4af9-aa66-6b3398db7214/72f9ef5d-b5b8-445b-87b9-90b7a3fa4305'
    vstsFeedPackage: 'node'
    vstsPackageVersion: '14.*'

- script: |
    # Fix the permissions on node and npm etc.
    chmod +x ...

- task: UseNode@1
  inputs:
    version: 14.x

Make sure you grant the Project Collection Build Service contributor permissions on the package feed.

Cache NodeJS tool in DevOps

1 Answers1