PHP Composer cache in Azure DevOps Container Job

Question

I have the following pipeline in Azure DevOps for building and deploying a PHP web app. The Azure agent spins up a container based on the image (mycustomimage/pipeline-build-php8) and builds and deploys everything in the container. How can I cache the PHP Composer dependencies when using container jobs? My understanding is that the Cache@2 task exists on the host where the Azure agent is listening (AzureAgentServer) and not in the container.

After running the pipeline, I'm seeing the following output in Azure DevOps > Pipelines > Jobs:

- Cache Composer Job Succeeded 
- CmdLine Job Succeeded 
- Post-job: Cache Composer ERROR
```
Getting a pipeline cache artifact with one of the following fingerprints: Fingerprint: composer|"Linux"|VtM/KI+cqM9cqvMUIO/MUln2YuE4vqOqlDb4t8f2NFc=
There is a cache miss.
tar: /home/azagent/azagentworker/_work/18/.composer: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
```
- Post-job: Checkout Succeeded 

trigger:
  branches:
    include:
      - UAT

variables:
- group: 'Pipeline Secrets'

jobs:
- job: DeploytoUAT
  displayName: 'Deploy to UAT'
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/UAT'))
  pool:
    name: 'UAT Pool Linux'
    vmImage: 'AzureAgentServer'
  container:
    image: mycustomimage/pipeline-build-php8
  steps:
  - task: DownloadSecureFile@1
    name: AZagentSSHkey
    displayName: 'Download SSH key for Azure container user'
    inputs:
      secureFile: 'azagent_azpcontainer_ssh.key'
  - task: Cache@2
    displayName: 'Cache Composer'
    inputs:
      key: 'composer | "$(Agent.OS)" | composer.lock'
      restoreKeys: |
        composer | "$(Agent.OS)"
        composer
      path: | 
        $(Pipeline.Workspace)/.composer
  - script: |
      # Add credentials so that the azagent_azpcontainer can SSH to hosts (web servers) without password
      mkdir ~/.ssh/ && cp $(AZagentSSHkey.secureFilePath) ~/.ssh/id_rsa && chmod 600 ~/.ssh/id_rsa
      composer config --global --auth http-basic.dev.azure.com token $AZURE_DEVOPS_EXT_PAT
      composer update myproject/anotherprojectrepo --no-interaction
      composer install --no-interaction --no-progress --prefer-dist --no-dev
      composer dump-autoload --no-dev --classmap-authoritative
      composer global require deployer/deployer --dev --ignore-platform-reqs
      export PATH=$PATH:~/.config/composer/vendor/bin/
      dep deploy uat
    env:
      AZURE_DEVOPS_EXT_PAT: $(AZURE_DEVOPS_PAT)

Answer 1

Perhaps obviously, perhaps hidden, the cache path is wrong.

However, it is a configuration detail.

Make it a variable and put it at the beginning of the file, so that it is visible upfront, and you know when you set it to the correct pathname, things will work.

variables:
  COMPOSER_CACHE_DIR: $(Pipeline.Workspace)/.composer

Then use that variable in the Cache@2 task:

  - task: Cache@2
      path: |
        $(COMPOSER_CACHE_DIR)

The important part is that the cache path is a configuration parameter of both the Cache@2 task path and composer(1).

Test, commit and push. It will already run.

Differentiation

Different to GNU Make, in Microsoft Azure Devops YAML Pipelines, the variables are automatically exported.

In GNU Make you need to mark the macro for export:

export COMPOSER_CACHE_DIR := $(abspath ./.composer)

More Information

PHP/Composer

For PHP projects using Composer, set the COMPOSER_CACHE_DIR environment variable used by Composer.

Additionally, consider setting the COMPOSER_NO_INTERACTION environment variable as you're executing Composer in a non-interactive build/ci context.

Example

variables:
  COMPOSER_CACHE_DIR: $(Pipeline.Workspace)/build/cache/composer
  COMPOSER_NO_INTERACTION: 1

steps:
- task: Cache@2
  inputs:
    key: 'composer | "$(Agent.OS)" | composer.lock'
    restoreKeys: |
      composer | "$(Agent.OS)"
      composer
    path: $(COMPOSER_CACHE_DIR)
  displayName: Cache composer

- script: composer install

Reference Documentation

From: Pipeline caching: PHP/Composer, it also has support information in case this is broken and additional Q&A.

In November 2019, Alexander Danilenko reported the composer cache example working good for them and that it hits the cache successfully, without any further explanation. (Ref.)

More Questions

You had some more questions, addressing them in the order you asked.

---

How can I cache the PHP Composer dependencies when using container jobs?

With Microsoft's Azure Devops Azure Pipelines Cache@2 - Cache v2 task.

It is a file-system-based cache, its path property must be an absolute pathname that resolves to an existing directory entry of a file or directory.

To use it with Composer, set it to the pathname of the Composer cache directory for the step that runs Composer and you want to cache.

See Pipeline caching: PHP/Composer, that resource also has support information next to a detailed explanation of how it works.

---

My understanding is that the Cache@2 task exists on the host where the Azure agent is listening to (AzureAgentServer) and not in the container.

That is imprecise, and more relevant is where a task runs: "A task performs an action in a pipeline" ¹ and "By default, all tasks run in the same context, whether that's on the host or in a job container." ².

Your understanding matches in so far the localization that the agent must have all tasks prerequisites installed, otherwise it will not be chosen for a job with task step that demands those prerequisites.

The demands of the Cache@2 task are: ³

Demands: None

Troubleshooting PHP Composer cache in Azure DevOps Container Job

The actual error message of the failure:

tar: /home/azagent/azagentworker/_work/18/.composer: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now

Is from tar(1) and commanded by the pathname configured:

  - task: Cache@2
      path: | 
        $(Pipeline.Workspace)/.composer

The Post-job message for the Cache@2 task sheds more light about the consequences of this misconfiguration:

Post-job: Cache Composer ERROR

There was no cache. So what? The pipeline shouldn't be affected by that. Caches mustn't exist, they're transparent.

• Given Composer did run, also without --no-cache, we can assume that Composer has created its cache directory, (This is the default behaviour of Composer, the cache directory is created very early after its invocation.)

• When tar(1) fails beyond recovery,

• Then the path property of the Cache@2 task is wrong.

The failure confirms here that you didn't configure a path that apart from Composer would have been created by the build. This should always be your first configuration test, however, you should execute it with the reasoning that you expect it to fail and why.

If you only guess the path when you write the recipe for the first time, then you're doing it wrong, as you don't know the path but expect it to be correct. Better is, to configure one intentionally wrong, so you expect to fail first.

Configure the correct path only after the test failed for the expected reason. Verify the reason you expect it to fail, is the reason it fails.

I do this even when I know the correct path. I configure a path that I know is wrong as I want to see it fail.

The solution is then to configure the correct path of the Composer cache directory.

It may help, but that depends on your projects build, to use a more speaking path.

$(Pipeline.Workspace)/.composer
$(Pipeline.Workspace)/.cache/composer
$(Pipeline.Workspace)/build/cache/composer

What speaks more to you?

While after the first red test you still may not know what the correct path is, but you may already know that it is a pathname of a directory. The concrete directory name is a configuration detail. We defer the details then, as we don't know it yet but continue preparing the recipe.

Therefore, we make it a parameter and put the configuration on top of the file, so that it becomes immediately visible how we configure the overall pipeline (build parameter or macro):

variables:
  COMPOSER_CACHE_DIR: $(Pipeline.Workspace)/.composer

And use it in the Cache@2 task:

  - task: Cache@2
      path: |
        $(COMPOSER_CACHE_DIR)

A simple variable extraction refactoring, here to make the configuration more explicit and parameterized.

If you now run the test again, the expectation would be to see it fail again, right? The only thing that has been done is extracting the parameter, so this shouldn't have much of a difference.

Well, you may be surprised then if you try it out.

YAML pipeline variables are automatically exported to all steps, including those self-defined tasks you use to execute composer(1).

And it is that COMPOSER_CACHE_DIR is also the name of Composers' parameter for the cache directory.

And that part was missing: The cache path wasn't yet configured to composers caching directory, so we configured both at once.

The pipeline should now run with the composer cache enabled and active. Running the pipeline a second time should confirm that.