Splitting out build and deployment pipelines and can't figure out where the k8s manifest uploads need to go

Question

So I took the system generated azure-pipelines.yml and now I'm splitting it out into build.yaml and deployment.yaml that each have their own pipelines in Azure DevOps.

I've added branch and build validation to the production branch so that PRs are required to merge to it. When a PR is created, it automatically runs the Build, and therefore build.yaml pipeline.

If it passes, it can be approved and merged into production. The merge then triggers the Deployment, and therefore deployment.yaml pipeline.

Build is working. Deployment is not and the reason is it can't find the k8s manifests... basically this section in the system generated azure-pipelines.yml:

- upload: k8s
  artifact: k8s

Here is the full context of what was automatically produced:

trigger:
- production

resources:
- repo: self

variables:

  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: '<GUID>'
  imageRepository: 'app'
  containerRegistry: 'appacr.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)'
  tag: '$(Build.BuildId)'
  imagePullSecret: 'appacr1c5a-auth'

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build stage
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Docker@2
      displayName: Build and push an api image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)-api
        dockerfile: $(dockerfilePath)/api/Dockerfile
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)

    - upload: k8s
      artifact: k8s

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  jobs:
    - deployment: Deploy
      displayName: Deploy
      pool:
        vmImage: $(vmImageName)
      environment: 'App Production AKS'
      strategy:
        runOnce:
          deploy:
            steps:
            - task: KubernetesManifest@0
              displayName: Create imagePullSecret
              inputs:
                action: createSecret
                secretName: $(imagePullSecret)
                kubernetesServiceConnection: 'App Production AKS'
                dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
            - task: KubernetesManifest@0
              displayName: Deploy to api Kubernetes cluster
              inputs:
                action: deploy
                kubernetesServiceConnection: 'App Production AKS'
                manifests: |
                  $(Pipeline.Workspace)/k8s/aks/api.yaml
                imagePullSecrets: |
                  $(imagePullSecret)
                containers: |
                  $(containerRegistry)/$(imageRepository)-api:$(tag)

And this is how I have it split:

# build.yaml (works)
trigger: none

resources:
- repo: self

variables:
- template: templates/variables.yaml

stages:
- stage: Build
  displayName: Build stage
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Docker@2
      displayName: Build and push an api image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)-api
        dockerfile: $(dockerfilePath)/api/Dockerfile
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)
    - upload: k8s
      artifact: k8s

And...

# deployment.yaml (does not work)
trigger: 
  branches: 
    include:
    - production

resources:
- repo: self

variables:
- template: templates/variables.yaml

stages:
# - template: templates/changed.yaml
- stage: Deploy
  displayName: Deploy stage
  # dependsOn: Build
  jobs:
  - deployment: Deploy
    displayName: Deploy
    pool:
      vmImage: $(vmImageName)
    environment: 'App Production AKS'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: PublishPipelineArtifact@1
            displayName: Publish k8s manifests
            inputs:
              targetPath: $(Build.SourcesDirectory)/k8s
              artifactName: k8s
              artifactType: pipeline
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              kubernetesServiceConnection: 'App Production AKS'
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)        
          - task: KubernetesManifest@0
            displayName: Deploy to api Kubernetes cluster
            inputs:
              action: deploy
              kubernetesServiceConnection: 'App Production AKS'
              manifests: |
                $(Pipeline.Workspace)/k8s/aks/api.yaml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository)-api:$(tag)

I've tried about a dozen different things for the publishing of the k8s manifests and the error message is usually something like:

##[error]Path does not exist: /home/vsts/work/1/s/k8s

# or 

##[error]No manifest file(s) matching /home/vsts/work/1/k8s/aks/api.yaml was found.

Suggestions for how to resolve this?

There is obviously some concept I am failing to grasp completely.

Publish the manifests as a build artifact, or clone the repository containing the manifests during your deployment pipeline. — Daniel Mann, Mar 14 '21 at 14:49

score 0 · Accepted Answer · answered Mar 14 '21 at 21:58

Well, whether or not this is considered good/practice is another matter, but I did get it working with the following:

# deployment.yaml (does not work)
trigger: 
  branches: 
    include:
    - production

resources:
- repo: self

variables:
- template: templates/variables.yaml

stages:
- stage: Publish
  displayName: Publish artifacts
  jobs:
  - job: Publish
    displayName: Publish
    pool:
      vmImage: $(vmImageName)
    steps:
    - upload: k8s
      artifact: k8s
- stage: Deploy
  displayName: Deploy stage
  dependsOn: Publish
  jobs:
  - deployment: Deploy
    displayName: Deploy
    pool:
      vmImage: $(vmImageName)
    environment: 'App Production AKS'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              kubernetesServiceConnection: 'App Production AKS'
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)        
          - task: KubernetesManifest@0
            displayName: Deploy to api Kubernetes cluster
            inputs:
              action: deploy
              kubernetesServiceConnection: 'App Production AKS'
              manifests: |
                $(Pipeline.Workspace)/k8s/aks/api.yaml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository)-api:$(tag)

Splitting out build and deployment pipelines and can't figure out where the k8s manifest uploads need to go

1 Answers1