(@aws-cdk/pipelines) Build stage for application source code

Question

I worked through the CDK Pipelines: Continuous delivery for AWS CDK applications tutorial, which gave an overview of creating a self-mutating CDK pipeline with the new CodePipeline API.

The tutorial creates a CodePipeline with the CDK source code automatically retrieved from a GitHub repo every time a change is pushed to the master branch. The CDK code defines a lambda with a typescript handler defined alongside the CDK.

For my use case, I would like to define a self-mutating CodePipeline that is also triggered whenever I push to a second repository containing my application source code. The second repository will also contain a buildspec that generates a Docker image with my application and uploads the image to ECR. The new image will then be deployed to Fargate clusters in the application stages of my pipeline.

I've created an ApplicationBuild stage after the PublishAssets stage, which includes a CodeBuild project. The CodeBuild project reads from my repository and builds / uploads the image to ECR; however, I need a way to link this CodeBuild to the deployment of the pipeline. It's not clear to me how to do this with the new cdk CodePipeline API.

Answer 1

In case anyone has the same problem, I was able to hack together a solution using the legacy CdkPipeline API following the archived version of the tutorial I mentioned in my question.

Here is a minimum viable pipeline stack that includes...

1. a CDK pipeline source action (in "Source" stage)
2. an application source action (in "Source" stage)
3. a CDK build action (in "Build" stage) + self-mutating pipeline ("UpdatePipeline" stage)
4. an application build action (in "Build" stage)

lib/cdkpipelines-demo-pipeline-stack.ts

import * as codepipeline from '@aws-cdk/aws-codepipeline';
import * as codepipeline_actions from '@aws-cdk/aws-codepipeline-actions';
import * as core from '@aws-cdk/core';
import {Construct, SecretValue, Stack, StackProps} from '@aws-cdk/core';
import {CdkPipeline, SimpleSynthAction} from "@aws-cdk/pipelines";
import * as iam from "@aws-cdk/aws-iam";
import * as ecr from "@aws-cdk/aws-ecr";
import * as codebuild from "@aws-cdk/aws-codebuild";

/**
 * The stack that defines the application pipeline
 */
export class CdkpipelinesDemoPipelineStack extends Stack {
    constructor(scope: Construct, id: string, props?: StackProps) {
        super(scope, id, props);

        const sourceArtifact = new codepipeline.Artifact();
        const cloudAssemblyArtifact = new codepipeline.Artifact();

        const pipeline = new CdkPipeline(this, 'Pipeline', {
            // The pipeline name
            pipelineName: 'MyServicePipeline',
            cloudAssemblyArtifact,

            // Where the source can be found
            sourceAction: new codepipeline_actions.GitHubSourceAction({
                actionName: 'GitHub',
                output: sourceArtifact,
                oauthToken: SecretValue.secretsManager('github-token'),
                owner: 'OWNER',
                repo: 'REPO',
            }),

            // How it will be built and synthesized
            synthAction: SimpleSynthAction.standardNpmSynth({
                sourceArtifact,
                cloudAssemblyArtifact,

                // We need a build step to compile the TypeScript Lambda
                buildCommand: 'npm run build'
            }),
        });
        const pipelineRole = pipeline.codePipeline.role;

        // Add application source action
        const appSourceArtifact = new codepipeline.Artifact();
        const appSourceAction = this.createAppSourceAction(appSourceArtifact);
        const sourceStage = pipeline.stage("Source");
        sourceStage.addAction(appSourceAction);

        // Add application build action
        const codeBuildServiceRole = this.createCodeBuildServiceRole(this, pipelineRole);
        const repository = this.createApplicationRepository(this, codeBuildServiceRole);
        const pipelineProject = this.createCodeBuildPipelineProject(
            this, codeBuildServiceRole, repository, 'REGION', 'ACCOUNT_ID');
        const appBuildOutput = new codepipeline.Artifact();
        const appBuildAction = this.createAppCodeBuildAction(
            this, appSourceArtifact, appBuildOutput, pipelineProject, codeBuildServiceRole);
        const buildStage = pipeline.stage("Build");
        buildStage.addAction(appBuildAction);

        // This is where we add the application stages...
    }

    createAppSourceAction(appSourceArtifact: codepipeline.Artifact): codepipeline_actions.GitHubSourceAction {
        return new codepipeline_actions.GitHubSourceAction({
            actionName: 'GitHub-App-Source',
            output: appSourceArtifact,
            oauthToken: SecretValue.secretsManager('github-token'),
            owner: 'SOURCE-OWNER',
            repo: 'SOURCE-REPO',
        });
    }

    createCodeBuildServiceRole(scope: core.Construct, pipelineRole: iam.IRole): iam.Role {
        const role = new iam.Role(scope, 'CodeBuildServiceRole', {
            assumedBy: new iam.ServicePrincipal('codebuild.amazonaws.com'),
        });
        role.assumeRolePolicy?.addStatements(new iam.PolicyStatement({
            sid: "PipelineAssumeCodeBuildServiceRole",
            effect: iam.Effect.ALLOW,
            actions: ["sts:AssumeRole"],
            principals: [pipelineRole]
        }));

        // Required policies to create an AWS CodeBuild service role
        role.addToPolicy(new iam.PolicyStatement({
            sid: "CloudWatchLogsPolicy",
            effect: iam.Effect.ALLOW,
            actions: [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            resources: ["*"]
        }));
        role.addToPolicy(new iam.PolicyStatement({
            sid: "CodeCommitPolicy",
            effect: iam.Effect.ALLOW,
            actions: ["codecommit:GitPull"],
            resources: ["*"]
        }));
        role.addToPolicy(new iam.PolicyStatement({
            sid: "S3GetObjectPolicy",
            effect: iam.Effect.ALLOW,
            actions: [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            resources: ["*"]
        }));
        role.addToPolicy(new iam.PolicyStatement({
            sid: "S3PutObjectPolicy",
            effect: iam.Effect.ALLOW,
            actions: [
                "s3:PutObject"
            ],
            resources: ["*"]
        }));
        role.addToPolicy(new iam.PolicyStatement({
            sid: "S3BucketIdentity",
            effect: iam.Effect.ALLOW,
            actions: [
                "s3:GetBucketAcl",
                "s3:GetBucketLocation"
            ],
            resources: ["*"]
        }));

        // This statement allows CodeBuild to upload Docker images to Amazon ECR repositories.
        // source: https://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker.html#sample-docker-running
        role.addToPolicy(new iam.PolicyStatement({
            sid: "ECRUploadPolicy",
            effect: iam.Effect.ALLOW,
            actions: [
                "ecr:BatchCheckLayerAvailability",
                "ecr:CompleteLayerUpload",
                "ecr:GetAuthorizationToken",
                "ecr:InitiateLayerUpload",
                "ecr:PutImage",
                "ecr:UploadLayerPart"
            ],
            resources: ["*"]
        }));

        return role;
    }

    createApplicationRepository(scope: core.Construct, codeBuildServiceRole: iam.Role): ecr.Repository {
        const repository = new ecr.Repository(scope, 'Repository', {
            repositoryName: 'cdkpipelines-demo-image-repository'
        });
        repository.grantPullPush(codeBuildServiceRole);
        return repository;
    }

    createCodeBuildPipelineProject(scope: core.Construct,
                                   codeBuildServiceRole: iam.Role,
                                   repository: ecr.Repository,
                                   region: string,
                                   accountId: string): codebuild.PipelineProject {
        return new codebuild.PipelineProject(scope, 'BuildProject', {
            buildSpec: codebuild.BuildSpec.fromSourceFilename("buildspec.yml"),
            environment: {
                buildImage: codebuild.LinuxBuildImage.fromCodeBuildImageId("aws/codebuild/standard:4.0"),
                privileged: true,
                computeType: codebuild.ComputeType.SMALL,
                environmentVariables: {
                    AWS_DEFAULT_REGION: {value: region},
                    AWS_ACCOUNT_ID: {value: accountId},
                    IMAGE_REPO_NAME: {value: repository.repositoryName},
                    IMAGE_TAG: {value: "latest"},
                }
            },
            role: codeBuildServiceRole
        });
    }

    createAppCodeBuildAction(scope: core.Construct,
                             input: codepipeline.Artifact,
                             output: codepipeline.Artifact,
                             pipelineProject: codebuild.PipelineProject,
                             serviceRole: iam.Role) {
        return new codepipeline_actions.CodeBuildAction({
            actionName: "App-Build",
            checkSecretsInPlainTextEnvVariables: false,
            input: input,
            outputs: [output],
            project: pipelineProject,
            role: serviceRole,
            type: codepipeline_actions.CodeBuildActionType.BUILD,
        })
    }
}

Answer 2

Thank you @sisyphushappy for a great answer. Their approach helped me understand how codepipeline works and how it can be manipulated to create this kind of setup. In addition to the above response, some of my observations, caveats i found and work-arounds are as follows:

1. This type of CDK self-mutating AWS CodePipeline can now be built using the latest SDK version. To access pipeline object, you need to build CodePipeline object and then use get method pipeline on it. However, this get method is only accessible after the pipeline has been built. So, you need to invoke buildPipeline() on CodePipeline object before accessing this method. Another caveat, you can't add more stages to the pipeline, after invoking buildPipeline(), hence add your stages before that step. But you can add more actions after invoking it. So this is what the psuedocode looks like:

const codePipeline = new CodePipeline(....configuration...);
codePipeline.addStage(...new stage...);
codePipeline.buildPipeline();
const pipeline = codePipeline.pipeline;

....Your Source Action build steps....

pipeline.stage("<StageName>").addAction(<CodePipelineAction>);

2. There's the issue of cyclic updates and outdated builds being run because App Source Build happens before SelfMutate (which is in UpdatePipeline stage that follows Build stage). So, I added a dummy stage after UpdatePipeline stage with a dummy stack and added my app source build action to this dummy stage. Worked like a charm!

# Before
Stages:
- Source (CDKSource, AppSource)
- Build (Synth, AppBuild)
- UpdatePipeline (SelfMutate)
- DevDeploy (Prepare, Deploy)

# After
Stages:
- Source (CDKSource, AppSource)
- Build (Synth)
- UpdatePipeline (SelfMutate)
- SourceBuild (Prepare, AppBuild, Deploy) # Prepare and Deploy are dummy actions and don't do much
- DevDeploy (Prepare, Deploy)