What is the best way to do CI/CD with AWS CDK (python) using GitLab CI?

Question

I am using AWS CDK (with Python) for a containerized application that runs on Fargate. I would like to run cdk deploy in a GitLab CI process and pass the git tag as an environment variable that replaces the container running in Fargate. I am currently doing something similar with CloudFormation (aws cloudformation update-stack ...). Is anyone else doing CI/CD with AWS CDK in this way? Is there a better way to do it?

Also, what should I use for my base image for this job? I was thinking that I can either start with a python container and install node or vice versa. Or maybe there is prebuilt container somewhere that I haven't been able to find yet.

Here is start that seems to be working well:

CDK:
  image: python:3.8
  stage: deploy
  before_script:
    - apt-get -qq update && apt-get -y install nodejs npm
    - node -v
    - npm i -g aws-cdk
    - cd awscdk
    - pip3 install -r requirements.txt
  script:
    - cdk diff
    - cdk deploy --require-approval never

Edit 2020-05-04:

CDK can build docker images during cdk deploy, but it needs access to docker. If you don't need docker, the above CI job definition should be fine. Here's the current CI job I'm using:

cdk deploy:
  image: docker:19.03.1
  services:
    - docker:19.03.5-dind
  stage: deploy
  only:
    - master
  before_script:
    - apk add --no-cache python3
    - python3 -V
    - pip3 -V
    - apk add nodejs-current npm
    - node -v
    - npm i -g aws-cdk
    - cd awscdk
    - pip3 install -r requirements.txt
  script:
    - cdk bootstrap aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION
    - cdk deploy --require-approval never

The cdk bootstrap is needed because I am using assets in my cdk code:

        self.backend_task.add_container(
            "DjangoBackend",
            image=ecs.AssetImage(
                "../backend",
                file="scripts/prod/Dockerfile",
                target="production",
            ),
            logging=ecs.LogDrivers.aws_logs(stream_prefix="Backend"),
            environment=environment_variables,
            command=["/start_prod.sh"],
        )

Here's more information on cdk bootstrap: https://github.com/aws/aws-cdk/blob/master/design/cdk-bootstrap.md

You can build your own python-nodejs-awscdk image. Push it publicly (hub.docker.com) and use it as a Gitlab runner base. Have another weekly Gitlab CI update the image with the latest stable python/nodejs/awscdk versions. — edd, Apr 29 '20 at 16:58

score 5 · Accepted Answer · answered Apr 30 '20 at 08:50

you definitely have to use CDK deploy inside the CI/CD pipeline if you have lambda or ECS assets, otherwise, you could run CDK synth and pass the resulting Cloudformation to AWS Code Deploy. That means a lot of your CI/CD will be spent deploying which might drain your free tier build minutes or just means you pay more (AWS Code Deploy is free)

I do something similar with Golang in CircleCi. I use the Go base image and install nodejs and cdk. I use this base image to build all my go binaries, the vuejs frontend and compile cdk typescript and deploy it.

FROM golang:1.13

RUN go get -u -d github.com/magefile/mage
WORKDIR $GOPATH/src/github.com/magefile/mage
RUN go run bootstrap.go

RUN curl -sL https://deb.nodesource.com/setup_12.x | bash -
RUN apt-get install -y nodejs
RUN npm i -g aws-cdk@1.36.x
RUN npm i -g typescript
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt update && apt install yarn

I hope that helps.

score 1 · Answer 2 · answered Oct 17 '22 at 13:36

Also, what should I use for my base image for this job? I was thinking that I can either start with a python container and install node or vice versa. Or maybe there is prebuilt container somewhere that I haven't been able to find yet.

For anyone looking for how to implement CI/CD with AWS CDK Python in 2022, here's a tested solution:

Use python:3.10.8 as the base image in your CI/CD (or any image with Debian 11)
Install Node.js 16 from NodeSource: curl -fsSL https://deb.nodesource.com/setup_16.x | bash - && apt-get install -y nodejs
Install aws-cdk: npm i -g aws-cdk

You can add the two latter steps as inline scripts in your CI/CD pipeline so you do not need to build your own Docker image.

Here's a full example for Bitbucket Pipelines:

image: python:3.10.8

run-tests: &run-tests
  step:
    name: Run tests
    script:
      # Node 16
      - curl -fsSL https://deb.nodesource.com/setup_16.x | bash - && apt-get install -y nodejs
      - npm i -g aws-cdk
      - pip install -r requirements-dev.txt
      - pytest

pipelines:
  pull-requests:
    "**":
      - <<: *run-tests
  branches:
    master:
      - <<: *run-tests

Note that the above instructions do not install Docker engine. In Bitbucket Pipelines, Docker can be used simply by adding

services:
  - docker

in the configuration file.

score 0 · Answer 3 · answered Dec 12 '22 at 17:26

If cdk deploy is giving you the error:

/usr/lib/node_modules/aws-cdk/lib/index.js:12422
    home = path.join((os.userInfo().homedir ?? os.homedir()).trim(), ".cdk");

then the node version is out of date. This can be fixed by updating the docker image which also requires pip3:

cdk deploy:
  image: docker:20.10.21
  services:
    - docker:20.10.21-dind
  stage: deploy
  only:
    - master
  before_script:
    - apk add --no-cache python3
    - python3 -V
    - apk add py3-pip
    - pip3 -V

What is the best way to do CI/CD with AWS CDK (python) using GitLab CI?

3 Answers3

Linked