Inject env variable into build stage of image

Question

Working on a CI/CD pipeline for a Django based API. I need to inject an environmental variable during the build stage to successfully build the image.

• python manage.py collectstatic is run in the Dockerfile when the image is being built
• collectstatic needs the SECRET_KEY in the settings.py to run
• SECRET_KEY is empty during this stage because it is read in with os.environ['SECRET_KEY]
• Because it is empty, the build fails
• Thus, I need to set the environment to have this variable

I am having trouble putting that much together, so wanted to see if anyone could assist me.

This is what I have so far:

Azure Pipeline

azure-pipelines.yml

trigger:
  branches:
    include:
    - master

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  secretKey: $(SECRET_KEY)

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and push stage
  jobs:  
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - bash:
      env:
        SECRET_KEY: $(secretKey)
    - task: Docker@2
      displayName: Build and push api image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)-api
        dockerfile: $(dockerfilePath)/api/Dockerfile
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)
    - upload: manifests
      artifact: manifests

Dockerfile

FROM python:3.7-slim
ENV PYTHONUNBUFFERED 1
WORKDIR /app
EXPOSE 5000
COPY requirements*.txt ./
RUN pip install -r requirements.txt
COPY . .
RUN python manage.py collectstatic
CMD ["gunicorn", "-b", ":5000", "--log-level", "info", "config.wsgi:application"]

Build fail log (probably not helpful because I know why it failed... SECRET_KEY isn't in env vars)

Step 8/18 : RUN python manage.py collectstatic
 ---> Running in 1f42a5c062aa
Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 325, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 79, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 66, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 157, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/config/settings.py", line 26, in <module>
    SECRET_KEY = os.environ['SECRET_KEY']
  File "/usr/local/lib/python3.7/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'SECRET_KEY'
The command '/bin/sh -c python manage.py collectstatic' returned a non-zero code: 1
##[error]The command '/bin/sh -c python manage.py collectstatic' returned a non-zero code: 1
##[error]The process '/usr/bin/docker' failed with exit code 1

I'm just not sure how to accomplish getting the environmental variable in there. My approach could be totally wrong from the beginning.

So how should I go about:

1. Declaring the environmental variables in the pipeline securely?
2. Passing said environmental variables into the build stage?

Thanks!

Answer 1

Your python manage.py collectstatic is running in the docker container, and its os.environ['SECRET_KEY] will try to get the environment variable of the container where it is running. But the environment variable SECRET_KEY you were setting in your pipeline is for build agent.

You can try following below steps to pass your pipeline env variable to the docker container.

1, I add an ARG and an ENV in your dockerfile ARG SECRET; ENV SECRET_KEY $SECRET

ENV SECRET_KEY refers to the value of ARG SECRET

FROM python:3.7-slim

ARG SECRET
ENV SECRET_KEY $SECRET

ENV PYTHONUNBUFFERED 1
WORKDIR /app
EXPOSE 5000
COPY requirements*.txt ./
RUN pip install -r requirements.txt
COPY . .
RUN python manage.py collectstatic
CMD ["gunicorn", "-b", ":5000", "--log-level", "info", "config.wsgi:application"]

2, I separate docker buildandpush task to dock build and dock push, as buildandpush command cannot accept arguments.

In docker build task. I passed the variable secretKey to the arguments field --build-arg SECRET=$(secretKey). So that when docker run build, ARG SECRET will be replaced by secretKey. And it will be passed to the ENV SECRET_KEY as defined in above dockerfile. So that SECRET_KEY will be set to the docker container's Environment variable.

Then you python code should be able to get the environment variable's value using os.environ['SECRET_KEY]

stages:
- stage: Build
  displayName: Build and push stage
  jobs:  
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Docker@2
      displayName: Build
      inputs:
        command: build
        repository: $(imageRepository)-api
        dockerfile: $(dockerfilePath)/api/Dockerfile
        containerRegistry: $(dockerRegistryServiceConnection)

        arguments: --build-arg SECRET=$(secretKey)

        tags: |
          $(tag)

    - task: Docker@2
      displayName: Push
      inputs:
        command: push
        repository: $(imageRepository)-api
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)


- upload: manifests
  artifact: manifests

Answer 2

You can inject env variables using --build-arg and corresponding ARG command in Dockerfile, but this is not secure. The key will become embedded in the image!

Your options if you want to be secure:

1. Download whatever stuff you need to using the secret key outside of Docker build, and just copy that in.
2. Use the (experimental, but less buggy than in the past) BuildKit backend for Docker, which supports passing in secrets securely.
3. Run a network server in another container, run your Docker build with --network container:<thatcontainerid> and now you can send queries to that other container to get the secrets. Or similarly run webserver on host, do --network host.

Longer writeup, which also mentions some other alternatives, like copying via multi-stage build: https://pythonspeed.com/articles/docker-build-secrets/

Answer 3

The way to do this securely in Docker 18.09+ is to use Buildkit and secrets.

Modify your pipeline as follows:

variables:
  DOCKER_BUILDKIT: 1

steps:
  - bash: |
      echo ${SECRET_KEY} > ${PIPELINE_WORKSPACE}/secret_key.txt
    displayName: Store secret
    env:
      SECRET_KEY: $(secretKey)

  - task: Docker@2
    displayName: Build
    inputs:
      command: build
      arguments: --secret id=secret_key,src=$(Pipeline.Workspace)/secret_key.txt

And in your Dockerfile:

RUN --mount=type=secret,id=secret_key \
  export SECRET_KEY=$(cat /run/secrets/secret_key) && \
  python manage.py collectstatic
CMD ["gunicorn", "-b", ":5000", "--log-level", "info", "config.wsgi:application"]