6

Problem

Right now I am writing a .gitlab-ci.yml for a Python project hosted on Gitlab.

I do not really know where to put the setup of a virtual environment and the installation of requirements from the requirements.txt.

My primary resource is the Gitlab Python.gitlab-ci.yml template.

In there, virtualenv is installed before every job, using the before_script keyword and a virtualenv named 'venv' is created:

before_script:
  - python -V  # Print out python version for debugging
  - pip install virtualenv
  - virtualenv venv
  - source venv/bin/activate

Additionally, caching for 'venv' gets set up:

cache:
  paths:
    - .cache/pip
    - venv/

I want to do the same in my script and additionally install from the requirements.txt in my repo. Afterwards, I want to execute code analysis (pylint, flake8) and tests in different stages. If I append pip install -r requirements.txt to before_script, it gets executed with every job, which produces unnecessary logs.

Question

Wouldn't it make more sense to

  1. Define cached sub-directories. (as seen above)
  2. Only call source venv/bin/activate in before_script.
  3. Add a stage or job for setup that calls virtualenv venv to generate an environment once. This job also calls pip install -r requirements.txt once.

Where would you create your venv and install your requirements?

LuGeNat
  • 179
  • 1
  • 2
  • 13
  • 1
    If the python package is running within a docker container, a virtualenv can be bypassed, by treating the container as the virtual environment. – tenacity Aug 26 '21 at 19:32

1 Answers1

5

For some python projects hosted in Gitlab the process I follow is:

  1. Create a Dockerfile. And here is where the pip install -r requirements is executed while creating the docker image that the different Runners will use in every different stage
  2. Pulling the Docker image created in step 1 in every other further stage of the pipeline ensures that you only run once the pip install -r requirements.txt probably in a stage you might call build

Example:

Dockerfile

FROM rasa/rasa-sdk:latest
MAINTAINER William Galindez Arias
COPY ./actions /app/actions/
COPY  ./requirements.txt /app/requirements.txt
EXPOSE 8000
WORKDIR /app
USER root
RUN pip install -r requirements.txt
USER 1001
ENTRYPOINT [""] 
CMD python -m rasa_sdk --actions actions -vv

The first step of the following .gitlab-ci.yml file builds the Docker image using the docker file above, and then pushes it for other stages

stages:
   - build 
   - test

build-image:
 stage: build
 script:
   - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
   - docker build -t $ACTIONS_CONTAINER_IMAGE -f Dockerfile .
   - docker push $ACTIONS_CONTAINER_IMAGE
 tags:
    - docker

test-action:
   stage: test
   image: $ACTIONS_CONTAINER_IMAGE
   script:
       - echo "testing  sdk "
William Arias
  • 147
  • 2