Goal
I would like to run a Github actions workflow that has multiple jobs. I would like those jobs to have as little duplication as possible (I've come to accept that some duplication may be unavoidable).
Current solution
My first job (build-dev) installs dependencies and my second job (unit-test) runs my unit tests. The code is:
jobs:
build-dev:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v4
with:
architecture: ${{ env.ARCHITECTURE }}
cache: pipenv
python-version: ${{ env.PYTHON_VERSION }}
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Install Pipenv
run: python -m pip install --upgrade pipenv
- name: Install dependencies
run: pipenv install --dev
unit-test:
runs-on: ubuntu-latest
needs: build-dev
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v4
with:
architecture: ${{ env.ARCHITECTURE }}
cache: pipenv
python-version: ${{ env.PYTHON_VERSION }}
- name: Install Pipenv
run: python -m pip install --upgrade pipenv
- name: Unit test
run: pipenv run unit-test
Expected solution
Ideally, I'd like my second job (unit-test) to have access to the tools added to the runner used in my first job (build-dev). I'd expect the code to look something like this:
jobs:
build-dev:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v4
with:
architecture: ${{ env.ARCHITECTURE }}
cache: pipenv
python-version: ${{ env.PYTHON_VERSION }}
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Install Pipenv
run: python -m pip install --upgrade pipenv
- name: Install dependencies
run: pipenv install --dev
unit-test:
runs-on: ubuntu-latest
needs: build
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v4
with:
architecture: ${{ env.ARCHITECTURE }}
cache: pipenv
python-version: ${{ env.PYTHON_VERSION }}
- name: Unit test
run: pipenv run unit-test
Note the removal of the Install Pipenv step in the second job.
Currently, using the code above results in the error:
Run pipenv run unit-test
/home/runner/work/_temp/ef4fd997-0bb7-4623-8a8e-9aeb9fb6192d.sh: line 1: pipenv: command not found
Error: Process completed with exit code 127.
due to Pipenv not being installed on the second runner.
What I've tried
As you can see from the above code, I've attempted caching. This seems to specifically cache the dependencies installed by pipenv, and not pipenv itself.
Previously, I've uploaded an artifact, and used the global env var PIPENV_VENV_IN_PROJECT=enabled to effectively do the same as caching. Again, the same happened. I kept the installed dependencies and not the installed tools. I guess, I could artifact the runners /bin but the seems heavyweight and sounds like it could cause problems.
Some further thoughts
Please correct me if I'm wrong here, Isn't the point of CI/CD tooling to build packages and push that package through a pipeline? It feels to me like Github Actions design is specifically structured around re-pulling the code, re-installing deps, re-doing all the things you shouldn't be doing multiple times in a 'pipeline'.
I do understand the the commit SHA obviously ensures the code is consistent with each pull, but still, it feels to me like it's not a pipeline.
For an analogy, you wouldn't produce oil in the Middle East, and 'pipeline' it to Europe, but at every pipeline junction just pour the oil away and re-dig for new (possibly the same) oil.
