Installing python package from private gitlab repo in Dockerfile

Question

I'm currently trying to install python packages from a private gitlab repo. Unfortunately, I get problems with the credentials. Is there any way to install this package without writing my credentials into the Dockerfile or adding my personal ssh key into it?

Dockerfile:

FROM python:3.9.12-buster AS production

RUN apt-get update && apt-get install -y git

COPY ./requirements.txt /app/requirements.txt

RUN pip install -r /app/requirements.txt

requirements.txt:

fastapi
uvicorn
cycler~=0.10.0
networkx
python-multipart
git+https://gitlab.private.net/group/private-repo.git@commit_hash#egg=foo

Error message:

#10 3.760   Cloning https://gitlab.private.net/group/private-repo.git (to revision commit_hash) to /tmp/pip-install-q9wtmf_q/foo_commit_hash     
#10 3.769   Running command git clone --filter=blob:none --quiet https://gitlab.private.net/group/private-repo.git /tmp/pip-install-q9wtmf_q/foo_commit_hash
#10 4.039   fatal: could not read Username for 'https://gitlab.private.net/group/private-repo.git': No such device or address
#10 4.060   error: subprocess-exited-with-error

Does this answer your question? [Pip install a private repo from Gitlab with Personal Access Token on Gitlab-CI](https://stackoverflow.com/questions/64266246/pip-install-a-private-repo-from-gitlab-with-personal-access-token-on-gitlab-ci) — Michael Delgado, May 14 '22 at 20:22
If you’re worried about invoking your access token within the docker build, you can always clone the repository in the ci run and then copy the repo into the container and install it from source. — Michael Delgado, May 14 '22 at 20:27
Does this answer your question? [How to download private repo from Dockerfile with bitbucket and golang project](https://stackoverflow.com/questions/72239378/how-to-download-private-repo-from-dockerfile-with-bitbucket-and-golang-project) — The Fool, May 14 '22 at 22:48
I don't think the above questions provide an appropriate answer to the question, which will use `git+https` in a Python requirements file in the context of a docker build. — sytech, May 14 '22 at 23:03
@sytech, why not? The linked question is pretty much the same. Download dependency from private git repo. OP can easily switch to SSH, which is by now the recommended way, aynways. — The Fool, May 15 '22 at 06:09

sytech · Answer 1 · 2022-05-14T23:03:56.033

Generally speaking, you can use multi-stage docker builds to make sure your credentials don't stay in the image.

In your case, you might do something like this:

FROM python:3.9.12-buster as download
RUN apt-get update && apt-get install -y git
RUN pip install --upgrade pip wheel
ARG GIT_USERNAME
ARG GIT_PASSWORD

WORKDIR /build
COPY requirements.txt .
# add password to requirements file
RUN sed -i -E "s|gitlab.private.net|$GIT_USERNAME:$GIT_PASSWORD@gitlab.private.net|" requirements.txt

# download dependencies and build wheels to /build/dist
RUN python -m pip wheel -w /build/dist -r requirements.txt

FROM python:3.9.12-buster as production
WORKDIR /app
COPY --from=download /build/dist /wheelhouse
# install dependencies from the wheels created in previous build stage
RUN pip install --no-index /wheelhouse/*.whl

COPY . .
# ... the rest of your dockerfile

In GitLab CI, you might use the build command like this:

script:
  # ...
  - docker build --build-arg GIT_USERNAME=gitlab-ci-token --build-arg GIT_PASSWORD=$CI_JOB_TOKEN -t $CI_REGISTRY_IMAGE .

Then your image will be built and the final image won't contain your credentials. It will also be smaller since you don't have to install git :)

As a side note, you can simplify this somewhat by using the GitLab PyPI package registry.

score 0 · Answer 2 · answered Jul 12 '22 at 05:40

So I also had to install my dependencies from private package repository for my python project. This was the Dockerfile I used for building my project.

ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
RUN apt-get update &&\
    apt-get install -y binutils libproj-dev gettext gcc libpq-dev python3-dev build-essential python3-pip python3-setuptools python3-wheel python3-cffi libcairo2 libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf2.0-0 libffi-dev shared-mime-info

RUN pip config set global.extra-index-url https://<personal_access_token_name>:<personal_access_token>@gitlab.com/simple/  
# you need to configure pip to pull packages from remote private repository.
# for gitlab you require personal access token to access them with read permissions


COPY . /code/

RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt

RUN --mount=type=cache,target=/root/.cache pip install -r /code/webapi/requirements.txt

WORKDIR /code/webapi

ENTRYPOINT /code/webapi/entrypoint.sh

Installing python package from private gitlab repo in Dockerfile

2 Answers2