Previously we've used our internal pip repository for source distributions only. Moving forward we want to host wheels as well to accomplish two things:
- serve our own code to both (local) developer machines and Alpine Docker environments
- create wheels for packages that don't have Alpine wheels
Unfortunately the wheels built with different libraries share the same artifact name and the second one gets rejected by the pip repository:
docker-compose.yml
version: '3'
services:
build-alpine:
build: alpine
image: build-alpine-wheels
volumes:
- $PWD/cython:/build
working_dir: /build
command: sh -c 'python setup.py bdist_wheel && twine upload --repository-url http://pypi:8080 -u admin -p admin dist/*'
build-debian:
build: debian
image: build-debian-wheels
volumes:
- $PWD/cython-debian:/build
working_dir: /build
command: bash -c 'sleep 10s && python setup.py bdist_wheel && twine upload --repository-url http://pypi:8080 -u admin -p admin dist/*'
pypi:
image: stevearc/pypicloud:1.0.2
volumes:
- $PWD/pypi:/etc/pypicloud/
alpine-test:
image: build-alpine-wheels
depends_on:
- build-alpine
command: sh -c 'while ping -c1 build-alpine &>/dev/null; do sleep 1; done; echo "build container finished" && pip install -i http://pypi:8080/pypi --trusted-host pypi cython && cython --version'
debian-test:
image: python:3.6
depends_on:
- build-debian
command: bash -c 'while ping -c1 build-debian &>/dev/null; do sleep 1; done; echo "build container finished" && pip install -i http://pypi:8080/pypi --trusted-host pypi cython && cython --version'
alpine/Dockerfile
FROM python:3.6-alpine
RUN apk add --update --no-cache build-base
RUN pip install --upgrade pip
RUN pip install twine
debian/Dockerfile
FROM python:3.6-slim
RUN apt-get update && apt-get install -y \
build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip
RUN pip install twine
pypi/config.ini
[app:main]
use = egg:pypicloud
pyramid.reload_templates = False
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pypi.default_read =
everyone
pypi.default_write =
everyone
pypi.storage = file
storage.dir = %(here)s/packages
db.url = sqlite:///%(here)s/db.sqlite
auth.admins =
admin
user.admin = $6$rounds=535000$sFuRqMc5PbRccW1J$OBCsn8szlBwr4yPP243JPqomapgInRCUavv/p/UErt7I5FG4O6IGSHkH6H7ZPlrMXO1I8p5LYCQQxthgWZtxe1
# For beaker
session.encrypt_key = s0ETvuGG9Z8c6lK23Asxse4QyuVCsI2/NvGiNvvYl8E=
session.validate_key = fJvHQieaa0g3XsdgMF5ypE4pUf2tPpkbjueLQAAHN/k=
session.secure = False
session.invalidate_corrupt = true
###
# wsgi server configuration
###
[uwsgi]
paste = config:%p
paste-logger = %p
master = true
processes = 20
reload-mercy = 15
worker-reload-mercy = 15
max-requests = 1000
enable-threads = true
http = 0.0.0.0:8080
virtualenv = /env
###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html
###
[loggers]
keys = root, botocore, pypicloud
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = INFO
handlers = console
[logger_pypicloud]
level = DEBUG
qualname = pypicloud
handlers =
[logger_botocore]
level = WARN
qualname = botocore
handlers =
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)s %(asctime)s [%(name)s] %(message)s
Setup and execution
git clone https://github.com/cython/cython
git clone https://github.com/cython/cython cython-debian
docker-compose build
docker-compose up
At the end I would like both test containers to be able to execute cython --version. Which works for the Alpine container:
alpine-test_1 | Collecting cython
alpine-test_1 | Downloading http://pypi:8080/api/package/cython/Cython-0.29.12-cp36-cp36m-linux_x86_64.whl (5.0MB)
alpine-test_1 | Installing collected packages: cython
alpine-test_1 | Successfully installed cython-0.29.12
alpine-test_1 | Cython version 0.29.12
But doesn't work for the Debian container:
debian-test_1 | Downloading http://pypi:8080/api/package/cython/Cython-0.29.12-cp36-cp36m-linux_x86_64.whl (5.0MB)
debian-test_1 | Installing collected packages: cython
debian-test_1 | Successfully installed cython-0.29.12
debian-test_1 | Traceback (most recent call last):
debian-test_1 | File "/usr/local/bin/cython", line 6, in <module>
debian-test_1 | from Cython.Compiler.Main import setuptools_main
debian-test_1 | File "/usr/local/lib/python3.6/site-packages/Cython/Compiler/Main.py", line 28, in <module>
debian-test_1 | from .Scanning import PyrexScanner, FileSourceDescriptor
debian-test_1 | ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
I find it particularly curious that both environments try to pull this wheel because there are all sorts of packages which don't work with Alpine (e.g. Pandas) in which case pip goes straight for the source distribution. I suppose I must be doing something wrong in that regard as well.
So now I'm wondering how I can create these wheels such that for each version of the software package two different wheels can live in the pip repository and have pip automatically download and install the correct one.
