Testing a 2.7-friendly Python package with one 3.x feature

Question

I've got an intellectual quandary that I would appreciate help with. The first two sections below give some preamble; the third contains the questions. I'm open to answers to my original question and suggestions on how to do it better/cleaner/etc.

The Setup

I've developed an HTTP-based library (using Requests) that works well in all versions of Python starting at 2.7. Recently, it was requested that the library support asynchronous HTTP requests via aiohttp, as well – as you're undoubtedly aware, this library requires Python >= 3.4.2.

I could just rip out Requests and replace it with aiohttp, but I have a non-trivial percentage of my user base who need the library to work in Python 2.x. So, I decided to use both libraries in the appropriate setting.

Envision a Client object that does all of the HTTP transactions:

import mylib

client = mylib.Client()
data = client.get()
client.update_setting(<setting_id>)
# etc.

After consideration, I think I want to do something like this:

client = mylib.Client(asynchronous=True)

...at which point Client's methods will now use the asynchronous, aiottp-driven variants. If the user attempts to instantiate a Client in this way on Python < 3.4.2, a warning is logged and the library falls back to using a synchronous Client.

Publishing

I've configured my setup.py to handle this: Python versions >= 3.4.2 will install aiohttp (and its dependencies), while versions < 3.4.2 will not:

import sys

import setuptools

BASE_ASYNC_PYTHON_VER = int(hex(0x030402f0), 16)
PACKAGES = ['mylib']
REQUIRED = ['requests']
EXTRAS = {}

# Handles environments with old versions of setuptools:
if int(setuptools.__version__.split(".", 1)[0]) < 18:
    if sys.hexversion < BASE_ASYNC_PYTHON_VER:
        REQUIRED.append('aiodns')
        REQUIRED.append('aiohttp')
        REQUIRED.append('cchardet')
    else:
        EXTRAS[":python_version>='3.4.2'"] = ['aiodns', 'aiohttp', 'cchardet']

# Removing extraneous stuff for this example:
setuptools.setup(
    name='mylib',
    version='1.0.0'
    description='Just for grims',
    packages=PACKAGES,
    install_requires=REQUIRED,
    extras_require=EXTRAS,

This works beautifully: Python 2 installations eschew those extra libraries and Python 3 installations include them.

The problem comes with...

Testing

I use pipenv to manage my dependencies and virtualenv while developing. My Pipfile looks like this:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
detox = "*"
pytest = "*"
requests-mock = "*"
tox = "*"
twine = "*"

[packages]
aiodns = "*"
aiohttp = "*"
cchardet = "*"
requests = "*"

Note that there doesn't appear to be a way to say, "Only install a package for version x.y.z of Python."

I also use tox to run my tests across multiple Python versions; my tox.ini looks like this:

[tox]
envlist = py27, py36

[testenv]
passenv=HOME
deps = pipenv
commands=
    pipenv install --dev
    pipenv run py.test tests

(where tests contains a bunch of pytest-friendly tests)

Unfortunately, this chokes: for both py27 and py36, tox attempts to install all of the packages (dev and "regular") from my Pipfile; when py27 tries to install aiohttp, it obviously chokes.

So, the question is: how can I adequately (and Pythonically) test both versions of Python with this logic and structure in place?

anthony sottile · Answer 1 · 2017-08-27T19:12:02.287

I did some investigation into this, but didn't come to a complete answer -- leaving the useful information I found while searching.

pipfile should probably support dependency markers. There's an example in the "let's use toml" issue on pipfile and it looks something like this:

# environment markers
SomeProject2 = {version = "==5.4", markers = {python_version = "< 2.7", sys_platform = "win32"}}

While attempting to get this to work with the latest version of toml, pipfile, pipenv I ran into the following problem with the python implementation of the toml parser: https://github.com/uiri/toml/issues/118

On another note, you generally shouldn't use branching logic in your setup.py to choose dependencies as these will get baked into a wheel built from your package. I've written up slides about why this is a problem if you'd like more information about that. The tl;dr for how to make that work gracefully is again with environment specifiers:

setup(
    ...
    extras_require={
        ':python_version=="2.7"': ['functools32'],
    },
)

Testing a 2.7-friendly Python package with one 3.x feature

The Setup

Publishing

Testing

1 Answers1