42

I've recently found poetry to manage dependencies. In one project, we use PyTorch. How do I add this to poetry?

We are working on machines that have no access to a CUDA GPU (for simple on the road inferencing/testing) and workstations where we do have access to CUDA GPUs. Is it possible to use poetry to ensure every dev is using the same PyTorch version?

There seems to be no obvious way to decide which PyTorch version to install. I thought about adding the different installation instructions as extra dependencies, but I failed to find an option to get the equivalent settings like:

pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html

I would be fine with setting the total path to the different online wheels, like: https://download.pytorch.org/whl/torch_stable.html/cpu/torch-1.3.1%2Bcpu-cp36-cp36m-win_amd64.whl

But I would rather not but them in git directly... The closest option I've seen in poetry is either downloading them manually and then using file = X command.

Seanny123
  • 8,776
  • 13
  • 68
  • 124
kai
  • 523
  • 1
  • 4
  • 7
  • 2
    I wonder how it'll work if with poetry if we're using the nightly version of pytorch. – alvas Oct 31 '22 at 18:02

6 Answers6

26

Currently, Poetry doesn't have a -f option (there's an open issue and an open PR), so you can't use the pip instructions. You can install the .whl files directly:

poetry add https://download.pytorch.org/whl/torch_stable.html/cpu/torch-1.3.1%2Bcpu-cp36-cp36m-win_amd64.whl

or add the dependency directly to your .toml file:

[tool.poetry.dependencies]
torch = { url = "https://download.pytorch.org/whl/torch_stable.html/cpu/torch-1.3.1%2Bcpu-cp36-cp36m-win_amd64.whl" }
GilZ
  • 6,418
  • 5
  • 30
  • 40
  • 9
    This is not a satisfying answer as this does not give any solution to the problem being: depending on the platform, point to the correct wheel. – ClementWalter Mar 03 '21 at 13:36
  • 2
    Note: This won't work if you have dependencies that rely on pytorch, since poetry fails to resolve the dependency versions (https://github.com/python-poetry/poetry/issues/4231#issue-931746833) – Jay Mody May 02 '22 at 17:16
18

Since poetry 1.2, you can do this:

poetry source add -p explicit pytorch https://download.pytorch.org/whl/cpu
poetry add --source pytorch torch torchvision

and it will install from the specified index-url.

(and this also works with https://download.pytorch.org/whl/cu118 )

tsvikas
  • 16,004
  • 1
  • 22
  • 12
  • This is the most appropriate answer at the time of writing: it exactly incorporates PyTorch's current official index-url for CPU-only installation and does not require any tools other than poetry itself. Thanks! – Shengdi Chen Jun 08 '23 at 23:00
  • This worked great! Thanks! – neonwatty Aug 04 '23 at 15:43
16

After spending a couple of hours on this issue, I found a "solution" by combining Poetry and pip just for PyTorch. You don't need to specify the wheel URLs directly and thus remain cross-platform.

I'm using Poe The Poet, a nice task runner for Poetry that allows to run any arbitrary command.

[tool.poetry.dev-dependencies]
poethepoet = "^0.10.0"

[tool.poe.tasks]
force-cuda11 = "python -m pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html"

You can run:

poetry install

and then:

poe force-cuda11  # relies on pip and use PyTorch wheels repo
Kirell
  • 9,228
  • 4
  • 46
  • 61
12

An updated solution from this issue in the Poetry github:

poetry add torch --platform linux --python "^3.7"
GilZ
  • 6,418
  • 5
  • 30
  • 40
Antiez
  • 679
  • 7
  • 11
  • 1
    Good link to the issue. Note that this other related issue https://github.com/python-poetry/poetry/issues/1616 is still open, so this is still an issue and no stable solution is currently available – ClementWalter Mar 03 '21 at 13:37
  • This _sort of_ worked for me but I bumped into other (I suspect related) dependency issues further down the track. The [Poe The Poet](https://stackoverflow.com/a/66644457/7499546) solution suggested by @Kirell solved my issue though :-) – Joshua Patterson Oct 21 '21 at 01:11
  • 3
    How do you specify the cuda version? I'm assuming that command will default to cpu. – David Waterworth Nov 23 '21 at 22:30
11

In late 2021, utilizing markers and multiple constraints should work.

$ poetry --version
Poetry version 1.1.11

# pyproject.toml
[tool.poetry.dependencies]
python = "~3.9"
torch = [
  {url = "https://download.pytorch.org/whl/cpu/torch-1.10.0%2Bcpu-cp39-cp39-linux_x86_64.whl", markers = "sys_platform == 'linux'"},
  {url = "https://download.pytorch.org/whl/cpu/torch-1.10.0%2Bcpu-cp39-cp39-win_amd64.whl", markers = "sys_platform == 'win32'", }
]
numpy = "^1.21.4"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

$ poetry install
The currently activated Python version 3.8.12 is not supported by the project (~3.9).
Trying to find and use a compatible version. 
Using python3.9 (3.9.9)
Creating virtualenv machine-learning in /home/redqueen/machine_learning/.venv
Updating dependencies
Resolving dependencies... (36.0s)

Writing lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing typing-extensions (4.0.1)
  • Installing numpy (1.21.4)
  • Installing torch (1.10.0+cpu https://download.pytorch.org/whl/cpu/torch-1.10.0%2Bcpu-cp39-cp39-linux_x86_64.whl)

NOTE: Numpy has to be listed. Otherwise you'll get an import error.

Without numpy:

$ python
Python 3.9.9 (main, Nov 23 2021, 00:34:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
/home/redqueen/machine_learning/.venv/lib/python3.9/site-packages/torch/package/_directory_reader.py:17: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:68.)
  _dtype_to_storage = {data_type(0).dtype: data_type for data_type in _storages}
>>> quit()

With numpy:

$ python
Python 3.9.9 (main, Nov 23 2021, 00:34:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False
>>> quit()

Reference:

https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies

Disclaimer

I do not have a Windows (or Mac) to test this on.

DataMinion
  • 380
  • 3
  • 10
  • 1
    When I use this approach, Poetry will still download torch from this url every time I run `poetry lock`. Is there any way to avoid this, to your knowledge? – roshambo Jan 23 '23 at 16:09
  • @roshambo I believe that's a bug with an open issue. See [here](https://github.com/python-poetry/poetry/issues/2415) or [here](https://github.com/python-poetry/poetry/issues/5902) – DataMinion Jan 29 '23 at 06:35
  • Thanks. I'm able to "get around" this by running `poetry lock --up-update` – roshambo Jan 31 '23 at 21:03
4

There is a fork that I am maintaining called relaxed-poetry It is a very young fork but it supports what you want with the following configuration:


# pyproject.toml

[tool.poetry.dependencies]
python = "^3.8"
torch = { version = "=1.90+cu111", source = "pytorch" }

[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu111/"
secondary = true

Check it if you like, it can be installed side by side with poetry.

bennyl
  • 2,886
  • 2
  • 29
  • 43
  • 1
    This maybe a game changer for ML. Very difficult to use Poetry with pytorch currently. – Kirell Nov 24 '21 at 12:50
  • Adding the poetry source lets me install the right version of pytorch but then I can't install anything else (e.g. black) because poetry searches only in download.pytorch.org – mic Aug 07 '22 at 19:57
  • With poetry 1.5.0 it works with `priority = "supplemental"` instead of `secondary` https://python-poetry.org/docs/repositories/#supplemental-package-sources – Michael B. May 29 '23 at 01:10