9

I am building a Python package using pyproject and poetry. My pyproject.toml looks like this:

[tool.poetry]
authors = ["test"]
description = ""
name = "test"
version = "0.1.0"

[tool.poetry.dependencies]
spacy = {extras = ["cuda113"], version = "^3.2.3"}
faiss-gpu = {version = "1.7.2", optional = true}

[tool.poetry.extras]
gpu = ["faiss-gpu"]

This successfully installs faiss-gpu as an extra using poetry install -E gpu.

However, I would like to install spacy[cuda113] (GPU version) only when poetry install -E gpu is provided. A normal poetry install should only install spacy (CPU version).

I've tried using the following configuration, but this makes all of spacy optional and doesn't install it. Only spacy[cuda113] (GPU version) must be optional.

[tool.poetry]
authors = ["test"]
description = ""
name = "test"
version = "0.1.0"

[tool.poetry.dependencies]
spacy = {extras = ["cuda113"], version = "^3.2.3", optional = true}
faiss-gpu = {version = "1.7.2", optional = true}

[tool.poetry.extras]
gpu = ["faiss-gpu", "spacy"]

Is there a way to make spacy[cuda113] optional but spacy as a required dependency?

azizbro
  • 3,069
  • 4
  • 22
  • 36

1 Answers1

1

Try like this:

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
authors = ["test"]
description = ""
name = "test"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.10"
faiss-gpu = {version = "1.7.2", optional = true}
spacy = {extras = ["cuda113"], version = "^3.2.3", optional = true}
Spacy = "^3.2.3"

[tool.poetry.extras]
gpu = ["faiss-gpu", "spacy"]

It generates the dist-info METADATA lines like this, which looks correct in terms of how pip will install test vs test[gpu]:

Metadata-Version: 2.1
Name: test
Version: 0.1.0
Summary: 
Author: test
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Provides-Extra: gpu
Requires-Dist: Spacy (>=3.2.3,<4.0.0)
Requires-Dist: faiss-gpu (==1.7.2); extra == "gpu"
Requires-Dist: spacy[cuda113] (>=3.2.3,<4.0.0); extra == "gpu"

Disclaimer: I think I am relying on bugs and/or implementation detail in poetry for this to work. Note the case sensitivity difference between "spacy" and "Spacy" in the dependency specification. The ordering of the lines is also crucial.

wim
  • 338,267
  • 99
  • 616
  • 750
  • Hi @wim. Thanks for answering but isn't quite the solution. The problem with your solution is that both "Spacy" and "spacy[cuda113]" are installed as extra gpu dependencies. As stated in my question `A normal poetry install should only install spacy (CPU version).` So I want the base spacy library to not be an extra and always be installed. – azizbro May 02 '22 at 05:14
  • could you provide some documentation on what happens when you specify a library using capital letters e.g. "Spacy" ? – azizbro May 02 '22 at 05:46
  • @azizbro I think you have some misunderstanding about how extras work - there is no difference between installing both "Spacy" and "spacy[cuda113]" vs installing only "spacy[cuda113]". Extras can only _add_ dependencies. In this case it adds the dependency for [cupy-cuda113](https://pypi.org/project/cupy-cuda113/). The capital letter is only there to trick the TOML loader into considering Spacy and spacy as unique keys - otherwise the file in the format that poetry expects is not valid TOML due to duplicate keys in a mapping. – wim May 02 '22 at 18:08
  • @vim I very much appreciate the effort in the answer, but with your settings, if I run `poetry install -E gpu`, only the normal spacy version gets installed, so it doesn't meet the requirements – azizbro May 16 '22 at 07:09
  • 2
    @azizbro **It does work**. I've tested and verified this as working on two different machines with different poetry versions now (Python 3.10 + poetry 1.1.13, and Python 3.8 + poetry 1.1.11). See https://replit.com/@wimglenn/ExampleSpacyGPU#pyproject.toml - created a fresh environment, removed the lockfile and then ran `poetry install -E gpu` which pulls in `cupy-cuda113` (check the pip freeze in the shell tab). The only way `cupy-cuda113` can be installed is via the gpu extra of spacy. – wim May 16 '22 at 18:15
  • @vim thanks for providing the replit. It does seem to work, so thank you! As the answer relies on implementation details that are not made explicit by poetry, I'll wait until they have a proper feature for this. – azizbro May 22 '22 at 23:43