44

Python's poetry dependency manager allows specifying optional dependencies via command:

$ poetry add --optional redis

Which results in this configuration:

[tool.poetry.dependencies]
python = "^3.8"
redis = {version="^3.4.1", optional=true}

However how do you actually install them? Docs seem to hint to:

$ poetry install -E redis

but that just throws and error:

Installing dependencies from lock file

[ValueError]
Extra [redis] is not specified.
Granitosaurus
  • 20,530
  • 5
  • 57
  • 82
  • do you also have a `[tool.poetry.extras]` section in your `pyproject.toml` file that defines the extra groups? because that is what the `-E` tag expects, not raw package names. – Arne Apr 02 '20 at 08:34
  • @Arne thanks, that's indeed the case! Such an akward specification: you need to specify package with `optional=true` under dependencies section and `group=['package']` under extras section, ugh. Could you add your comment as an answer? – Granitosaurus Apr 02 '20 at 09:26
  • Sure, I'll also try to explain why it's like this as well, even though the docs aren't very descriptive there. – Arne Apr 02 '20 at 10:44

4 Answers4

52

You need to add a tool.poetry.extras group to your pyproject.toml if you want to use the -E flag during install, as described in this section of the docs:

[tool.poetry.extras]
caching = ["redis"]

The key refers to the word that you use with poetry install -E, and the value is a list of packages that were marked as --optional when they were added. There currently is no support for making optional packages part of a specific group during their addition, so you have to maintain this section in your pyproject.toml file by hand.

The reason behind this additional layer of abstraction is that extra-installs usually refer to some optional functionality (in this case caching) that is enabled through the installation of one or more dependencies (in this case just redis). poetry simply mimics setuptools' definition of extra-installs here, which might explain why it's so sparingly documented.

Arne
  • 17,706
  • 5
  • 83
  • 99
  • 6
    For anyone getting the `Extra is not specified` error, you have to run `poetry update` first. – Sachin Raja May 15 '21 at 15:41
  • 2
    Is there a short variant so install _all_ extras? – Stefan Falk Jul 30 '21 at 14:17
  • @StefanFalk no, not unless the package maintainer explicitly creates something like an `[all]` extra. It's not even that easy to simply list all available extras: https://github.com/pypa/pip/issues/4824 – Arne Jul 31 '21 at 12:41
  • 1
    Alternatively for those getting the `Extra is not specified` error, `poetry lock --no-update` is all that is needed to update the lockfile if you don't want to update all your dependencies at the same time. – samGbos May 16 '22 at 19:56
  • 6
    as of ~v1.2, `poetry install --all-extras` will install, you guessed it, all the extras (re. @StefanFalk 's question) – abb Oct 03 '22 at 23:56
34

I will add that not only you have to have this extras section added by hand, as well your optional dependencies cannot be in dev section.

Example of code that won't work:

[tool.poetry]
name = "yolo"
version = "1.0.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "2.7"
Django = "*"

[tool.poetry.dev-dependencies]
pytest = "*"
ipdb = {version = "*", optional = true}

[tool.poetry.extras]
dev_tools = ["ipdb"]

But this WILL work:

[tool.poetry]
name = "yolo"
version = "1.0.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "2.7"
Django = "*"
ipdb = {version = "*", optional = true}

[tool.poetry.dev-dependencies]
pytest = "*"

[tool.poetry.extras]
dev_tools = ["ipdb"]
Drachenfels
  • 3,037
  • 2
  • 32
  • 47
  • Thank you! Is this documented in the `poetry` docs? I couldn't find it and was pulling my hair out trying to figure out why my extras wouldn't install until I found your answer. – blthayer Mar 09 '22 at 00:34
  • Exactly my thought, Poetry is a bit dry on the topic of extras in documentation so I spend quite some time to figure it all out. – Drachenfels Apr 01 '22 at 15:11
  • @Drachenfels I don't understand.... if you put it in `[tool.poetry.dependencies]` isn't it always going to be installed? – rjurney Jul 15 '22 at 20:45
  • As a application / library maintainer, it kind of makes sense to take "dev-dependencies can't be part of extras" as a given, because dev dependencies are never part of the _distribution_ anyway. What you are discussing are probably distinct dev-setups, where, as far as poetry is concerned, there is only a singular dev-all. – Arne Jul 18 '22 at 18:20
  • Jfc, I'm not saying you're doing it wrong, I just want to say that I can see why the poetry authors were blind to this use-case (as was I when I wrote my answer). It's an issue users run into, so it should at least be documented. – Arne Jul 18 '22 at 18:22
9

This is now possible (with Poetry version 1.2; perhaps even an earlier version), using the "extras" group:

poetry add redis --group=extras

It will appear in the section

[tool.poetry.group.extras.dependencies]

which is also newer style (compared to [tool.poetry.extras] or [tool.poetry.extras.dependencies]

See the documentation. Interestingely, this still follows the older style, [tool.poetry.extras], and doesn't show the use of poetry add, but the above result is what I get.

9769953
  • 10,344
  • 3
  • 26
  • 37
6

Up-voted Drachenfels's answer.

Dev dependency could not be optional, otherwise, no matter how you tweak it with extras or retry with poetry install -E, it will just never get installed.

This sounds like a bug but somehow by design,

...this is not something I want to add. Extras will be referenced in the distributions metadata when packaging the project but development dependencies do not which will lead to a broken extras.

— concluded in Poetry PR#606 comment by one maintainer. See here for detailed context: https://github.com/python-poetry/poetry/pull/606#issuecomment-437943927

I would say that I can accept the fact that optional dev-dependency cannot be implemented. However, at least Poetry should warn me when I have such a config. If so, I wouldn't have been confused for a long time, reading each corner of the help manual and found nothing helpful.

I found some people did get trap in this problem (Is poetry ignoring extras or pyproject.toml is misconfigured?) but their questions are closed, marked duplicated and re-linked to this question. Thus I decided to answer here and give more details about this problem.

一年又一年
  • 71
  • 1
  • 4
  • I've been caught by this as well, with pip `package[extra]` tends to install the "correct" version of extra and many packages I install use it in this manner, so I'll always use it in preference to installing `package` and `extra` separately. I guess poetry should resolve this properly but when I type `poetry add package[extra]` I expect it to treat both as required. – David Waterworth Aug 13 '23 at 23:31