Is poetry ignoring extras or pyproject.toml is misconfigured?

Question

I have new project yolo created by poetry.

I did followed steps:

poetry new
poetry add requests
poetry add -D pytz
poetry add -D --optional --extras=dev ipdb
poetry lock

My toml file looks as follow:

[tool.poetry]
name = "yolo"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.6"
requests = "^2.24.0"

[tool.poetry.dev-dependencies]
pytest = "^5.2"
pytz = "^2020.1"
ipdb = {version = "^0.13.3", optional = true, extras = ["dev"]}

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

I removed the environment:

$ poetry env list
yolo-_0wi_Pw3-py3.6 (Activated)
$ poetry env remove yolo-_0wi_Pw3-py3.6
Deleted virtualenv: .cache/pypoetry/virtualenvs/yolo-_0wi_Pw3-py3.6

Now if I try to do:

$ poetry install
Creating virtualenv yolo-_0wi_Pw3-py3.6 in .cache/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 17 installs, 0 updates, 0 removals

  - Installing six (1.15.0)
  - Installing wcwidth (0.2.5)
  - Installing zipp (3.2.0)
  - Installing importlib-metadata (2.0.0)
  - Installing pyparsing (2.4.7)
  - Installing attrs (20.2.0)
  - Installing certifi (2020.6.20)
  - Installing chardet (3.0.4)
  - Installing idna (2.10)
  - Installing more-itertools (8.5.0)
  - Installing packaging (20.4)
  - Installing pluggy (0.13.1)
  - Installing py (1.9.0)
  - Installing urllib3 (1.25.10)
  - Installing pytest (5.4.3)
  - Installing pytz (2020.1)
  - Installing requests (2.24.0)
  - Installing yolo (0.1.0)

No ipdb, as expected.

But if I try:

$ poetry install --extras='dev'
Installing dependencies from lock file

[ValueError]
Extra [dev] is not specified.

To sort of clarify and explain more.

Toml file generated in my question is automatic work of poetry and there is a fix that requires manual intervention.

[tool.poetry]
name = "yolo"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.6"
requests = "^2.24.0"

[tool.poetry.dev-dependencies]
pytest = "^5.2"
pytz = "^2020.1"
ipdb = {version = "^0.13.3", optional = true, extras = ["dev"]}

[tool.poetry.extras]
dev = ["ipdb"]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Needless to say, it's confusing as hell.

But from now on if we do:

$ poetry install -E dev

It will work as expected and ipdb will be installed.

I will add that there is fix to extras, but it requires manual correction of pyproject.toml, so either I don't know something or is it a bug? — Drachenfels, Sep 24 '20 at 14:49

Drachenfels · Answer 1 · 2020-09-24T17:02:21.147

4

Spending some time me and my friend we figured out what is going on.

When you do:

poetry install --extra=dev ipdb

What in fact happens is that you specify you want ipdb that should use extra 'dev' that ipdb may or may not use.

Hence in toml it will be declared as:

[tool.poetry.dev-dependencies]
    ipdb = {version = "^0.13.3", extras = ["dev"]}

What in fact I want to achieve is to specify that extra for yolo project exists, it's called dev and includes installing ipdb. This is achieved by adding new section to poetry:

[tool.poetry.extras]
    dev = ["ipdb"]

The confusing factor is that both use keyword extra, while context is completely different. And main package extra is also in a different style than extra as part of the dependency definition.

edited Sep 24 '20 at 17:02

answered Sep 24 '20 at 15:09

Drachenfels

3,037
2
32
47

I kind of have to disagree with you on the last part: It's not the _context_ that's different, but your **perspective**. You're seeing the exact same keyword (for the exact same functionality), but from both sides. `extras` is used to define optional packaging addons, whether building or consuming. The `ipdb` project had a `tool.poetry.extras` section (or something equivalent) in its packaging configuration, that became your dependency line. Just like when someone uses _your_ package as a dependency, it'll show up as a dependency line like `yolo = {version = "^0.1.0", extras = ["dev"]}`. – FeRD Oct 25 '21 at 00:19
@FeRD if you describe your way then indeed, you are correct. My point was more that it's mind-boggling, year and some later and I am still unsure if what I did is correct. And I guess a lot of people will fall into this trap that adding --extra will mark the package as an extra and not that it will pass extra as an argument for that package. – Drachenfels Nov 16 '21 at 16:34

Is poetry ignoring extras or pyproject.toml is misconfigured?

1 Answers1

Linked