Poetry: Max retries exceeded with url in gitlab runner

Question

I have a simple .gitlab-ci.yml script that builds my python project.

image: python:3.9.6-slim-buster
variables:
    PIP_DEFAULT_TIMEOUT: 300
before_script:
    - pip install poetry==1.1.7
    - poetry config virtualenvs.create false
    - poetry install

When I run the CI pipeline, I periodically get such errors and the job is interrupted with a failure.

First type of error:

...
  • Installing toml (0.10.2)
  • Installing uvloop (0.16.0)
  • Installing watchgod (0.8.2)
  • Installing websockets (10.3)
  ConnectionError
  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/flake8-eradicate/1.2.1/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa5c5625dc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
  at /usr/local/lib/python3.9/site-packages/requests/adapters.py:565 in send
      561│             if isinstance(e.reason, _SSLError):
      562│                 # This branch is for urllib3 v1.22 and later.
      563│                 raise SSLError(e, request=request)
      564│ 
    → 565│             raise ConnectionError(e, request=request)
      566│ 
      567│         except ClosedPoolError as e:
      568│             raise ConnectionError(e, request=request)
      569│ 
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Second type of error:

...
  • Installing gitpython (3.1.27)
  OSError
  Could not find a suitable TLS CA certificate bundle, invalid path: /usr/local/lib/python3.9/site-packages/certifi/cacert.pem
  at /usr/local/lib/python3.9/site-packages/requests/adapters.py:263 in cert_verify
      259│             if not cert_loc:
      260│                 cert_loc = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH)
      261│ 
      262│             if not cert_loc or not os.path.exists(cert_loc):
    → 263│                 raise OSError(
      264│                     f"Could not find a suitable TLS CA certificate bundle, "
      265│                     f"invalid path: {cert_loc}"
      266│                 )
      267│ 
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

What is most interesting is that these errors are triggered on completely different libraries at different times. I have to do a retry many times so that it installs everything stably. What could be the problem and how to solve it?

For info: I use dockerized gitlab runner with docker executor on CentOS 7

score 1 · Answer 1 · answered Jun 16 '22 at 04:29

I think this has to do more with poetry. Could be more likely due to parallel installations.

You can either check restrict multiple installers but this will slow down the installations.

poetry config installer.max-workers=1

Had seen a discussion in poetry where they said this is less likely to happen in 1.2 but since 1.2 isn't release you can use 1.2.0b1 and check

Check this out for more details https://github.com/python-poetry/poetry/issues/3336.

score 1 · Answer 2 · answered Sep 05 '22 at 16:45

Regarding the

Could not find a suitable TLS CA certificate bundle, invalid path: .../site-packages/certifi/cacert.pem

This is probably caused because poetry install is acting in the very environment where poetry is installed, causing it to uninstall its own dependencies because they aren't listed in the lockfile. This is explicitly warned in CI recommendations in the installation guide:

https://python-poetry.org/docs/#ci-recommendations

If you install Poetry via pip, ensure you have Poetry installed into an isolated environment that is not the same as the target environment managed by Poetry. If Poetry and your project are installed into the same environment, Poetry is likely to upgrade or uninstall its own dependencies (causing hard-to-debug and understand errors).

You would notice because the logs from the install instruction start not creating a virtualenv and follow removing packages that are about to break the installation:

Installing dependencies from lock file
Package operations: 58 installs, 0 updates, 2 removals
  • Removing certifi (2022.6.15)
  • Removing setuptools (65.3.0)

Solution:

poetry config virtualenvs.create true

and be careful using poetry inside tox (tox creates venv, tox installs poetry, poetry removes certifi)

Regarding the

Max retries exceeded with url: ... Failed to establish a new connection: [Errno -2] Name or service not known

I think this deserves a totally different Q&A, but this might simply be some PyPI downtime you unluckily observed.

Poetry: Max retries exceeded with url in gitlab runner

2 Answers2