pipenv locking for NumPy fails on Python3.9 and pip21.0.1

Question

When I try to install pandas via pipenv I get a bunch of numpy errors after the locking process. When I try to install bs4 via pipenv it works fine, but for some reason, it fails with pandas. I have tried deleting and reinstalling pipenv and removing the environment, but nothing has worked.

I am using Python 3.9.1 and pip 21.0.1.
I am on macOS Big Sur.

Here is the code snippet of the errors I get. The errors occur during the locking process (writing to pipenv.lock):

  Copying numpy.egg-info to build/bdist.macosx-11-x86_64/wheel/numpy-1.20.0-py3.9.egg-info
  running install_scripts
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 181, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 211, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 197, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 513, in <module>
      setup_package()
    File "setup.py", line 505, in setup_package
      setup(**metadata)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-install-o5k4mrv9/numpy/numpy/distutils/core.py", line 169, in setup
      return old_setup(**new_attr)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 328, in run
      impl_tag, abi_tag, plat_tag = self.get_tag()
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 252, in get_tag
      plat_name = get_platform(self.bdist_dir)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 48, in get_platform
      result = calculate_macosx_platform_tag(archive_root, result)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 356, in calculate_macosx_platform_tag
      assert len(base_version) == 2
  AssertionError

  ########### EXT COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : clang

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   :
                :
    SSE41       : SSE SSE2 SSE3 SSSE3
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1
    Extra checks: none
    Detect      : SSE SSE2 SSE3 SSSE3 SSE41
                : numpy/core/src/umath/_umath_tests.dispatch.c
                :
    SSE42       : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2
    Extra checks: none
    Detect      : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
                :
    AVX2        : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mavx2
    Extra checks: none
    Detect      : AVX F16C AVX2
                : numpy/core/src/umath/_umath_tests.dispatch.c
                :
    (FMA3 AVX2) : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2
    Extra checks: none
    Detect      : AVX F16C FMA3 AVX2
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
                :
    AVX512F     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f
    Extra checks: AVX512F_REDUCE
    Detect      : AVX512F
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
                :
    AVX512_SKX  : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512bw -mavx512dq
    Extra checks: AVX512BW_MASK AVX512DQ_MASK
    Detect      : AVX512_SKX
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
  CCompilerOpt._cache_write[796] : write cache to path -> /private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-install-o5k4mrv9/numpy/build/temp.macosx-11-x86_64-3.9/ccompiler_opt_cache_ext.py

  ########### CLIB COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : clang

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt._cache_write[796] : write cache to path -> /private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-install-o5k4mrv9/numpy/build/temp.macosx-11-x86_64-3.9/ccompiler_opt_cache_clib.py
  ----------------------------------------
  ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 758, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 702, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 684, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1395, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 823, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 229, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 350, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 306, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 66, in _setup_isolation
    self.req.build_env.install_requirements(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/build_env.py", line 202, in install_requirements
    call_subprocess(args, spinner=spinner)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/kaiznanji/.local/share/virtualenvs/Kalgo-wH0XXFPH/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/notpip install --ignore-installed --no-user --prefix /private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-4nfnxp9e/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.21,<3' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy; python_version>='"'"'3.9'"'"'' Check the logs for full command output.

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
bs4 = "*"
requests = "*"
pandas = "*"
joblib = "*"

[dev-packages]

[requires]
python_version = "3.9"

Is this macOS Big Sur? And is this Python 3.9 from Homebrew? — Gino Mempin, Feb 06 '21 at 02:47
Can you [edit] to specify the *exact* command you used? (is it just `pipenv install pandas`?) Or if you are using a Pipfile, post a copy of that instead. — Gino Mempin, Feb 07 '21 at 10:03
Sorry for the late response. This seems to happen whenever i try to access any call in pipenv that locks the pipfile.lock like `pipenv lock` also. I edited with the pipfile. — cskid, Feb 17 '21 at 06:26
Part of `pipenv install` is to generate/update the Pipenv.lock file. The problem is actually with numpy, not pandas, and is probably related to this: https://github.com/numpy/numpy/issues/17569. Seems numpy 1.20.0 has installation problems. There is a newer numpy 1.20.1 which seems to install successfully. — Gino Mempin, Feb 17 '21 at 08:53
I used python 3.8 with Pipenv and it fixed it. Numpy has installation problems with python 3.9 right now. Thanks for the help! — cskid, Feb 17 '21 at 17:02
I'm seeing this problem recurring with Big Sur, python 3.9, pipenv lock, numpy, and even the same stack trace in the middle. Best I can tell, my platform string is "macosx-11-x86_64" and the code that is trying to get the version and sub-version from that string is trying to parse '11' into a major and minor version and assert that there are two parts to the version. So if it were "11.0", it appears, this would work. I'm currently trying to figure out if I can manually set my platform tag. updates if I make it work. — LisaD, Sep 29 '21 at 20:59

score 2 · Answer 1 · edited Jan 31 '22 at 21:52

2

I face this problem constantly. Sometimes this solution works, but not all the time. Run this in your terminal:

export SYSTEM_VERSION_COMPAT=1

If someone else understands why this works sometimes, please feel free to comment or edit my answer.

edited Jan 31 '22 at 21:52

Dani

556
7
23

answered Jan 30 '22 at 21:08

Austin Wolff

367
3
10

score 0 · Answer 2 · answered Sep 29 '21 at 23:27

The stacktrace from the original poster as well as mine included an error in bdist_wheel.py. Opening that exact file, it was importing 'distutils' to call 'distutils.util.get_platform()'. But when I started a python shell to see what that call returned on my machine, the active version of distutils didn't have 'util' subpackage. That's a clue that the python version I'm using is not the version being used by pipenv to build libraries.

That led me to figure out what version of python I really wanted to use and make sure pipenv was using that as well as currently installed packages:

> which python
/usr/local/Caskroom/miniforge/base/bin/python

>pipenv install --python=/usr/local/Caskroom/miniforge/base/bin/python  --site-packages

This works so far and may help some folks, however specifying the python version does not work for pipenv lock so it's probably not a complete solution.

Ultimately I probably need to go back to basics and use pyenv to properly manage the python version and keep things consistent.

score 0 · Answer 3 · answered Apr 26 '22 at 11:24

0

Running this first solved it for me: export SYSTEM_VERSION_COMPAT=1

answered Apr 26 '22 at 11:24

Christopher Broderick

439
3
10

pipenv locking for NumPy fails on Python3.9 and pip21.0.1

3 Answers3