12

I'm trying to install pyarrow with pip3 on OSX 11.0.1, and getting error messages.

I'm using Python 3.9 and not sure if that is the problem.

Here is the error summary:

  ERROR: Command errored out with exit status 1:
   command: /usr/local/opt/python@3.9/bin/python3.9 /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/mk/2fgx_1s96zjd1r9xzhs2ht_00000gn/T/pip-build-env-pev1z3i2/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel
       cwd: None

Here is the complete (very long) error output, I'm not sure how to read this to find the source of the problem: https://pastebin.com/RQcEuwbz

It seems like perhaps pyarrow had problems with Python 3.9, but these should now have been fixed? I'm a bit stumped about what to do next.

UPDATE: Ah, found a PR to make pyarrow work with Python 3.9 that hasn't been merged yet - so I guess the pip version of pyarrow doesn't work with Python 3.9 and that is the problem. It looks like the PR hasn't been merged because it doesn't work on Windows, but that doesn't matter to me since I'm on OSX. Is there a way I could tell pip to install pyarrow from the version in the PR?

Richard
  • 62,943
  • 126
  • 334
  • 542
  • I tried `pip3 install git+https://github.com/terencehonles/arrow/tree/arrow-102242.git@fcb561bf960f23d3c1b85c791bb2acff8e2575a6` but that gives me `ERROR: Command errored out with exit status 128`. Also I'm not sure if this is a good idea! – Richard Nov 22 '20 at 10:19

3 Answers3

11

As of today Dec. 16 2020, neither fastparquet nor pyarrow have wheels for Python 3.9

This will hopefully change soon.

Uwe L. Korn
  • 8,080
  • 1
  • 30
  • 42
Aaron Elliot
  • 155
  • 9
  • 1
    PyArrow issues for linux wheels: https://issues.apache.org/jira/browse/ARROW-10224, macOS: https://issues.apache.org/jira/browse/ARROW-10904, Windows: https://issues.apache.org/jira/browse/ARROW-10905 – Tim Swast Dec 22 '20 at 15:40
  • If you use a conda environment, you can use `pyarrow` using Python 3.9. It is just that wheels are not available. – Uwe L. Korn Dec 27 '20 at 08:31
2

Update 2021: Pyarrow has had wheels for Python3.9 since Feb/2021, Pyarrow Version 3.0.0

See Pyarrow Github

I just had to upgrade the pip version.

(Edit queue of the previous answer is full so I am posting an answer myself)

Jessica Winter
  • 55
  • 8
0

From the linked error logs one can see that the build fails due to numpy not being able to be installed. Thus you first need to get a running numpy installation before you can proceed with pyarrow.

      12 warnings and 1 error generated.
      error: Command "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/common -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/npymath -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/common -Ibuild/src.macosx-10.15-x86_64-3.9/numpy/core/src/npymath -c numpy/random/mtrand/mtrand.c -o build/temp.macosx-10.15-x86_64-3.9/numpy/random/mtrand/mtrand.o -MMD -MF build/temp.macosx-10.15-x86_64-3.9/numpy/random/mtrand/mtrand.o.d" failed with exit status 1
Uwe L. Korn
  • 8,080
  • 1
  • 30
  • 42
  • Thanks. I already have a working copy of numpy, though: `$ pip3 install numpy` produces `Requirement already satisfied: numpy in /usr/local/lib/python3.9/site-packages (1.19.4)`. – Richard Nov 22 '20 at 17:33
  • Try using `--no-use-pep517 --no-build-isolation` as flags to `pip3 install`. This should then use the `numpy` from the installed environment. – Uwe L. Korn Nov 23 '20 at 13:59
  • Thanks. Unfortunately that just gives me a different error https://pastebin.com/raw/tKz4SDN8 – Richard Dec 26 '20 at 19:08
  • 1
    There are no wheels for Python 3.9 & macOS yet, either use Python 3.8 or build from source as you are doing. In your case you are missing the C++ part of Arrow, you can get that via `brew install apache-arrow` – Uwe L. Korn Dec 27 '20 at 08:30
  • Thank you for continuing to help! I ran `brew install apache-arrow`, now I get a different error instead: `error: can't copy 'build/lib.macosx-11-x86_64-3.9/pyarrow/include/arrow': doesn't exist or not a regular file`. Full output https://pastebin.com/raw/uSmx2K1t – Richard Dec 28 '20 at 15:26