How do I resolve missing ARROW_PYTHON_INSTALL_DIR on MacOS?

Question

My end goal: Install autofaiss such that it can be imported as a library in python3.11 on MacOS X Ventura.

I think the main problem I'm running into is that I don't know how to debug pip3 / brew / cmake errors on MacOS -- where are the respective files/libraries supposed to live, and how do I find that out?

I'm using "brew install python" to install Python. I'm using "brew install apache-arrow" to install Arrow. There are likely many other requirements, too, which have previously been installed with pip3 or brew. (For completeness: I briefly tried installing a few things with conda, too, but stopped doing that -- who knows what files get left over by that?)

I'm trying to use "pip3 install autofaiss" but it fails because it can't find ARROW_PYTHON_INSTALL_DIR when trying to build the wheel for it using cmake. The last bit of the error log is:

      -- Arrow version: 11.0.0
      -- Found the Arrow shared library: /opt/homebrew/lib/libarrow.1100.0.0.dylib
      -- Found the Arrow import library: ARROW_IMPORT_LIB-NOTFOUND
      -- Found the Arrow static library: /opt/homebrew/lib/libarrow.a
      -- Found Arrow: /opt/homebrew/include (found version "11.0.0")
      -- Arrow version: 11.0.0 (CMake package configuration: Arrow)
      -- Arrow SO and ABI version: 1100
      -- Arrow full SO version: 1100.0.0
      -- Found the Arrow core shared library: /opt/homebrew/lib/libarrow.1100.0.0.dylib
      -- Found the Arrow core import library: ARROW_IMPORT_LIB-NOTFOUND
      -- Found the Arrow core static library: /opt/homebrew/lib/libarrow.a
      -- Could NOT find ArrowPython (missing: ArrowPython_DIR)
      CMake Error at /opt/homebrew/Cellar/cmake/3.26.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR
        ARROW_PYTHON_LIB_DIR) (found version "11.0.0")
      Call Stack (most recent call first):
        /opt/homebrew/Cellar/cmake/3.26.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
        cmake_modules/FindArrowPython.cmake:76 (find_package_handle_standard_args)
        CMakeLists.txt:229 (find_package)
      
      
      -- Configuring incomplete, errors occurred!
      error: command '/opt/homebrew/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects

I have also tried git cloning https://github.com/criteo/autofaiss and doing "make install" there. The "make install" works, but it doesn't get picked up by the Python3 installed by Homebrew. However, if I try to pip3 install -r requirements.txt from the autofaiss package, it also fails, not able to install pyarrow, with the same error.

There are three possible answers to this question I'm looking for:

What do I need to do to make pip3 install pyarrow not run into this error?
How can I make the Python that's installed by brew install python pick up the autofaiss built from the git checkout? (Or, alternatively, configure it to install itself in the location brew-python wants it)
Context information I need to debug this myself: What locations will brew python look in on MacOS? What locations will cmake look in on MacOS? How do I debug this from first principles?

How do I resolve missing ARROW_PYTHON_INSTALL_DIR on MacOS?

0 Answers0