failed - pip install delta-lake-reader[azure]

Question

We are building an api in FastAPI framework and need to access delta lake storage from python.

I am going through the article https://pypi.org/project/delta-lake-reader/

to install the the package but it is failing with below error:

I am using Visual Studio Code for writing the code and have created virtual environment. Working on python 3.11.3

User's image

Error:

Please guide me solve this issue.

(.venv) PS C:\Users\vsahu015\deltalake\OI_Blob_Manager> pip install delta-lake-reader[azure]
Collecting delta-lake-reader[azure]
  Using cached delta_lake_reader-0.2.13-py3-none-any.whl (12 kB)
Collecting adlfs<0.8.0,>=0.7.5 (from delta-lake-reader[azure])
  Using cached adlfs-0.7.7.tar.gz (37 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting aiohttp<4.0.0,>=3.7.3 (from delta-lake-reader[azure])
  Using cached aiohttp-3.8.4-cp311-cp311-win_amd64.whl (317 kB)
Collecting fsspec<2022.0,>=2021.5 (from delta-lake-reader[azure])
  Using cached fsspec-2021.11.1-py3-none-any.whl (132 kB)
Collecting pandas<2.0.0,>=1.3.1 (from delta-lake-reader[azure])
  Using cached pandas-1.5.3-cp311-cp311-win_amd64.whl (10.3 MB)
Collecting pip<22.0.0,>=21.0.1 (from delta-lake-reader[azure])
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Collecting pyarrow<6.0.0,>=5.0.0 (from delta-lake-reader[azure])
  Using cached pyarrow-5.0.0.tar.gz (739 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [222 lines of output]
      Ignoring numpy: markers 'python_version < "3.9"' don't match your environment
      Collecting cython>=0.29
        Using cached Cython-0.29.35-py2.py3-none-any.whl (988 kB)
      Collecting numpy==1.19.4
        Using cached numpy-1.19.4.zip (7.3 MB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error

        Preparing metadata (pyproject.toml) did not run successfully.
        exit code: 1

        [194 lines of output]
        setup.py:67: RuntimeWarning: NumPy 1.19.4 may not yet support Python 3.11.
          warnings.warn(
        Running from numpy source directory.
        setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
          run_build = parse_setuppy_commands()
        Processing numpy/random\_bounded_integers.pxd.in
        Processing numpy/random\bit_generator.pyx
        Processing numpy/random\mtrand.pyx
        Processing numpy/random\_bounded_integers.pyx.in
        Processing numpy/random\_common.pyx
        Processing numpy/random\_generator.pyx
        Processing numpy/random\_mt19937.pyx
        Processing numpy/random\_pcg64.pyx
        Processing numpy/random\_philox.pyx
        Processing numpy/random\_sfc64.pyx
        Cythonizing sources
        blas_opt_info:
        blas_mkl_info:
        No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
        customize MSVCCompiler
          libraries mkl_rt not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']       
          NOT AVAILABLE

        blis_info:
          libraries blis not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']
          NOT AVAILABLE

        openblas_info:
          libraries openblas not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']     
        get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
        customize GnuFCompiler
        Could not locate executable g77
        Could not locate executable f77
        customize IntelVisualFCompiler
        Could not locate executable ifort
        Could not locate executable ifl
        customize AbsoftFCompiler
        Could not locate executable f90
        customize CompaqVisualFCompiler
        Could not locate executable DF
        customize IntelItaniumVisualFCompiler
        Could not locate executable efl
        customize Gnu95FCompiler
        Could not locate executable gfortran
        Could not locate executable f95
        customize G95FCompiler
        Could not locate executable g95
        customize IntelEM64VisualFCompiler
        customize IntelEM64TFCompiler
        Could not locate executable efort
        Could not locate executable efc
        customize PGroupFlangCompiler
        Could not locate executable flang
        don't know how to compile Fortran code on platform 'nt'
          NOT AVAILABLE

        atlas_3_10_blas_threads_info:
        Setting PTATLAS=ATLAS
          libraries tatlas not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']       
          NOT AVAILABLE

        atlas_3_10_blas_info:
          libraries satlas not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']       
          NOT AVAILABLE

        atlas_blas_threads_info:
        Setting PTATLAS=ATLAS
          libraries ptf77blas,ptcblas,atlas not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']
          NOT AVAILABLE

        atlas_blas_info:
          libraries f77blas,cblas,atlas not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']
          NOT AVAILABLE

        accelerate_info:
          NOT AVAILABLE

        C:\Users\vsahu015\AppData\Local\Temp\pip-install-qvpa_09h\numpy_d10a25859e50498fb6ca81231b1a5850\numpy\distutils\system_info.py:1914: UserWarning:
            Optimized (vendor) Blas libraries are not found.
            Falls back to netlib Blas library which has worse performance.
            A better performance should be easily gained by switching
            Blas library.
          if self._calc_info(blas):
        blas_info:
          libraries blas not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']
          NOT AVAILABLE

        C:\Users\vsahu015\AppData\Local\Temp\pip-install-qvpa_09h\numpy_d10a25859e50498fb6ca81231b1a5850\numpy\distutils\system_info.py:1914: UserWarning:
            Blas (http://www.netlib.org/blas/) libraries not found.
            Directories to search for the libraries can be specified in the
            numpy/distutils/site.cfg file (section [blas]) or by setting
            the BLAS environment variable.
          if self._calc_info(blas):
        blas_src_info:
          NOT AVAILABLE

        C:\Users\vsahu015\AppData\Local\Temp\pip-install-qvpa_09h\numpy_d10a25859e50498fb6ca81231b1a5850\numpy\distutils\system_info.py:1914: UserWarning:
            Blas (http://www.netlib.org/blas/) sources not found.
            Directories to search for the sources can be specified in the
            numpy/distutils/site.cfg file (section [blas_src]) or by setting
            the BLAS_SRC environment variable.
          if self._calc_info(blas):
          NOT AVAILABLE

        non-existing path in 'numpy\\distutils': 'site.cfg'
        lapack_opt_info:
        lapack_mkl_info:
          libraries mkl_rt not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']       
          NOT AVAILABLE

        openblas_lapack_info:
          libraries openblas not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']     
          NOT AVAILABLE

        openblas_clapack_info:
          libraries openblas,lapack not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']
          NOT AVAILABLE

        flame_info:
          libraries flame not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']        
          NOT AVAILABLE

        atlas_3_10_threads_info:
        Setting PTATLAS=ATLAS
          libraries lapack_atlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib
          libraries tatlas,tatlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib
          libraries lapack_atlas not found in C:\
          libraries tatlas,tatlas not found in C:\
        <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
          NOT AVAILABLE

        atlas_3_10_info:
          libraries lapack_atlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib
          libraries satlas,satlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib
          libraries lapack_atlas not found in C:\
          libraries satlas,satlas not found in C:\
        <class 'numpy.distutils.system_info.atlas_3_10_info'>
          NOT AVAILABLE

        atlas_threads_info:
        Setting PTATLAS=ATLAS
          libraries lapack_atlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib
          libraries ptf77blas,ptcblas,atlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib        
          libraries lapack_atlas not found in C:\
          libraries ptf77blas,ptcblas,atlas not found in C:\
        <class 'numpy.distutils.system_info.atlas_threads_info'>
          NOT AVAILABLE

        atlas_info:
          libraries lapack_atlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib
          libraries f77blas,cblas,atlas not found in c:\Users\vsahu015\deltalake\OI_Blob_Manager\.venv\lib
          libraries lapack_atlas not found in C:\
          libraries f77blas,cblas,atlas not found in C:\
        <class 'numpy.distutils.system_info.atlas_info'>
          NOT AVAILABLE

        lapack_info:
          libraries lapack not found in ['c:\\Users\\vsahu015\\deltalake\\OI_Blob_Manager\\.venv\\lib', 'C:\\']       
          NOT AVAILABLE

        C:\Users\vsahu015\AppData\Local\Temp\pip-install-qvpa_09h\numpy_d10a25859e50498fb6ca81231b1a5850\numpy\distutils\system_info.py:1748: UserWarning:
            Lapack (http://www.netlib.org/lapack/) libraries not found.
            Directories to search for the libraries can be specified in the
            numpy/distutils/site.cfg file (section [lapack]) or by setting
            the LAPACK environment variable.
          return getattr(self, '_calc_info_{}'.format(name))()
        lapack_src_info:
          NOT AVAILABLE

        C:\Users\vsahu015\AppData\Local\Temp\pip-install-qvpa_09h\numpy_d10a25859e50498fb6ca81231b1a5850\numpy\distutils\system_info.py:1748: UserWarning:
            Lapack (http://www.netlib.org/lapack/) sources not found.
            Directories to search for the sources can be specified in the
            numpy/distutils/site.cfg file (section [lapack_src]) or by setting
            the LAPACK_SRC environment variable.
          return getattr(self, '_calc_info_{}'.format(name))()
          NOT AVAILABLE

        numpy_linalg_lapack_lite:
          FOUND:
            language = c
            define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]

        C:\Users\vsahu015\AppData\Local\Temp\pip-build-env-9s19zcbj\overlay\Lib\site-packages\setuptools\_distutils\dist.py:275: UserWarning: Unknown distribution option: 'define_macros'
          warnings.warn(msg)
        running dist_info
        running build_src
        build_src
        building py_modules sources
        creating build
        creating build\src.win-amd64-3.11
        creating build\src.win-amd64-3.11\numpy
        creating build\src.win-amd64-3.11\numpy\distutils
        building library "npymath" sources
        error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
        [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      Encountered error while generating package metadata.

      See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

score 0 · Accepted Answer · answered Jun 19 '23 at 07:02

I tried installing

pip install delta-lake-reader[azure]

with Python 3.11 and Python 3.10 but received the same error code as yours:-

enter image description here

This is because, pyarrow module that has the dependency on numpy==1.21.2. Numpy 1.21.2 version that has pyarrow does not work with Python 3.10 or above. Refer here.

Thus I downgraded my Python version to 3.9 and installed Python 3.9 version and tried installing pip install delta-lake-reader[azure] and it got installed successfully, Refer below:-

I used --user flag to avoid any administration related issues while pip installing the module:-

 pip install delta-lake-reader[azure] --user

enter image description here

Now, I switched my Terminal to Python 3.11 and the above module got installed as requirements satisfied:-

enter image description here

Thank you much. This really helps. I also later realized that It will only work on 3, 3.8 and 3.9 but your provided additional information is quite helpful and will try to check that as well. — VIJAY SAHU, Jun 21 '23 at 12:35

failed - pip install delta-lake-reader[azure]

1 Answers1