5

I'm trying to link my code (scipy.linalg.solve notably) to a given library like OpenBLAS.

I am using python 3.7.7 in a virtual environment:

Python 3.7.7 (default, Mar 29 2020, 18:12:06) 
[Clang 11.0.3 (clang-1103.0.32.29)] on darwin

Also of note, I installed numpy and scipy into my virtual environment using pip

pip install numpy
pip install scipy


>>> scipy.__version__
'1.4.1'

>>> numpy.__version__
'1.18.0'

>>> scipy.show_config()
lapack_mkl_info:
    NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_mkl_info:
    NOT AVAILABLE
blis_info:
    NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

>>> numpy.show_config()
lapack_mkl_info:
    NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_mkl_info:
    NOT AVAILABLE
blis_info:
    NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

When I do:

$ brew install openblas
Warning: openblas 0.3.9 is already installed and up-to-date

I have OpenBLAS, and my scipy/numpy tells me that they are linked to it. However, when I do:

from numpy.distutils.system_info import get_info
info = get_info('blas_opt')
print(info)

I then get:

{'extra_compile_args': ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'], 'extra_link_args': ['-Wl,-framework', '-Wl,Accelerate'], 'define_macros': [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]}

It still tries to use the Accelerate library and not use OpenBLAS.

If I look at some links in the numpy package:

$ otool -L ~/.virtualenvs/myvenv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so 

/Users/myself/.virtualenvs/myvenv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so:
@loader_path/../.dylibs/libopenblasp-r0.3.7.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)

There it does tell me it links to OpenBLAS.

So, anyone knows what's going on here? What step do I need to take in order to use the OpenBLAS?

BryanH
  • 5,826
  • 3
  • 34
  • 47
William Abma
  • 415
  • 3
  • 14
  • Why, exactly, are you doing this? What python distribution are you using? – Igor Rivin Jun 08 '20 at 23:31
  • I'm solving some system of equations in scipy as part of a larger software. I need to use the DGESV method to do so, and the performance is suspiciously low compared to what I was expecting based on C calculations on a different machine. I'm using python 3.7.7 (sorry, forgot to mention that, I'll update my question with that information) – William Abma Jun 08 '20 at 23:39
  • Actually, my question was about the distribution - you should be using anaconda, which comes with all the mkl good stuff. – Igor Rivin Jun 09 '20 at 01:42
  • I would very very much like to avoid anaconda for various reasons, but if it's the only way to efficiently get a MKL linking done properly, i'll definitely look into that. Right now I'm using virtualenvs and pip or install from source when necessary – William Abma Jun 09 '20 at 02:11
  • In that case, you are on your own. Anaconda's primary goal from the beginning was to make a high performance distribution for numerical computation, so if you want to not use that, feel free to waste a couple of months of your time :( – Igor Rivin Jun 09 '20 at 02:14
  • I mean, a couple years ago maybe, but now, conda doesn't have much going for it versus pip honestly. I've never had any dependency issue, use virtual environments that are easy to maintain without having an additional layer I don't need. Now, if anaconda is the only way to make what I need happen, I will happily try it out again, but that means that a bug issue is due in the pip project or scipy/numpy. – William Abma Jun 09 '20 at 02:23
  • I will add that i likely have a wrong idea about what Anaconda can do if used the right way, i.e. as you're saying. Thanks for your suggestion, i'll definitely try it out to test MKL. – William Abma Jun 09 '20 at 02:30

0 Answers0