0

Currently I am attempting to build Numpy-1.9.1for Intel's MKL using the Intel compilers on CentOS 7. I have Intel Parallel XE Studio 2015 C++ and Fortran for Linux installed, and in my terminal I can use both 'icc' and 'ifort' command, they are both found without issue. I have also run:

$ source /opt/intel/composer_xe_2015/bin/compilervars.sh intel64

In accordance with this guide from Intel's webpage for doing exactly what I am trying to do: https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl, I have attempted to build numpy using this command:

$ sudo python setup.py config --compiler=intelem build_clib --compiler=intelem build_ext --compiler=intelem install

The resulting message is:

Running from numpy source directory.
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_suite'
  warnings.warn(msg)
non-existing path in 'numpy/f2py': 'docs'
non-existing path in 'numpy/f2py': 'f2py.1'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

non-existing path in 'numpy/lib': 'benchmarks'
lapack_opt_info:
openblas_lapack_info:
  libraries openblas not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

lapack_mkl_info:
mkl_info:
  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running config
running build_clib
running build_src
build_src
building py_modules sources
building library "npymath" sources
Could not locate executable icc
Could not locate executable ecc
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config
C compiler: icc -O3 -g -fPIC -fp-model strict -fomit-frame-pointer -openmp -xhost

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.7 -c'
icc: _configtest.c
sh: icc: command not found
sh: icc: command not found
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
  File "setup.py", line 251, in <module>
    setup_package()
  File "setup.py", line 243, in setup_package
    setup(**metadata)
  File "/home/myles/Downloads/numpy-1.9.1/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/myles/Downloads/numpy-1.9.1/numpy/distutils/command/build_clib.py", line 63, in run
    self.run_command('build_src')
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/myles/Downloads/numpy-1.9.1/numpy/distutils/command/build_src.py", line 153, in run
    self.build_sources()
  File "/home/myles/Downloads/numpy-1.9.1/numpy/distutils/command/build_src.py", line 164, in build_sources
    self.build_library_sources(*libname_info)
  File "/home/myles/Downloads/numpy-1.9.1/numpy/distutils/command/build_src.py", line 299, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/home/myles/Downloads/numpy-1.9.1/numpy/distutils/command/build_src.py", line 386, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 686, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

I am using CentOS 7 as my operating system for this. It appears that for some reason the build script cannot find the icc.

I need numpy build with the Intel compilers because from what I have gathered, I need numpy build with the Intel compilers and Intel MKL in order for automatic offloading to a Xeon Phi to be available. If anyone has built numpy for automatic offloading to a Xeon Phi, or just built numpy with the Intel compilers in this matter I would appreciate the input on fixing this error.

mgallagher
  • 466
  • 6
  • 14
  • 1
    Try running just the build commands without using `sudo` (run `sudo make clean` first to clean up). It's possible that the `sudo` command is using `root`'s `$PATH` instead of your own. If that's the case, `/opt/intel/bin` (or `/opt/intel/composer_xe_2015/bin`) won't be searched, leading to the `command not found` error for `icc`. – MattDMo Jan 06 '15 at 21:39
  • @MattDMo Wow, just another thing I never would have suspected being this new to linux. It turns out this was exactly the problem. The Intel guide assumed root access and so the command would work fine if I had set up in root. Instead I ran the build commands without `sudo`, and then the `python setup.py install` with `sudo` and everything works. My Numpy now works with my xeon phi perfectly. Thanks so much! – mgallagher Jan 07 '15 at 02:03

1 Answers1

1

On Unix/Linux systems, the sudo command (SuperUser DO) is set up to use the environment variables defined for the root user, not the user running the command. This can lead to problems if you install a program in a non-standard location, then need to run it with superuser privileges. For example, on OS X systems (which run on a flavor of BSD Unix), /usr/local/bin is not included by default in the PATH environment variable. You may set up your user's account to include this directory in your PATH, but if you try to use sudo with a program in there, it will not be found, unless you modify root's environment (or the system's environment) to include /usr/local/bin in the PATH.

This is likely the cause of your issues. icc and ifort are in (IIRC) in /opt/intel/bin and/or /opt/intel/composer_xe_2015/bin (one may be symlinked to the other), and you've added at least one of those directories to your $PATH environment variable, but when using sudo to execute the build commands, the programs are not being found. To get around this, don't build using sudo. First, to clear out any leftover temp files, run

sudo make clean

then run

python setup.py config --compiler=intelem

and confirm everything set up properly. Next, run

python setup.py build_clib --compiler=intelem build_ext --compiler=intelem

to build everything. If that succeeded with no errors, optionally run 

python setup.py test

(I don't think you need the compiler argument for this target) to verify the build, then run

sudo python setup.py install

to install everything in your system's site-packages directory. sudo is needed here as site-packages is owned by root.

MattDMo
  • 100,794
  • 21
  • 241
  • 231