7

I want to create a virtual environment with some python packages on a cluster where different intel processors are available (eg sandy bridge old and haswell new). The login node, from where the installation has to happen, is haswell.

So I:

pip install virtualenv
cd my_project_folder
virtualenv my_project

If I do like this, when I login to a sandy-bridge, and I run python I get:

Please verify that both the operating system and the processor support Intel(R) MOVBE, F16C, FMA, BMI, LZCNT and AVX2 instructions.

because the python executable was created with compiler flags that only work for haswell nodes. I can tell virtualenv the python executable I want to use:

virtualenv -p /usr/bin/python2.7 my_project

but then when I

pip install numpy

he finds and takes the haswell compiled version of it, which won't run on sandy-bridge arch again.

How can I instruct pip to not use packages compiled with the haswell flags?

For some reasons I could explain I cannot create the virtenv using a sandy-bridge processor.

Evan Teran
  • 87,561
  • 32
  • 179
  • 238
fabiob
  • 273
  • 4
  • 14
  • 1
    So basically, you are trying to use a virtual environment which was created on machine A, on machine B? – hoefling Oct 27 '18 at 12:36
  • yes. however the question is more general. can I decide the c-compiler flags using the pip command? – fabiob Oct 30 '18 at 16:22
  • 1
    Sure you can pass the compiler flags to the installation, the usual `CFLAGS="-foo" CPPFLAGS="-bar" LDFLAGS="baz" pip install qux` etc, but to me, it looks like your problem is that you already have stuff installed and compiled and are trying to call the compiled stuff from another machine with an incompatible runtime, or did I misunderstand you? – hoefling Oct 30 '18 at 17:32
  • Also don't forget that the virtualenv is not an isolated portable installation of Python. It's merely a shim and still relies a lot on stuff that's in the base installation. Maybe a better solution would be fixing venv creation on the machine with the sandy bridge? It you want, I can try to help in doing that. – hoefling Oct 30 '18 at 17:35
  • Or maybe it's wise to compile another Python distribution from source, if you don't have root access on the machine with the sandy bridge etc. – hoefling Oct 30 '18 at 17:36
  • a bit of misunderstanding: true I have sth installed and compiled, but it is not a problem to redo it. I want to install and compile on haswell something that could also work on sandy. since I have a python interpreter that works as I wish, could a solution be ```CFLAGS=`/usr/bin/python2.7-config --cflags` pip install numpy```? (I do not have a terminal here) – fabiob Oct 31 '18 at 17:32
  • 2
    `CFLAGS=$(/usr/bin/python2.7-config --cflags)` should have no effect as those flags are applied by default. However, compilation flags are only applied if you actually compile `numpy` from source; to enforce that, you should add `--no-binary=numpy` to the install command. Additionally, increase verbosity with `-vvv` to see the compilation details. The full command would thus be smth like `CFLAGS="-O1" pip install numpy -vvv --no-binary=numpy`. You should be able to see what flags are applied by the C compiler and linker to each C extension built in the package. – hoefling Oct 31 '18 at 18:11
  • thanks a lot for the dedication @hoefling. if I do not use the ```--no-binary=numpy```, how does pip decide which binary to use? I haven't found a guide online that explains how pip (install) works (many explain how to use it), so if you also point me to a documentation that does that, or another question here on SO I would be very grateful. this [https://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/quickstart.html] sounds like a long detour, but I guess if I learn how to package some code I will also answer my question. – fabiob Oct 31 '18 at 18:48
  • 1
    If you issue `pip install numpy`, `pip` queries the current platform arch, python interpreter version and ABI and goes through the list of available indexes, looking for a binary wheel with the name matching the queried data. It doesn't look for specific compiler options; the first one matching the python version and platform arch will be installed. If no wheel is found, `pip` looks for a source dist and compiles the package from source then. – hoefling Oct 31 '18 at 20:50

0 Answers0