OpenMDAOv1.x: warning: parallel derivs not running under MPI

Question

I just finished installing OpenMDAOv1.3 on our super computer. The installation was successful and all tests either passed or skipped. However, when I ran the tests I got the following warning:

*path/OpenMDAO/openmdao/core/driver.py:228: UserWarning: parallel derivs %s specified but not running under MPI
  warnings.warn("parallel derivs %s specified but not running under MPI")

I'm not sure what to do about this (if anything) and so I am looking for information about the implications of the warning message. We are planning on running in parallel with the built in MPI functionality of OpenMDAO. I currently have openmpi-1.8.4 loaded on the system.

Answer 1

You can ignore that warning... it's just because the test is not running under MPI. (The test will be fixed to suppress the warning).

There are additional steps you need to take to run in parallel. This has not yet been documented, but some of us have been using the following steps when working in an anaconda environment:

sudo apt-get install openmpi-bin openmpi-doc libopenmpi-dev
pip install mpi4py # do not use conda install, this causes an mpich conflict
sudo apt-get install liblapack-dev gfortran
pip install --no-deps git+https://bitbucket.org/petsc/petsc@v3.5 
conda install cython
pip install --no-deps  git+https://bitbucket.org/petsc/petsc4py@3.5

To verify you are good to go, you can run the tests in the mpitests directory as follows:

cd mpitests
find . -name "*.py" -exec python {} \;

Alternatively you can use the testflo package from

pip install git+https://github.com/naylor-b/testflo.git

and from the OpenMDAO directory, issue the following command:

testflo .  -i

Note that this assumes a Linux type operating system.. Parallel execution on Windows is not currently supported

Answer 2

To get OpenMDAO succesfully running in parallel using Conda, OpenMPI, mpi4py, PETSc and petsc4py I have adapted this procedure for OS X to Ubuntu:

Prerequisites

Ubuntu

sudo apt-get install libibnetdisc-dev
sudo apt-get install libblas-dev libatlas-dev liblapack-dev

Install OpenMPI

Download OpenMPI: https://www.open-mpi.org/software/ompi/v1.10/

Extract and configure

cd ~/Downloads/openmpi-1.10.1
mkdir build
cd build
# I prefer to keep /usr/local clean, so I put it in /opt/openmpi
./configure --prefix=/opt/openmpi --with-devel-headers --enable-binaries
make
sudo make install

Add the following to your bash profile

OS X

export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:/opt/openmpi/lib
export PATH=${PATH}:/opt/openmpi/bin

Ubuntu

export LD_LIBRARY_PATH=LD_LIBRARY_PATH=/opt/openmpi/lib:$LD_LIBRARY_PATH
export PATH=/opt/openmpi/bin:$PATH

Activate your conda environment

source activate myenv

Install mpi4py

git clone https://github.com/mpi4py/mpi4py.git ./mpi4py.git
cd mpi4py.git
python setup.py build --mpicc=/opt/openmpi/bin/mpicc
python setup.py install

Install PETSc

git clone https://bitbucket.org/petsc/petsc.git ./petsc.git
cd petsc.git
python setup.py build
python setup.py install

Install petsc4py

git clone https://bitbucket.org/petsc/petsc4py.git ./petsc4py.git
cd petsc4py.git
python setup.py build
python setup.py install