1

I understand that OpenMPI uses OpenIB, and OpenIB uses Ip over Infiniband (IPoIB).

I don't understand why not use native IB, if is it faster than IPoIB?

Is there any implementation of MPI, that it use native IB?

Brayme Guaman
  • 175
  • 2
  • 12

2 Answers2

3

OpenIB is the early name used by the Open Fabrics Alliance. As early as in 2005 the name OpenIB was tossed away in favor of Open Fabrics. Open Fabrics Alliance distributes OFED a software stack that supports many protocols and APIs including DAPL, IPoIB, SCSI over RDMA and many more. OFED relies on low level device drivers provided by hardware vendors.

Some hardware vendors distribute their custom builds of OFED. These custom builds do not differ much from any other OFED distribution except they are bundled together with device drivers.

In the past OFED used to include an MPI implementation, namely OpenMPI, but is no longer doing so (you probably read the OpenMPI FAQ).

OpenMPI still uses openib name for its OFED based InfiniBand Byte Transfer Level component.

In different periods of time OpenMPI supported vendor specific InfiniBand APIs as well, such as Mellanox mVAPI (mvapi BTL), and Mellanox Messaging Library mxm.

However, openib offers portability and if you use the latest versions, you are likely to have performance as good as that of the vendor specific APIs.

The openib component is not using IP over IB. If you want to use IP over IB, use the tcp component instead and set up mpiexec hostfiles accordingly.

Dima Chubarov
  • 16,199
  • 6
  • 40
  • 76
  • Thanks but I still don't understand. Why use IPoIB instead of IB native? What is the advantage of IPoIB? – Brayme Guaman Mar 28 '17 at 15:18
  • 1
    Here says http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-6.html Openmpi uses IPoIB from openib. – Brayme Guaman Mar 28 '17 at 19:20
  • 1
    @BraymeGuaman good point. The documentation you link to says that IPoIB is used only for starting processes when the MPI envrironment has not been set up yet. IPoIB *is not* used for actual data movement between MPI processes with openib BTL. – Dima Chubarov Mar 29 '17 at 04:41
-1

AFAIK, MVAPICH2 is an MPI distribution that does not use IPoIB, i.e., it directly uses verbs.

Anuj Kalia
  • 803
  • 8
  • 16
  • What do you mean "verbs"? I'm new to infiniband. – Brayme Guaman Mar 28 '17 at 00:50
  • 1
    Verbs refers to the common user-mode interface for InfiniBand-like RDMA devices. It consists of functions such as `ibv_post_send` and `ibv_post_recv` for sending and receiving data, respectively. This may be useful: http://www.hpcadvisorycouncil.com/pdf/building-an-rdma-capable-application-with-ib-verbs.pdf. – Anuj Kalia Mar 28 '17 at 04:03