0

There is a cluster of 10 nodes (x86_64 CPU + 8 GB RAM) connected by Infiniband or Compact PCI Serial (ie PCI-Express 8x 3.0) with the possibility of RDMA (shared memory). How should I allocate shared memory between nodes in the cluster to use MPI (OpenMPI, MPICH2, Intel MPI) with openib BTL (Byte Transfer Layer), do I have to use for this mmap(), SysV(), MPI_COMM_ALLOC_MEM() or as something else?

What are the functions usual used for this purpose, or can you give an example of source code for allocating of shared memory for RDMA when using Infiniband or Compact PCI Serial (ie PCI-Express 8x 3.0)?

Community
  • 1
  • 1
Alex
  • 12,578
  • 15
  • 99
  • 195
  • 1
    The `sm` BTL transfers messages within a single node only. RDMA transfers are managed by the `openib` BTL. If you'd like to learn to program InfiniBand, look for information on using the InfiniBand verbs API. – Hristo Iliev Sep 03 '13 at 11:16
  • @Hristo Iliev Is this about "OpenIB Alliance was renamed in 2007 to be the OpenFabrics Alliance"? http://en.wikipedia.org/wiki/OpenFabrics_Alliance Is that, this question refers to the `Infiniband(RDMA)/PCI-E(RDMA)` and not directly related to the MPI? – Alex Sep 03 '13 at 11:35
  • 1
    InfiniBand RDMA does not directly support shared memory in the usual sense of the word. One needs a layer on top of it in order to provide the sharing semantics. Otherwise another mechanism like one-sided MPI or DSM should be used in order to access memory on remote nodes. – Hristo Iliev Sep 04 '13 at 07:24
  • @Hristo Iliev Big thanks! But if I self implemented its own driver that provides a mapping memory nodes (connected by Infiniband / PCI-E) into a single virtual address space, in this case I need to use the openib-BTL or sm-BTL(http://www.open-mpi.org/faq/?category=sm#what-is-sm)? – Alex Sep 05 '13 at 12:31
  • 1
    I'm not sure that I fully understand your question. If you write a driver that can provide some kind of single system image with virtual shared memory over InfiniBand (similar to what ScaleMP's vSMP does), then Open MPI would use the `sm` BTL out of the box.. – Hristo Iliev Sep 05 '13 at 14:17
  • @Hristo Iliev Thanks, yes, you understand correct. i.e. `openib` needed only as layer over Infiniband to produce "shared-memory"? Do you know, does any others implementation of MPI except OpenMPI, such as MPICH2 or Intel MPI support `openib` and `sm`? – Alex Sep 05 '13 at 14:36
  • 1
    I get the impression that you are confusing things here. `openib` and `sm` are modules specific to Open MPI. They do not implement shared memory. `openib` transfers messages over InfiniBand and `sm` transfers messages over shared memory. Most other MPI implementations also come with modular architectures and bring different modules that implement message transfers over various media. M(VA)PICH and Intel MPI (which is based on MPICH) do. – Hristo Iliev Sep 05 '13 at 17:47
  • @Hristo Iliev Yes, I really do not know enough about the openib and sm. I have ready RDMA over Infiniband - windows of mapped memory in the single virtual address space and I need to understand what I must use for versatility and easy transition between: OpenMPI, MPICH2, Intel MPI? – Alex Sep 05 '13 at 18:11

0 Answers0