1

I have supplementary processes which should exchange files over IB network. Then the files will be used by other processes.

The workflow is following:

  1. Create files in /dev/shm
  2. Resize files appropriately
  3. Map files in process' VM
  4. Register mmaped area with ibv_reg_mr
  5. Initiate RDMA operation for data transfer

It turned out that the bottleneck in my scheme is ib_reg_mr (I measured separately registering 3Gb memory takes 1.78 seconds). It seems that it triggers mapping of the memory region to the process's address space. Unfortunately, this operation is not needed, because the receiving process does not use this memory. The memory should be mapped and used by other processes later.

So I thought it would be wise to circumvent altering page tables for the receiving process, but I was not able to find if it is possible at all.

Could you advise me, if there is a way to initiate a data transfer on a memory region without mapping it to process' address space?

Brett Holman
  • 743
  • 6
  • 18
mcsim
  • 1,647
  • 2
  • 16
  • 35

2 Answers2

2

As far as I know there is no way to register a memory region without having it mapped to the process page tables. During registration the pages are pinned so that the mapping doesn't change while the device is accessing the pages, and this requires mapping them to the process. With on-demand-paging, you could delay the mapping of the pages into the process until the HCA actually uses them, but eventually they will be mapped.

haggai_e
  • 4,689
  • 1
  • 24
  • 37
1

Please try FRWR (Fast memory Registration Work Request) instead. It is an API that does memory registration via ibv_post_send verb - much faster way:

http://lists.openfabrics.org/pipermail/general/2008-May/050235.html

http://lxr.free-electrons.com/source/include/rdma/ib_verbs.h?v=3.2#L734

kliteyn
  • 1,917
  • 11
  • 24
  • I found description of ibv_send_wr data data structure in Linux Kernel Networking book, but unfortunately this struct has another definition in /usr/include/infiniband/verbs.h on the machine, which is relevant for me. I assume this means that kernel on this machine does not support this feature. And I can't upgrade the kernel. But anyways thank you, I may be using this interface in the future. – mcsim Jun 06 '16 at 17:17
  • @mcsim you don't need to update kernel. Don't know which distro and kernel are you using, but if it is not too old, you can just install OFED package. – kliteyn Jun 07 '16 at 19:48