How to use SRQ with different connections in libibverbs

Question

How to use SRQ when connected to more than one connections. lets say, there are three connections namely process 0, 1 and 2. for creating SRQ, we need to call

struct ibv_srq *ibv_create_srq(struct ibv_pd *pd, struct ibv_srq_init_attr *srq_init_attr);

for the above call we need to provide a protection domain, in my knowledge, protection domain is allocated specific to each connection by a call

ibv_alloc_pd(id->verbs)

in which id is created for each channel. basically, my question is how to assign a SRQ to different QP belonging to different Connection ID with different protection domain, or in other words, different connections can have a same protection domain?

Answer 1

A single SRQ or PD cannot be shared among multiple processes. The point of SRQ is to reduce the number of receives that need to be posted when a single process has many QPs. And sharing an SRQ among processes wouldn't really make sense. When you post a receive to the SRQ, the buffer would have to come from the address space of one of the processes. Then if a receive on a different process's QP used that work request, that process wouldn't be able to access the data it had received (since the buffer belongs to a different process).

You can of course share SRQs and all other verbs data structures between multiple threads in a single process.

The only verbs object that can be shared among multiple processes is an XRC domain (XRCD). See ibv_open_xrcd() etc.

Answer 2

The latest version of perftest should have example code for using SRQs: https://www.openfabrics.org/downloads/perftest/.

RDMAMojo also explains how to create Queue Pairs that use a SRQ: http://www.rdmamojo.com/2012/12/21/ibv_create_qp/.