posted recv buffers returned synchronously or async after calling rdma_disconnect()

Question

When calling rdma_disconnect(), do I get completion queue events for all posted recv work requests before rdma_disconnect() returns, or should I expect them to come in after rdma_disconnect() has returned?

Answer 1

The receives will complete (with a "flush error" status) asynchronously (possibly) after rdma_disconnect() returns. As you can see from the source for rdma_disconnect(), all that it does is transition the QP to the error state and send a disconnect request to the other side.

Transitioning a QP to the error state does guarantee that all pending work requests posted to the QP will be completed with error status, but the modify QP operation returns immediately without waiting for the queues to drain. Similarly rdma_disconnect() doesn't wait for all pending work requests to finish -- in fact it would be hard to see how it could, since the RDMA CM doesn't really have any way to know how many work requests are queued, let alone peek into the associated CQ to see when they all complete.

Chapter 10 of volume 1 of the IB spec goes into great detail about work request processing if you are wondering about the corner cases about requests that are in flight at the time of the transition to error state, etc.