1

let two infiniband process execute on two machines. When one process terminates, all its resources (allocated file, buffers, ...) are freed even the memory that the process registered for RDMA WRITE/READ operations.

Can someone has references on how the remote Channel Adapter on the other process knows that the buffer memory is now free ? Which element of the infiniband stack does this control and when ?

Fopa Léon Constantin
  • 11,863
  • 8
  • 48
  • 82
  • If the the data is being transferred directly to memory owned by the process, it's up to driver on the local machine to ensure that when the process exits that the transfer is aborted and/or that the exit of the process is delayed until its completed or aborted. The driver can also potentially use intermediate buffers owned by the driver instead. – Ross Ridge Sep 24 '15 at 15:06
  • @RossRidge, however, things are different in the case of RDMA WRITE/READ because the remote process has no notifications that such operations are performed. My understanding is that there is no mechanism (interruption, driver callback, ...) to inform the remote process of what is going on actually in its buffer. Does it allow process to continue to read or write on a remote free memory ? or something `special` is perform when a memory is freed ? – Fopa Léon Constantin Sep 24 '15 at 15:24
  • Nothing changes. The driver on the same machine as the exiting process has to ensure that the transfer is either aborted, the process doesn't exit, or the process's memory isn't actually accessed by the hardware in the first place. – Ross Ridge Sep 24 '15 at 15:40
  • @RossRidge do you have references for more detail on your answer ? thanks for your time ! – Fopa Léon Constantin Sep 24 '15 at 15:52
  • It's just the way drivers have to work in the presence of DMA operations, otherwise they'd be horribly broken. Note that it appears that the remote process needs to first "register" any memory before it becomes available as a target of a RDMA operation. This means the driver can "unregister" it automatically when process exits. Whether that results in any current transfer being aborted, the driver blocking until the transfer is finished or if the driver owners the memory when its registered, I don't know. It probably depends on the implementation. – Ross Ridge Sep 24 '15 at 17:07

2 Answers2

2

When a Linux process terminates all its open file descriptors are closed. A process that uses RDMA would have an open device file connected with the ib_uverbs module. When that file descriptor is closed the module cleans up every open RDMA resource opened by the process, including any memory region and queue pair. This clean up includes notifying the HCA's driver and through it the device itself.

If the remote machine continues attempting to perform RDMA operations, the HCA will refuse its requests, because the relevant resources would be closed.

haggai_e
  • 4,689
  • 1
  • 24
  • 37
  • Thank you for those valuable details. By `the HCA will refuse its request` do you mean that HCA will not send and ACK ou NACK after receiving a remote rdma read/write ? – Fopa Léon Constantin Sep 25 '15 at 12:19
  • The HCA is required to send NAKs if the MR is closed but the QP is still open. Once the QP is closed it seems no response if I recall correctly. – haggai_e Sep 25 '15 at 12:21
  • Thanks you again for your time (+1) – Fopa Léon Constantin Sep 25 '15 at 12:24
  • Is it possible that `ib_uverbs` module uses interrupts to notify HCA's driver about all of this ? – Fopa Léon Constantin Sep 25 '15 at 12:29
  • 1
    Interrupts can be used by the device to notify the driver about events, not the other way around. For instance, it is possible that interrupts are used by the device to notify the driver and ib_uverbs that a resource cleanup command has completed. – haggai_e Sep 25 '15 at 12:31
0

When a connected peer exits without sending a disconnect, the IB core initiates a clean up procedure which also sends a RDMA DISCONNECT MAD packet to the other RNIC if it's RC QP. On seeing a disconnect, the other end RNIC cleansup it's SQs/RQs and stuff

Anji M
  • 11
  • 2