Peer-to-peer communications between PCIe devices?

Question

In order to enable p2p communication between NVMe SSDs and other PCIe devices, I wonder if I need to make some modifications to the Linux kernel and the NVMe driver, and something else? And what modifications should I make? I have searched for some days, but there is little information about this, I need some references. Thanks a lot!

Answer 1

There's an NVMe spec. feature called "Controller Memory Buffer Write Data". In turn CMB Write Data supports P2P transactions between an NVMe device and something else on the PCIe bus. While most of the CMB spec has been implemented in the latest upstream kernel, CMB Write Data has not. Problem with CMB is it offers very little memory to play with for these P2P transactions. This is one of the reasons no one has implemented the feature upstream yet. You can however play with it via the user-space NVMe driver/framework, SPDK.

Also note that there's a new NVMe spec. feature called Persistent Memory Region. This has far more memory to play with and I suspect someone will contribute it upstream in the not too distant future.

https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2018/20180808_SOFT-201-1_Bates.pdf is useful background material.