0

I am having trouble understanding the context from http://www.redbooks.ibm.com/redbooks/pdfs/sg247351.pdf

It is describing how the transport layer is implemented but every info I have read only explains the features not how the transport layer is implemented.

InfiniBand architecture offers a significant improvement for the transport layer over other technologies such as Ethernet: all functions are implemented in the hardware.

Is there a way to find out the basic idea of how the functions are implemented in hardware for Infiniband transport layer? For example, is it using the logical controls and registers to run all the necessary transport layer functions?

Sungho Hong
  • 340
  • 2
  • 16

2 Answers2

0

InfiniBand is a well defined end-to-end interconnect protocol, and is officially defined and maintained by the InfiniBand Trade Association.

You can find the complete specifications on the IBTA website: http://www.infinibandta.org/content/pages.php?pg=technology_public_specification

In the above specs, you can find anywhere from physical layer definitions to high-level connection management and subnet administrator protocols. This is the guide that InfiniBand hardware vendors use for implementing their products. Vendor-specific implementations can differ significantly in how and where functionality is implemented and is confidential in many cases.

Yuval Degani
  • 196
  • 3
  • Thanks for the reply, but the specifications link you have mentioned does not answer my question. It seems as you say it is vendor specific or confidential. I'll try to find some other way to solve this question. – Sungho Hong May 14 '18 at 21:29
  • The InfiniBand specification defines how the transport layer should be implemented (either in hardware or software), and also defines how it should be exposed to users. – Yuval Degani May 16 '18 at 07:59
  • Thanks for the reply, but it does not explain whether the logic of the transport layer comes from the hardware or the software. I am figuring out this issue step by step with the Infiniband providers. – Sungho Hong May 19 '18 at 11:24
0

I have figured out and confirmed with the Infiniband providers, although the document explains that transport layer is built on hardware, it is only referring to the the queue pairs in HCA. Accordingly, most of the logics are just working in the background process from the Infiniband driver.

Generic Process

1) Application is written to use InfiniBand.

2) Interfaces in the Mellanox driver (Like MPI) allow the application, through the driver, to write directly to memory space (posting a work request) on the server (now we are into Transport layer).

3) Memory space details between source and destination servers are exchanged, so that communication will now flow between these RAM memory regions. This is a key benefit to InfiniBand, because it is bypassing the kernel/CPU and exchanging data in hardware (between memory regions using queue pairs between the HCAs (adapters). When we say data is exchanged in hardware, we are talking about server memory, and the HCA.

4) The HCA will segment packets according to MTU and push data out onto the wire. At the other end, it will be reassembled. This is still "Transport Layer".

5) As the bits get pushed onto the wire – this is physical layer.

Conclusion

The driver is in the background, telling the OS how to interoperate with the HCA (deciding transport protocol InfiniBand, vs, Ethernet, MTU, CM vs, RD queue pair exchange mode, etc.). Additionally, it supplies the InfiniBand programming interfaces and additional utilities for troubleshooting etc. I guess you could say there is some logic here, provided by the OFED HCA driver.

Sungho Hong
  • 340
  • 2
  • 16