What are the performance and architectural differences between PCIe and QPI?

Question

PCIe 3.0 x16 and QPI 1.1 (20 lanes) have identical effective bandwidth (16 GB/s). So, I wanted to get a rough picture about the differences between the two.

What are the differences between the two in terms of latency and message rate (number of packets or TLPs per second)? For latency, my ballpark numbers are 20 ns for QPI and 200 ns for PCIe 3.0. Are these good estimates? If yes, why is PCIe's latency so much higher - is it due to the wire length?

Are there significant architectural differences between the two, apart from the fact that QPI provides cache snooping? To my knowledge, both use a layered protocol: transport layer through physical layer.

Answer 1

The two have fairly different messaging types due to their different roles. QPI is directly concerned with implementing cache coherency via the MESIF protocol and NUMA via a distributed directory. PCIe has no such notions, although they share in common memory read and write and completion message types (see here for some PCIe basics). They have similar power states and a priority scheme implemented through virtual channels. Both use credit-based flow control but there's no guarantee of any commonality in what sort of credits are maintained by QPI versus PCIe endpoints (as far as I know, the specifics of QPI credits are an Intel trade secret).

Message rate for each is usually expressed in GT/s. Typical QPI rates are 4.8, 6.4 and 8 GT/s, and 5 or 8 GT/s for PCIe.

Your latency estimates for both are probably low. QPI is on the order of a few hundred ns per hop. Note that 4+ socket systems may have more than one QPI hop between pairs of sockets. PCIe might be closer to 500ns, although again, that depends on the system topology. Latency between a processor socket's main memory and a PCIe card hanging directly off that socket (the PEG slot) is going to be lower than between that same memory and a card hanging off the south bridge.