I/O connectivity, a new (5th) generation is showing off

Design Space for Network I/O

When communication takes place, a commercially available solution is to use the I/O Bus. But the design space for network I/O is much more versatile. For performance reasons the communication path is closest to the CPU, minimizing hops, as every transition involves latency (Yet another reason why Programmed I/O (PIO) is used in favor to DMA for ultra low latency systems, e.g In Trading, HPC). Over the years solutions were presented which were exploring several options, including a solution which would use the DIMM slot for communication! [5].

Courtesy Bruening, Giloi [4]

Network I/O Evolution

10 and 100 Mbit devices were standard for a long time. But it was the trend to using commercial off the shelf (COTS) for HPC which was driving a need for improved network environments. Improvement was needed in two areas

  1. lowering latency (typically measured as a roundtrip operation and dividing the elapsed time by 2)
  2. increasing bandwidth (typically an amount of data exchanged in time)

Press fast forward… PCIe Gen5

PCI-E / PCIe comes with the concept of lanes and each lane offering so called Gigatransfers (GT). GTs include an en/decoding overhead, starting with Gen3 a 128b/130b line coding scheme was used. This overhead can be neglected but earlier generations had an 8b/10b encoding scheme which did come with a 20% penalty on data rates.

Throwing the gauntlet onto PCIe Gen5: 400GbE

While PCI express Gen3 got good traction over the last 8 years, industry basically skipped PCIe Gen4 (16GT) and just recently PCIe Gen5 (32GT) is becoming an option because of the release of latest AMD and Intel CPUs (Raphael/Alder Lake, respectively) . At this point we can look out for a system which is well suited for future networking trends, like 400GbE.

More to PCIe Gen5 …

What we should also mention is that PCIe Gen5 allows for other protocols to run over the physical layer. After quite a few consortiums and specifications like Gen-Z, CCIX the Compute Express Link (CXL) finally got enough momentum to kick off. This hints towards driving PCIe Gen5 in many ways including cache coherency [7]. Exciting times ahead.

References

[1] PCI-X
https://en.wikipedia.org/wiki/PCI-X

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
NEIO Systems, Ltd.

NEIO Systems, Ltd.

http://fastsockets.com || low latency, networking experts, 10GbE++, FPGA trading, Linux and Windows internals gurus