Be (back) on Time!
Announcement: First 10GbE (IEEE 802.3ae) /25GbE Windows based IEEE 1588 compliant PTP software stack using hardware timestamps giving unpreceded accuracy.
“Be (back) on time! “ — while that call probably reminds you of your childhood, we would rather like to put this into the context of OS time synchronization. Microsoft Windows time synchronization to be precise. No millisecond, neither microsecond but nanosecond accuracy. [ For those stopping their reads within this chapter due to personal disbelieves, take with you: Windows OS (10 and Server 2019 and later) time synchronization and time reporting to applications can be achieved with nanoseconds accuracy. And actual time stamping precision for your data within pico-seconds. ]
Ever thought how you could synchronize your Windows network hosts with just a few nanoseconds accuracy? Here is how, but some detour first, as there is history.
Quite a few people being serious about time synchronization in a somewhat related field of finance and low latency have shifted to an operating system like Linux a decade ago.
Coming from Windows one reason for this move was the lack of features like packet timestamping support becoming mainstream. With all fairness, the road for TX and RX timestamps with hardware support also took a while on Linux. Well… We see Windows catching up in some ways. The improvements started when Microsoft made an effort to enter the HPC market. And indeed a few clusters made it into the Top500 list. That effort is now all wrapped up with some nice enhancements as a result to the server versions in an area of Technical Computing, Azure et al. Hence, most recently the Windows 2019 Server editions offers so called ‘precise’ adjustment / query functions with nanoseconds resolution. However, simple functionality, like that of ingress data getting RX timestamps even as of today aren’t supported and there is no standard way of obtaining them. We need HW assisted RX and TX timestamps though for best accuracy. More on this later.
We see Microsoft being more innovative and user friendly, even embracing Linux in a way of allowing for a good amount of interoperability, e.g allowing Linux to run side by side natively using a concept called WSL [6]. While we be waiting anxiously for additional enhancements, we’ll jump start the area on timestamping and start by providing a unique solution to be challenged. Our hardware platform are the ExaNICs from Exablaze, now Cisco [8], which have proven for a long time to be the right choice for low latency in a trading context. Given their accuracy of the hardware timestamping (TCXO and OCXO oscillators available), and resolution of just a few nanoseconds, they serve very well as a device for implementing best in class (PTP) synchronization.
We will know recap PTP in general and then present the results using our PTP solution achieving nanosecond sync accuracy for Windows hosts using hardware timestamps for both TX and RX, making it ideal not only for use cases in the finance world, but also for time-sensitive Windows-hosted measurement systems like for example openPDC.
PTP Protocol
If we want to synchronize our (Windows) network/hosts, then PTP is the ways to go. What you can achieve with dedicated hardware such as the ExaNIC devices, was presented very well in thorough detail at a STAC conference [4]. So a quick reminder what PTP is: PTP works by using a two-way exchange of timing messages, known as “event messages”.
It is easy to calculate a “round trip delay” from this, and the protocol then estimates the one-way message delay by simply halving the round-trip delay. However, the calculation is based on timestamps and hence only true hardware timestamps can lead to a precise solution.
You’ll find further articles [1, 3] giving a more detailed insight into the protocol, but en bref two factors are key to achieve best accuracy.
- Hardware based timestamping
- Boundary / Transparent clocks
Software based Timestamping:
When looking at software based timestamping we see the following criteria:
— Timestamp at Application or OS layer
— Get time from system clock
— Error is (relatively) huge as a consequence
In this context the slave software solution must compensate for the internal oscillator on the computer motherboard using software timestamping. Usually the local oscillator on the motherboard is of poor quality and the software timestamping is affected by the operating system latency. Hence, synchronization within microseconds is achievable with a software slave to a master. Results will vary and are between tens to hundreds of microseconds, even milliseconds using regular network devices. The latter is especially true on Windows OS versions prior to Windows 10/2016.
When looking at the impact on accuracy, then every component that handles the PTP packets after they are received on the wire before the timestamp is taken will increase the synchronization error. Software adds the most error since both processor load and the delay associated with handling interrupts impact how quickly a synchronization request is processed. Fortunately, only certain PTP actions are time critical. The most time critical PTP actions are recording timestamps of PTP packets, adjusting and maintaining the synchronized local clock. By placing these components in the MAC the NIC accesses PTP packets as soon as they are available from the wire. However, this solution is only given for dedicated hardware.
Hardware Timestamping
— Hardware assisted timestamp at PHY or MAC layer
— Get time from PTP Hardware Clock (PHC) on NIC
— Minimized error
With this method, the user space PTPD adjusts the NIC clock in a most accurate way and its counterpart, the phc2sys utility will sync the host clock. More details on this in our solution section below.
PTP Grandmaster functionality
While locked to GPS, the Grandmaster clock can provide precise nanosecond timestamp resolution and accuracy better than 30 nanoseconds referenced to GPS. A Grandmaster clock incorporates a local reference oscillator that is disciplined to GPS. See the screenshot of the command prompt for stat details of the Grandmaster. In our setup we are using the ExaNIC X10-GM [7] with an OCXO oscillator. This oscillator is the reference clock used with dedicated hardware for the precise timestamp of the incoming delay request and outgoing sync packets. The dedicated hardware approach is unaffected by operating system or network traffic latency.
PTP Slave with Hardware Timestamping, in general and as part of the ExaNIC on Windows solution
Hardware timestamps with a PTP software daemon provide precise nanosecond timestamp resolution with dedicated hardware. In our solution we use the PCIe Gen3 compliant ExaNIC X25 NICs (formerly Exablaze, now Cisco SmartNIC) with an TCXO oscillator offering a resolution of 4.2ns. As such, the hardware timestamp enabled slave solution has many advantages over the software slave, such as an improved oscillator, a 1PPS output for measurements compared to the master, and dedicated hardware that is unaffected by the operating system latency. Synchronization of better than 100 nanoseconds is achievable, even in a larger network with a transparent or IEEE 1588 compliant (boundary) Ethernet switch. Ideally ExaLINK Fusion et al.
Coming with our kernel bypass solution, the ExaNIC Windows Solution [0] provides a similar PHC subsystem. It features a ptpd daemon (fsock_ptpd) and phc2sys utility (fsock_phc2sys) and allows to access and configure/tune PHC via precise gettime/settime/adjtime system calls (Windows 10/Windows Server 2019). The improvements can be seen in the following perceived sync results.
References:
[0] ExaNIC Windows Drivers, libfsock for ExaNICs, http://www.fastsockets.com , NEIO Systems, Ltd.
[1] Li, Ying & Noseworthy, Bob & Laird, Jeff & Winters, Timothy & Carlin, Timothy. (2014). “A study of precision of hardware time stamping packet traces” 102–107. 10.1109/ISPCS.2014.6948700.
[2] Ken ICHIKAW. A Precision Time Protocol on Linux, LinuxCon, Japan 2014
[3] Li, Zhi & Zhong, Zhen-lin & Zhu, Wang-chun & Qin, Bin-yi. (2013). “A Hardware Time Stamping Method for PTP Messages Based on Linux system” TELKOMNIKA Indonesian Journal of Electrical Engineering. 11. 10.11591/telkomnika.v11i9.3255.
[4] Dr. Matthew Grosvenor et al. “Plain Old PTP. Better than you think?” STAC 2018 https://stacresearch.com/system/files/resource/files/STAC-Summit-29-Oct-2018-Exablaze.pdf
[5] OpenPDC https://github.com/GridProtectionAlliance/openPDC
[6] Windows Subsystem for Linux (WSL) https://docs.microsoft.com/en-us/windows/wsl/
[7] ExaNIC X10-GM, https://exablaze.com/docs/exanic/user-guide/exanic-gm/
[8] Cisco SmartNICs, https://www.cisco.com/c/en/us/products/interfaces-modules/nexus-smartnic/index.html
