The Mellanox LinkX Interconnects group introduced a new type of Active Optical Cable (AOC) for 200G HDR (High Data Rate) InfiniBand systems called the “H-Cable”. Named because of its shape that enables cross connecting up to four InfiniBand switches (or ports) together using – a single cable!
These unique active optical cables enable saving considerable CAPEX (CAPital EXpense) and OPEX (OPerating EXpense) by:
By using 200G HDR switch and interconnects but splitting each port into two 100G HDR100 links.
Not only does the port splitting enable a whopping 80-ports of 100G HDR100 but also the H-Cable enables reducing the number of actual ports needed to accomplish a cross connect of up to four switches from 8-ports and four AOCs using individual EDR AOCs to 4-ports and one cable using the new 200G cross connect H-Cable. This approach saves both switch ports and transceiver ends which are expensive not to mention power and cabling costs.
This unique AOC cable is based on four 200G HDR “SR4-like” multi-mode transceivers. Each transceiver consists of four lanes of 50G PAM4. Two of the lanes are grouped together to form a 100G link called HDR100 (2x50G PAM4). The 100G-based HDR100 links are cross connected inside the AOC cable. (See diagram below)
The SB7700-Series EDR InfiniBand switches offer 36-EDR 100G ports. Mellanox’s new 1RU Quantum 8700-series 200G HDR-based switches offers 40-ports of 200G HDR or 80-ports of HDR100 when using 1:2 splitter cables – 80-ports of HDR100 in a single 1RU chassis! This enables 40-ports down to servers and storage and 40-port up to other switches.
In the figures below, it is clear the benefits of the H-Cable AOC approach in simplicity, increased reliability, lower maintenance and fewer cables and switch ports.
Single H-Cable and 4-port HDR Switch Configuration
PAM4 is a new electrical and optical modulation technique becoming popular in both 200G and 400G networks. PAM4 that enables two data bits to be sent per clock instead of the traditional 1-bit per clock by creating four levels in a signal (00,01,10,11) instead of just two (1,0). This doubles the net data transfer rate while keeping the clocking optics and electronics at the same 25G rate which is considerably less expensive than doubling the clock rate to 50G.
QSFP56 is slightly longer than the QSFP28 form-factor at the optical connector end. The extra space is needed to enable more electronics to be included and increased thermal dissipation up to 5 Watts. The Mellanox QSFP56 HDR transceivers and transceivers used in the H-Cable typically dissipates about 4.5 Watts. Also, the “56” means the form-factor shell supports the higher frequency components at 56G and suppressing the associated electromagnetic noise from getting out. The QSFP56 pluggable transceivers uses the same 8-fiber MPO-12 optical connector as the QSFP28 and can be used in installed infrastructures of OM3 and OM4 multi-mode fiber and MPO-12 optical connectors without issues.
The HDR and H-Cable AOCs are based on the same 200G transceiver design as the Mellanox individual connectorized 200G HDR transceiver – which is very similar to the Ethernet 200G SR4 transceiver as well. Mellanox designs, builds and tests its 200G transceivers and AOCs to ensure above average high quality and reliability.
In short, no! The modulation schemes are different.
EDR uses 4-channels of 25G NRZ modulation – this is the traditional two-level 1,0 modulation scheme.
HDR100 uses 2-channels of 50G PAM4 – this is a four-level 00,01,10,11 modulation scheme.
Plugging an EDR AOC into an HDR switch will result in EDR 100G line rates. All good. The 50G PAM4 SerDes in the switch will convert to 25G NRZ. Switch backward compatibility.
But the reverse is very different! Plugging a 100G HDR100 AOC into a 100G EDR switch port will convert the 100G HDR100 into 50G! How? The EDR switch can only see two-levels in the four level 50G PAM4 HDR100 AOC. So, it converts the 50G PAM4 into 25G. But that’s not all! The HDR100 has only 2-lanes, (2x50G PAM4) not four like EDR (4x25G). So, 2-lanes x 25G = 50G. It works … but at half the rated speed. And 50G is not really an InfiniBand standard speed.
The H-cable provides cross-connect capability between ToR (Top of Rack) and Spine switches. It enables a single HDR InfiniBand QSFP56 switch port to operate as two 100G HDR100 ports but physically uses only one port. It creates a non-blocking fat-tree topology thereby doubling the number of ports available compared to linking with individual 100G EDR AOCs. This eliminates a third-level of switching, which is needed for traditional large cluster designs of 1600 up to 3200 HDR100 ports, and results in significant CAPEX and OPEX savings. Additionally, by reducing a layer of switching systems, the cluster latency is reduced, which results in improved system performance.
Mellanox’s new InfiniBand H-Cable delivers a series of financial and configuration benefits as well as increasing overall system reliability, simplicity and system latency. Contact Mellanox sales for more information.
Contact your Mellanox sales representative for availability and pricing options and stay tuned to my blog more interconnect news and tips.
Sign up for free ebook download here: LinkX Cables and Transceivers ebook