Mellanox Introduces 200G InfiniBand H-Cable Active Optical Cables Cross-Connect Splitter

Mellanox Active Optical Cable (AOC) for 200G HDR, also known as “H-Cables”  
200G, Cables

Active Optical Cables (AOC) for 200G HDR Eliminates 3rd Layer of InfiniBand Switches & Delivers Big Savings

The Mellanox LinkX Interconnects group introduced a new type of Active Optical Cable (AOC) for 200G HDR (High Data Rate) InfiniBand systems called the “H-Cable”. Named because of its shape that enables cross connecting up to four InfiniBand switches (or ports) together using – a single cable!

These unique active optical cables enable saving considerable CAPEX (CAPital EXpense) and OPEX (OPerating EXpense) by:

  • Reducing the number of layers of InfiniBand switches in large configurations
  • 75% reduction in number of AOC cables compared to implementing via 100G EDR AOCs
  • Frees up four expensive switch ports for other uses

What’s the Magic?

By using 200G HDR switch and interconnects but splitting each port into two 100G HDR100 links.

  • The HDR link consists of 4 channels at 50G PAM4.
  • HDR100 uses 2 channels of 50G PAM 4 for 100G.

Not only does the port splitting enable a whopping 80-ports of 100G HDR100 but also the H-Cable enables reducing the number of actual ports needed to accomplish a cross connect of up to four switches from 8-ports and four AOCs using individual EDR AOCs to 4-ports and one cable using the new 200G cross connect H-Cable. This approach saves both switch ports and transceiver ends which are expensive not to mention power and cabling costs.

This unique AOC cable is based on four 200G HDR “SR4-like” multi-mode transceivers. Each transceiver consists of four lanes of 50G PAM4. Two of the lanes are grouped together to form a 100G link called HDR100 (2x50G PAM4).  The 100G-based HDR100 links are cross connected inside the AOC cable. (See diagram below)

The SB7700-Series EDR InfiniBand switches offer 36-EDR 100G ports. Mellanox’s new 1RU Quantum 8700-series 200G HDR-based switches offers 40-ports of 200G HDR or 80-ports of HDR100 when using 1:2 splitter cables – 80-ports of HDR100 in a single 1RU chassis!  This enables 40-ports down to servers and storage and 40-port up to other switches.

H-Cable Feature Summary:

  • A single Active Optical Cable (AOC) cable in a cross connect configuration
  • Based on 200G HDR InfiniBand transceivers – similar to a 200GbE SR4 Ethernet transceiver
  • 4-channels of 50G PAM4 modulation per transceiver-end
  • QSFP56 form-factor
  • Multi-mode VCSEL-based
  • 5Watt power dissipation per transceiver-end (typical).
  • From 3 to 30-meter lengths of Low-Smoke, Zero- Halogen (LSZH) OM4 fiber

In the figures below, it is clear the benefits of the H-Cable AOC approach in simplicity, increased reliability, lower maintenance and fewer cables and switch ports.

Single H-Cable and 4-port HDR Switch Configuration

Similar Configuration Using Four 100G EDR AOCs and 8-EDR Switch Ports

 

What is PAM4?

PAM4 is a new electrical and optical modulation technique becoming popular in both 200G and 400G networks.  PAM4 that enables two data bits to be sent per clock instead of the traditional 1-bit per clock by creating four levels in a signal (00,01,10,11) instead of just two (1,0).  This doubles the net data transfer rate while keeping the clocking optics and electronics at the same 25G rate which is considerably less expensive than doubling the clock rate to 50G.

What is QSFP56?

QSFP56 is slightly longer than the QSFP28 form-factor at the optical connector end. The extra space is needed to enable more electronics to be included and increased thermal dissipation up to 5 Watts.  The Mellanox QSFP56 HDR transceivers and transceivers used in the H-Cable typically dissipates about 4.5 Watts. Also, the “56” means the form-factor shell supports the higher frequency components at 56G and suppressing the associated electromagnetic noise from getting out. The QSFP56 pluggable transceivers uses the same 8-fiber MPO-12 optical connector as the QSFP28 and can be used in installed infrastructures of OM3 and OM4 multi-mode fiber and MPO-12 optical connectors without issues.

200G Transceivers

The HDR and H-Cable AOCs are based on the same 200G transceiver design as the Mellanox individual connectorized 200G HDR transceiver – which is very similar to the Ethernet 200G SR4 transceiver as well. Mellanox designs, builds and tests its 200G transceivers and AOCs to ensure above average high quality and reliability.

Mellanox 200G 4-channel QSFP56 Transceiver for HDR InfiniBand and Ethernet

Is InfiniBand HDR100 the Same as EDR? – Both Transfer at 100G!

In short, no!  The modulation schemes are different.

EDR uses 4-channels of 25G NRZ modulation – this is the traditional two-level 1,0 modulation scheme.

HDR100 uses 2-channels of 50G PAM4 – this is a four-level 00,01,10,11 modulation scheme.

Plugging an EDR AOC into an HDR switch will result in EDR 100G line rates. All good.  The 50G PAM4 SerDes in the switch will convert to 25G NRZ.  Switch backward compatibility.

But the reverse is very different! Plugging a 100G HDR100 AOC into a 100G EDR switch port will convert the 100G HDR100 into 50G! How? The EDR switch can only see two-levels in the four level 50G PAM4 HDR100 AOC. So, it converts the 50G PAM4 into 25G. But that’s not all!  The HDR100 has only 2-lanes, (2x50G PAM4) not four like EDR (4x25G). So, 2-lanes x 25G = 50G.  It works … but at half the rated speed. And 50G is not really an InfiniBand standard speed.

H-Cable Used in Systems

The H-cable provides cross-connect capability between ToR (Top of Rack) and Spine switches. It enables a single HDR InfiniBand QSFP56 switch port to operate as two 100G HDR100 ports but physically uses only one port. It creates a non-blocking fat-tree topology thereby doubling the number of ports available compared to linking with individual 100G EDR AOCs. This eliminates a third-level of switching, which is needed for traditional large cluster designs of 1600 up to 3200 HDR100 ports, and results in significant CAPEX and OPEX savings. Additionally, by reducing a layer of switching systems, the cluster latency is reduced, which results in improved system performance.

200G ToR Switches and Spine Switches with FAT-tree Topology

Mellanox LinkX H-Cable Benefits:

  • Eliminates the need for a 3rd layer of switches in large clusters deployments; >1600 node clusters
  • Enables linking up to 4 InfiniBand switches ports with a single cable with links running at HDR100 speeds
  • Saves 50% of the transceiver-ends and 50% of port usage over using 4 EDR AOCs with 8 transceiver-ends
  • Single 4-transceiver cable using 4-switch ports versus 4 AOCs with 8 EDR transceivers and 8-switch ports
  • Fewer cables and transceiver-ends = simpler configurations and increased reliability from fewer components
  • Frees up 4 switch ports for other uses ($cost/port x 4 ports)
  • Four x 4.5W HDRs verses 8 EDRs at 2.4W saves ~ 1 Watt over all
  • Reduction in switch layers yield lower system latency delays
  • Increase in system reliability from using fewer optical transceivers and fiber links

Summary

Mellanox’s new InfiniBand H-Cable delivers a series of financial and configuration benefits as well as increasing overall system reliability, simplicity and system latency.  Contact Mellanox sales for more information.

Supporting Resources:

Contact your Mellanox sales representative for availability and pricing options and stay tuned to my blog more interconnect news and tips.

Sign up for free ebook download here: LinkX Cables and Transceivers ebook

 

About Brad Smith

Brad is the Director of Marketing at Mellanox, based in Silicon Valley for the LinkX cables and transceivers business focusing on hyperscale, Web 2.0, enterprise, storage and telco markets. Recently, Brad was Product Line Manager for Intel’s Silicon Photonics group for CWDM4/CLR4 and QSFP28 product lines and ran the 100G CLR4 Alliance. Director of Marketing & BusDev at OpSIS MPW Silicon Photonics foundry. President/COO of LuxSonar Semiconductors ( Cirrus Logic) and co-founder & Director of Product Marketing of NexGen, a X86-compatible CPU company sold to AMD - now the X86 product line. Brad also has ~15 years in technology market research as Vice president of the Computer Systems group at Dataquest/Gartner; VP/Chief Analyst at RHK and Light Counting networking research firms. Brad started his career at Digital Equipment near Boston with the VAX 11/780 and has served as CEO, president/COO and on the board-of-directors of three start-up companies. Brad has a BSEE degree from the University of Massachusetts; MBA from University of Phoenix and holds 2 optical patents

Comments are closed.