All posts by Brad Smith

About Brad Smith

Brad is the Director of Marketing at Mellanox, based in Silicon Valley for the LinkX cables and transceivers business focusing on hyperscale, Web 2.0, enterprise, storage and telco markets. Recently, Brad was Product Line Manager for Intel’s Silicon Photonics group for CWDM4/CLR4 and QSFP28 product lines and ran the 100G CLR4 Alliance. Director of Marketing & BusDev at OpSIS MPW Silicon Photonics foundry. President/COO of LuxSonar Semiconductors ( Cirrus Logic) and co-founder & Director of Product Marketing of NexGen, a X86-compatible CPU company sold to AMD - now the X86 product line. Brad also has ~15 years in technology market research as Vice president of the Computer Systems group at Dataquest/Gartner; VP/Chief Analyst at RHK and Light Counting networking research firms. Brad started his career at Digital Equipment near Boston with the VAX 11/780 and has served as CEO, president/COO and on the board-of-directors of three start-up companies. Brad has a BSEE degree from the University of Massachusetts; MBA from University of Phoenix and holds 2 optical patents

Mellanox Demonstrates New 400G DR4 and 200G SR4 Optical Transceivers with High Data Reliability at ECOC’19

Mellanox is demonstrating new LinkX products as well as showcasing the full line of 100/200/400G cables and optical transceivers at the European Convention for Optical Communications (ECOC) Sept 21st in Dublin, Ireland.

Here’s a look at what to expect at our booth:

  • An active DACs, QSA56 port adapters and 400G DR4 transceivers announcement:
    • Active DAC cables that extend the length of 200GbE/HDR InfiniBand to 3 and 4 meters
    • QSA56 port adapter enables inserting any SFP cables and transceivers into a QSFP56 switch or network adapter port and supports devices reaching from 0.5m to 10km.
    • 100G SFP-DD 2x50G PAM4 DAC cables
    • 400G DR4 QSFP-DD transceiver capable of 500-meter reach with single-mode fiber
  • A showing of our full line of 100G, 200G and 400G DAC and AOC cables and transceivers
  • Live demos with
    • 400G SR8 OSFP 8-channel, 100-meter multi-mode transceiver
    • Ixia, A Keysight Business, new A400GE-QDD test system with Mellanox’s new 400G DR4 and 200G SR4 transceivers. a live demonstration will exhibit better than IEEE specified FEC (Forward Error Correction) error density and BER 1E-15 performance.

End To End with Mellanox

Mellanox LinkX products are designed to work seamlessly with Mellanox 100G/200G Spectrum switches and ConnectX-5/6 network adapters. We are one of the few companies that builds its own switch, adapter and transceiver silicon as well and full systems, cables and transceivers.

All Mellanox cables and transceivers are designed for very low Bit Error Ratios (BER) of 1E-15, about 1,000 times better than the IEEE industry standard. Error-free connectivity for both Ethernet and InfiniBand is particularly important for latency sensitive applications, such as HPC networking, storage, artificial intelligence and hyperscale computing.

Mellanox is a leading network adapter supplier for servers and storage and recently announced shipping 1 million ConnectX network adapters in one quarter to hyperscale, enterprise, and HPC customers.

See LinkX cables and transceivers, Spectrum switches, and ConnectX network adapters in action in Mellanox booths: CIOE Hall 1, 1A22-1 and ECOC Hall 1, booth 53.

European Convention for Optical Communications (ECOC) Sept 21st in Dublin, Ireland.

Supporting Resources: 

You can find more technical details about Mellanox DAC, AOCs cables, transceivers and ICs:

Contact your Mellanox sales representative for availability and pricing options and stay tuned to my blog more interconnect news and tips.

Sign up for free ebook download here:LinkX Cables and Transceivers ebook

Mellanox Active Optical Cable (AOC) for 200G HDR, also known as “H-Cables”

Mellanox Introduces 200G InfiniBand H-Cable Active Optical Cables Cross-Connect Splitter

The Mellanox LinkX Interconnects group introduced a new type of Active Optical Cable (AOC) for 200G HDR (High Data Rate) InfiniBand systems called the “H-Cable”. Named because of its shape that enables cross connecting up to four InfiniBand switches (or ports) together using – a single cable!

These unique active optical cables enable saving considerable CAPEX (CAPital EXpense) and OPEX (OPerating EXpense) by:

  • Reducing the number of layers of InfiniBand switches in large configurations
  • 75% reduction in number of AOC cables compared to implementing via 100G EDR AOCs
  • Frees up four expensive switch ports for other uses

What’s the Magic?

By using 200G HDR switch and interconnects but splitting each port into two 100G HDR100 links.

  • The HDR link consists of 4 channels at 50G PAM4.
  • HDR100 uses 2 channels of 50G PAM 4 for 100G.

Not only does the port splitting enable a whopping 80-ports of 100G HDR100 but also the H-Cable enables reducing the number of actual ports needed to accomplish a cross connect of up to four switches from 8-ports and four AOCs using individual EDR AOCs to 4-ports and one cable using the new 200G cross connect H-Cable. This approach saves both switch ports and transceiver ends which are expensive not to mention power and cabling costs.

This unique AOC cable is based on four 200G HDR “SR4-like” multi-mode transceivers. Each transceiver consists of four lanes of 50G PAM4. Two of the lanes are grouped together to form a 100G link called HDR100 (2x50G PAM4).  The 100G-based HDR100 links are cross connected inside the AOC cable. (See diagram below)

The SB7700-Series EDR InfiniBand switches offer 36-EDR 100G ports. Mellanox’s new 1RU Quantum 8700-series 200G HDR-based switches offers 40-ports of 200G HDR or 80-ports of HDR100 when using 1:2 splitter cables – 80-ports of HDR100 in a single 1RU chassis!  This enables 40-ports down to servers and storage and 40-port up to other switches.

H-Cable Feature Summary:

  • A single Active Optical Cable (AOC) cable in a cross connect configuration
  • Based on 200G HDR InfiniBand transceivers – similar to a 200GbE SR4 Ethernet transceiver
  • 4-channels of 50G PAM4 modulation per transceiver-end
  • QSFP56 form-factor
  • Multi-mode VCSEL-based
  • 5Watt power dissipation per transceiver-end (typical).
  • From 3 to 30-meter lengths of Low-Smoke, Zero- Halogen (LSZH) OM4 fiber

In the figures below, it is clear the benefits of the H-Cable AOC approach in simplicity, increased reliability, lower maintenance and fewer cables and switch ports.

Single H-Cable and 4-port HDR Switch Configuration

Similar Configuration Using Four 100G EDR AOCs and 8-EDR Switch Ports

 

What is PAM4?

PAM4 is a new electrical and optical modulation technique becoming popular in both 200G and 400G networks.  PAM4 that enables two data bits to be sent per clock instead of the traditional 1-bit per clock by creating four levels in a signal (00,01,10,11) instead of just two (1,0).  This doubles the net data transfer rate while keeping the clocking optics and electronics at the same 25G rate which is considerably less expensive than doubling the clock rate to 50G.

What is QSFP56?

QSFP56 is slightly longer than the QSFP28 form-factor at the optical connector end. The extra space is needed to enable more electronics to be included and increased thermal dissipation up to 5 Watts.  The Mellanox QSFP56 HDR transceivers and transceivers used in the H-Cable typically dissipates about 4.5 Watts. Also, the “56” means the form-factor shell supports the higher frequency components at 56G and suppressing the associated electromagnetic noise from getting out. The QSFP56 pluggable transceivers uses the same 8-fiber MPO-12 optical connector as the QSFP28 and can be used in installed infrastructures of OM3 and OM4 multi-mode fiber and MPO-12 optical connectors without issues.

200G Transceivers

The HDR and H-Cable AOCs are based on the same 200G transceiver design as the Mellanox individual connectorized 200G HDR transceiver – which is very similar to the Ethernet 200G SR4 transceiver as well. Mellanox designs, builds and tests its 200G transceivers and AOCs to ensure above average high quality and reliability.

Mellanox 200G 4-channel QSFP56 Transceiver for HDR InfiniBand and Ethernet

Is InfiniBand HDR100 the Same as EDR? – Both Transfer at 100G!

In short, no!  The modulation schemes are different.

EDR uses 4-channels of 25G NRZ modulation – this is the traditional two-level 1,0 modulation scheme.

HDR100 uses 2-channels of 50G PAM4 – this is a four-level 00,01,10,11 modulation scheme.

Plugging an EDR AOC into an HDR switch will result in EDR 100G line rates. All good.  The 50G PAM4 SerDes in the switch will convert to 25G NRZ.  Switch backward compatibility.

But the reverse is very different! Plugging a 100G HDR100 AOC into a 100G EDR switch port will convert the 100G HDR100 into 50G! How? The EDR switch can only see two-levels in the four level 50G PAM4 HDR100 AOC. So, it converts the 50G PAM4 into 25G. But that’s not all!  The HDR100 has only 2-lanes, (2x50G PAM4) not four like EDR (4x25G). So, 2-lanes x 25G = 50G.  It works … but at half the rated speed. And 50G is not really an InfiniBand standard speed.

H-Cable Used in Systems

The H-cable provides cross-connect capability between ToR (Top of Rack) and Spine switches. It enables a single HDR InfiniBand QSFP56 switch port to operate as two 100G HDR100 ports but physically uses only one port. It creates a non-blocking fat-tree topology thereby doubling the number of ports available compared to linking with individual 100G EDR AOCs. This eliminates a third-level of switching, which is needed for traditional large cluster designs of 1600 up to 3200 HDR100 ports, and results in significant CAPEX and OPEX savings. Additionally, by reducing a layer of switching systems, the cluster latency is reduced, which results in improved system performance.

200G ToR Switches and Spine Switches with FAT-tree Topology

Mellanox LinkX H-Cable Benefits:

  • Eliminates the need for a 3rd layer of switches in large clusters deployments; >1600 node clusters
  • Enables linking up to 4 InfiniBand switches ports with a single cable with links running at HDR100 speeds
  • Saves 50% of the transceiver-ends and 50% of port usage over using 4 EDR AOCs with 8 transceiver-ends
  • Single 4-transceiver cable using 4-switch ports versus 4 AOCs with 8 EDR transceivers and 8-switch ports
  • Fewer cables and transceiver-ends = simpler configurations and increased reliability from fewer components
  • Frees up 4 switch ports for other uses ($cost/port x 4 ports)
  • Four x 4.5W HDRs verses 8 EDRs at 2.4W saves ~ 1 Watt over all
  • Reduction in switch layers yield lower system latency delays
  • Increase in system reliability from using fewer optical transceivers and fiber links

Summary

Mellanox’s new InfiniBand H-Cable delivers a series of financial and configuration benefits as well as increasing overall system reliability, simplicity and system latency.  Contact Mellanox sales for more information.

Supporting Resources:

Contact your Mellanox sales representative for availability and pricing options and stay tuned to my blog more interconnect news and tips.

Sign up for free ebook download here: LinkX Cables and Transceivers ebook

 

Mellanox Supports the Open Eye MSA With Long History of Designing Analog Optical Transceiver Electronics

June 2, 2019  Brad Smith, Director of Marketing, LinkX Cables and Transceivers

Summary

  • Early first-generation 50G PAM4 lasers and ICs had some “issues” causing engineers to resort to using powerful DSPs to clean up the signals in the receiver.
  • Next generation 50G PAM4 lasers and ICs have improved significantly raising the question, “Do we still need high-cost, power and latency consuming DSPs?”
  • Returning to traditional analog electronics in the receiver CDRs yields benefits of lower cost, size, complexity, power and latency and return to a simpler, lower cost test procedure.
  • IEEE is holding firm on DSPs, but a splinter group formed that believes DSPs are overkill.With the high number of optical transceivers used in data centers, there are tremendous pressures to lower the CapEx (product cost) and the OpEx (operating costs such as power consumptions) and maximize performance in reducing latency delays in the systems.The top 30 most important items on the transceiver buyers list of requirements are:#1-27   Low cost#28      Low power#29      Low latency#30      InteroperabilityThe Open Eye MSA is attempting to target all of these factors at once. All of these factors will be critically important with future co-packaged optics with chiplets and next-generation switch ICs for 25Tb and 51Tb.

Open Eye MSA Participants

The IEEE holds standards meetings that last several years in duration to get all parties to agree, then the standard is set and those that didn’t get their way or want to do something different go off and form splinter groups. Interestingly, sometimes these groups hit upon the most popular products such as the 100G PSM4, CWDM4 widely adopted in hyperscale systems.

Founding members of the Open Eye group are MACOM and Semtech.

Promoters and Contributors as of this writing include:

Analog-based receivers offer lower cost, lower power and lower latency – all benefits derived from not using a DSP and simpler testing techniques particularly dropping TDEQ testing in favor of more traditional Open Eye techniques, hence the group’s name.

The group plans to start with 53G single-mode optics, then multi-mode and later to target 100G PAM4 per lane for the next generation 400G and 800G transceivers. The plan is to be fully compatible with existing data center and enterprise infrastructure as the industry doesn’t need two versions of DR4, FR4, LR4 etc. transceivers. Making the interoperability happen will be challenging.  The goal is to have the first specifications out by September ECOC and CIOE trade shows.

Key product targets would be the 50G PAM4-based multi-mode 100G/200G/400G SR2, SR4, SR4.2, SR8 and single-mode 400G DR4 (500m), FR4 (2 km), LR4 (10 km).

Signal “EYE” Diagrams

On an optical scope, overlapping displaying a pulse that rises from digital 0-to-1-to-0 with a signal starting from a 1-to-0-to-1 yields what engineers call a signal eye diagram.  So-called because the pattern looks like a pair of eyes between two rails.

 

Signal eye diagram

Signal eye diagram

NRZ Signal with Open Eyes and Distorted Eyes

This displays any noise, reflections, rise and fall time edges, jittering and other parameters that distort transmission signals. The goal is to have wide open eyes that clearly distinguish each signal bit and approximate the square pulses. Noise makes this determination harder for the electronics and increases the probability of error bits.

PAM4 signal modulation

When PAM4 modulation first came into play for optical transceivers, it created a lot of controversary in the transceiver engineering and standards community on the viability of implementing it – especially at higher speeds such as 100G PAM4. PAM4 modulates the signal to four levels in the same time and amplitude space instead of two as with NRZ, hence is much harder to detect the data signals from the noise.

PAM4 is like Playing Tennis with 4 Nets!

To simplify the problem, think of signaling as a tennis game. With older NRZ all one had to do was hit the ball over the net. One side was a zero, the other was a one. Didn’t matter how far the ball went over the net 1” or 100 feet. Just get it over the net and it’s a 0 or a 1.

With PAM4, it’s a tennis game with 4 nets (4-signal levels)!  And you can’t stray beyond the individual net area! Additionally, the tennis court is the same size, so everything is compressed. This makes signal detection much more difficult as noise can interfere much more.

PAM4 with DSPs

Digital Signal Processors (DSPs) have been used to convert the analog signal to digital and compute mathematics in the digital processor (a type of microprocessor) to clean up the signals before converting back to analog.  This gives the programmer a lot of variable and math to play with. DSPs are very accurate but requires a DSP chip in the receiver which introduces higher cost, power and latency delays.

Today, early samples of 200G and 400G transceivers almost all use DSPs, but the debate is starting to rage on over whether DSPs are really needed going forward given the advancements in the quality of second generation of core lasers and other control ICs. As there are a hundred different variables to consider on thing for sure is this debate will go on for quite a while and even more so at 100G PAM4 where things are expected to get very hard.

The Open Eye MSA wants to develop transceiver specifications that interoperate with IEEE and DSP-based transceiver specifications but using analog electronics at lower cost, power, and latency and still maintaining interoperability with other devices, thereby meeting the buyers 1-30 criteria!

To be fair, a second-generation of DSPs are on the horizon that will offer perhaps lower power form smaller IC geometries and maybe a lower cost than the current generation, but the latency is likely to still exist the DSP is still a digital computer and the analog signals need to be converted to digital back and forth. Also, the next generation cost and power targets are still likely to be significantly above an analog approach as the DSP transceiver approach still requires an additional chip, additional power, latency and cost over the analog approach.  The gaps may shrink but we’ll have to see how much as new ICs roll out and what the actual Open Eye implementation come out with.

T-Deck-Q (TDECQ)

Moving from non-return-to-zero (NRZ) with digital 1,0s to PAM4 brought on a new set of test techniques.  The goal is to have nice clean open eyes. However, measuring the eye diagram became very complicated at 50G PAM4 as three eyes were compressed into the same voltage height as with NRZ. So, a new technique was introduced called TDECQ. TDECQ testing forces almost of the equalizations to be done in the DSP receiver instead of being split between the transmitter and receiver. Additionally, some companies have found that while the device may pass the TDECQ test, it can still fail in the field. Lastly, this type of testing is complicated and more expensive than the traditional Open Eye method the MSA wants to return to now that newer higher-quality lasers and control ICs are available.

MSA Time-to-market is essential

Many standards bodies with large groups can take years to come to a consensus – sometimes outliving the product usefulness of the product itself before the next generation kicks in. Both the hyperscalers and HPC communities are pushing very hard on the next generations.

The Open Eye group plans to have a small core group to develop the baseline draft specification in order to product the final specification as soon as possible and not get delayed with a large group as minimal time-to-market is essential. Once initial specifications are established the group will open up to a wider audience of contributors.

MSA Open Eye: Minimal Changes & Large

The change is in the transceiver electronics circuitry and should not impact the other areas of the optical and electrical infrastructure.

  • Fibers: Compatible with existing single-mode and multi-mode fiber infrastructures and link budgets
  • Optical connectors: compatible with existing MPO-12, MPO-16, LC Duplex, MOP/APC and the new Corning MDC and Senko CS/SN optical connectors for 400G
  • Interoperability: The group believes that both analog and DSP-based transceivers should be able to interoperate if both are tested to the same test standards.
  • Optical only: Both DSP and analog types of circuitry are used on the optical side so should not interfere with the electrical side in switches or network adapters.
  • Testing: Changing testing requirement from using TDECQ to testing via Eye Opening technique equals simpler and fast testing and reduced costs since optical testing is a major contributor to cost.
  • Re-timer usage: Enables using retimers
  • Package indifference – supporting QSFP/QSFP-DD, SFP/SFP-DD, and OSFP/DSFP

The Open Eye group rough estimates with the following benefits:

  • Significantly lower latency – no A/D and D/A conversion or DSP digital computations. Low latency is critical in High Performance Computing and hyperscale application such as maps, search and now becoming very popular with Ethernet AI applications. Estimates of 40-100X lower latency.
  • 10%-25% lower power –no A/Ds, D/As or DSP electronics – fewer circuitry, less power
  • Simpler module construction – besides the DSP, other components could be deleted
  • Simpler manufacturing testing – optical testing is a major cost component in optical transceivers manufacturing accounting for 35-50% of the total cost. OE modules should be faster to test hence lower cost.

Mellanox’s Long History of Building Analog Transceiver Electronics

What is not well known is Mellanox designs and builds its own single-mode and multi-mode transceiver ICs using an 8- and 12-inch wafer, fabless manufacturing model. Additionally, we design and build our own DAC, AOC and transceivers as well. We have been designing our own transceiver ICs since the acquisition of IPtronics in 2013. We use these ICs in our SFP and QSFP transceivers and AOC cables and splitters.

Steen Gundersen, Vice President of LinkX cable and transceiver products said, “By taking advantage of the inherent benefits of fully analog module architectures enabled by the Open Eye MSA, Mellanox and our industry peers are helping datacenter and High-Performance Computing operators achieve the low latency necessary for latency-sensitive computing applications.  Low-power and low-cost are two key attributes every transceiver buyer demands.  Low-latency for both Mellanox’s Ethernet and InfiniBand product lines is critically important in High Performance Computing and Ethernet hyperscale systems.  Mellanox has a long history in designing, fabless manufacturing, and selling analog transceiver ICs.”

OFC 2019 Mellanox cable and transceiver display with switches & adapters

Key Take-Aways:

In the next few months, both next generation DSP and new analog receiver designed transceivers will appear and the market place will make its call with its dollar votes based on the real value proposition. Both camps will argue strongly for their respective cases.  Promises of “tomorrow’s IC will be better” will be bantered back and forth. The marketing “food fights” will start up! The real benefits versus estimated benefits will sort themselves out. No doubt, the debate will not end soon. But there appears to be a significant gap in cost, power, latency and simplicity between the two groups and these elements are key in the buyer’s minds. Interoperability and whether the Open Eye approach can be used at 100G PAM4 will likely be at the forefront.

Visit: www.openeye-msa.org

About Mellanox & LinkX™

Mellanox Technologies, Ltd. (NASDAQ: MLNX), a leading supplier of optical transceivers and high-performance, end-to-end smart interconnect solutions for data center servers and storage systems. Mellanox offers a full line of cables and transceivers for hyperscale and datacenter applications:

  • 25G/50G/100/200/400Gb/s DAC & AOC cables and transceivers
  • 200G InfiniBand Quantum switches – 40 ports of QSFP56
  • 200G Ethernet Spectrum-2 50G PAM4 switches from 16-to-64 ports
  • 200G ConnectX-6 network adapters with 2 ports of 200Gb/s QSFP56 or SFP56
  • Optical transceiver IC components TIAs/Laser Drivers, etc.

Supporting Resources: 

You can find more technical details about Mellanox DAC, AOCs cables, transceivers and ICs:

Contact your Mellanox sales representative for availability and pricing options and stay tuned to my blog more interconnect news and tips.

Sign up for free ebook download here: LinkX Cables and Transceivers ebook

Mellanox showing 100G and 200/400G Cables and Transceivers at OFC 2019

Mellanox LinkX at OFC19 – 100 and 200/400G Cables and Transceivers

The Optical Fiber Conference was in San Diego Mar 4-7th with 16,000+ attendees and 700 booths. OFC is LinkX’s biggest show of the year as it focuses on future products and was all about 200G/400G using with 50G PAM4 modulation.

The Mellanox LinkX team showed an incredible 6 live demonstrations!

  1. 400G DR4 – 500m single-mode transceiver in QSFP-DD form-factor
  2. 400G SR8 – 100m multi-mode transceiver in OSFP
  3. 200G SR4 – 100m multi-mode in IB & EN running in 2 servers w/ConnectX-6 dual QSFP56
  4. 100G SR2 – 2m DAC cable running in 2 servers w/ConnectX-6 dual SFP-DD
  5. Keysight/Ixia chassis with 200G & 400G DACs, AOCs, transceivers in QSFP-DD/QSFP56
  6. In the Ethernet Alliance Booth running 400G SR8 in OSFP

Mellanox 100 and 200/400G Cable and Transceiver Demonstrations at OFC 2019

Six Live 100/200/400G Cable and Transceiver Demonstrations

Mellanox 100G 2x50G PAM4 SFP-DD Network adapter

100G Product Line

Also shown was our complete line of 25G/50G/100G DAC and AOC cables and optical transceivers with our 25/100G-based network adapters, and network switches (underneath in picture below). Mellanox designs and build its own InfiniBand and Ethernet switches and adapter ICs and manufactures the adapter boards and switch chassis as well. In addition, Mellanox’s LinkX team designs multi-mode and single-mode transceiver ICs and builds its own DAC, AOC and transceiver products.  Unlike many other suppliers, Mellanox designs and manufactures the entire switch-cable/transceiver-adapter link. This enables us to provide outstanding product performance by matching circuit internals to optimize the signal integrity and guarantee optimal performance. One of our key selling points is a BER rating for cables and transceivers of 1E-15 pre-FEC compared to competitors post-FEC.

Mellanox LinkX Cables and Transceivers, Spectrum Switches and ConnectX Adapters at OFC 2019

LinkX Cables and Transceivers, Spectrum Switches and ConnectX Adapters

Keysight/Ixia Demonstration

Keysight/Ixia was kind enough to lend us one of their new QSFP-DD test chassis where together we demonstrated:

  • 200G SR4 multi-mode transceivers
  • 200G AOC cable
  • 400G DAC cables
  • 200G Active DAC cable

Keysight/Ixia AresONE Demonstration with Mellanox LinkX 200/400GCables and Transceivers

Keysight/Ixia AresONE Demonstration with LinkX 200/400G Cables and Transceivers

The screen shot below shows amazing Pre-FEC Bit Error Ratio numbers that most cables and transceiver suppliers achieve Post-FEC processing in the switch! This demonstrates the high manufacturing quality of the parts – one of Mellanox’s key value-add.

This is a carry over from our cables and transceiver manufacturing for the InfiniBand market where FEC introduces way too much latency delay in the sytem to be used. InfiniBand EDR switches have an 80-ns port-to-port latency and FEC needs about 120ns to compute. We apply these high-quality manufacturing techniques to the Ethernet product line to offer the best BER ratings in the industry – important when you are computing at 100G and any errors can jamb up the system and induce retrnsmission delays.

Keysight/Ixia AresONE Demonstration of LinkX Cables and Transceivers

Key Takeaways:

100G cables and transceivers are in full swing in the market and rapidly being adopted in hyperscale, HPC, enterprise and storage.  Mellanox offers a complete line of switches, PCIe and OCP form-factor network adapters including cables and transceivers for 100G and now 200G that are shipping. Two times the bandwidth at only a slight premium in price above 100G systems and links.  In spite of all the conference and market noise, 400G complete end-to-end systems are still a way off and will see more of the pieces come together in late 2020 and 2021.

 Supporting Resources:

You can find more technical details about Mellanox DAC, AOCs cables, transceivers and ICs:

Contact your Mellanox sales representative for availability and pricing options and stay tuned to my blog more interconnect news and tips.

Sign up for free ebook download here: LinkX Cables and Transceivers ebook

 

 

 

 

 

 

 

Mellanox 200G Cables and Transceivers, LinkX, Debuts at OCP

The First Showing of 200G End-to-End Interconnects

Mellanox showed the LinkX 200G Cables and Transceivers Display at Open Compute Project in March in San Jose, CA.  LinkX is always a crowd pleaser and constant source of people taking pictures – especially the 100G and 200G cables and transceivers that were on display.  OCP was the first showing of the complete the end-to-end offering of 200G DACs, AOCs and multi-mode transceivers with the ConnectX-6 dual-port 200G Ethernet adapter and the new Spectrum-2 200G Ethernet switch.

LinkX 50, 100 and 200Gbe Cables and Transceivers

Single 50G-to-Quad 50G Splitter DAC and AOC Cables and Transceivers

New Spectrum-2 Ethernet Switch

Offering 32-ports of 200G QSFP56 Ethernet, the new Spectrum-2 switch is Mellanox’s next generation of end-to-end Ethernet interconnect systems and offers 128 SerDes.

Mellanox 200G 32-Port SN3700 Spectrum-2 Ethernet Switch

New 200G 32-Port SN3700 Spectrum-2 Ethernet Switch

Splitting Hairs and Ports!

The Spectrum-2 switch supports 128 SerDes in the Switch IC.  By using the LinkX 200G-to-Dual 100G splitter DAC or AOC cables or even transceivers with splitter fiber cables (shown above), the Spectrum-2 switch can support 64-ports of 100G QSFP56 (2x50G PAM4) or 128-ports of 50G using 4x50G SFP56 splitter. Note that only the switch ports can be split and not the network adapter ports.

Upwards and Downwards Compatibility

Although, the 100G is based on 2-lanes of 50G PAM4, it is not downward compatible with the 4-channel x 25G NRZ ports.  The Spectrum-2 does support upwards compatibility of using 4x25G NRZ devices into the 200G ports but not the reverse. Because the 50G PAM4 signal modulation scheme is different from previous NRZ generations, its recommended to use all 50G-PAM4-based Spectrum-2 switches, ConnectX-6 network adapters including cables and transceivers.

QSA Port Adapter

Lastly, we recently announced the QSA56 port adapter.  Invented and patented by Mellanox, this device is one of those things in life a data center builder can’t live without!  Sort of like that 3-prong -to-2-prong 110V plug adapter for power cords.  Cheap but when you need it, …..you really need it.

This device enables using just about any SFP56, SFP28 or SFP+ DAC, AOC or transceiver and passing a single-channel through into a quad-channel switch or network adapter port. Hence, the name QSFP-to-SFP Adapter. The QSA 56 has a small EEPROM inside that tells the switch or network adapter what it is but otherwise, the device is passive.

New 50G QSA QSFP-to-SFP Port Adapter – On the Left with SFP56 AOC inserted Part Way

Key Take-Aways:

200G end-to-end switches, network adapters including cables and transceivers are here now and generally available (GA).  Why wait for 400G when you can double your bandwidth today at a small premium price over 100G? Upgrading to 200G from 100G entails a minimum of changes, unlike moving to 8-channel 400G with new form-factors, internal gear boxes, optical multiplexers and new optical connectors not to mention 10-15 Watts of power per device!

———————————————

 Supporting Resources: 

You can find more technical details about Mellanox DAC, AOCs cables, transceivers and ICs:

Contact your Mellanox sales representative for availability and pricing options and stay tuned to my blog more interconnect news and tips.

Sign up for free ebook download here: LinkX Cables and Transceivers ebook

Ixia, a Keysight Business, & Mellanox Demonstrate Live Traffic 200G Transceivers & 400G DAC Cables

Ixia, a Keysight Business, and Mellanox demonstrated together new high-speed products 400G DAC Cables and the AresONE Ethernet at the European Convention for Optical Communication (ECOC) in Rome last month.

Mellanox Demonstrates 400G DAC Cables at ECOC

We demonstrated our new 200Gb/s SR4 multi-mode Ethernet transceiver (orange cable in the photo above) which is also used in our 200G AOCs as well.

  • Base on 4-channels of 50G PAM4
  • Enables up to 100-meter reach using OM4 multi-mode fiber
  • Based on QSFP56 form-factor
  • Utilizes standard MPO optical connector
  • Supports both 200G Ethernet and InfiniBand HDR

Also, demonstrated several of our new 400G DAC cables based on 8x50G PAM4 using the QSFP-DD form-factor (black cables).  The dual cable form-factor enables easier handling and a tighter bend radius.

Ixia Demonstrates AresONE Ethernet test solution

Thananya Baldwin, Vice President, Strategic Programs, Ixia Solutions Group, Keysight Technologies demonstrated their new AresONE Ethernet test solution enabling 8-ports of QSFP-DD testing at rates of 200G or 400G with simulated Ethernet traffic totaling 3.2 Tb.

Based on Ixia’s first generation, field-proven K400 test platform, the award winning AresONE enables full line-rate traffic generation functionality for transmit, receive, and capture with real time analysis of received traffic. The Ixia-developed industry-leading intellectual property for the critical elements of 400GE: MAC, PCS, FEC, and FEC error injection and statistics enable faster test feature development and response times as industry standards evolve.

Facilitating RFC benchmark testing, stress testing, and hardware/ASIC bring-up in high-port-count test beds. It can be used for optics and cable qualification at 50G, 100G, 200G and 400G rates, interoperability, and functional test. With “IxNetwork”, Ixia offers the broadest and highest performance Layer 2/3 routing protocol emulation coverage and performance available in the industry.

Ixia Test Statistics GUI & Mellanox Transceivers and DACs

After 40 mins of testing at 200GE line rate, Mellanox’s 200GbE SR4 transceiver test results (column B & C below) show a Pre-FEC Bit Error Rate (BER) of ~5.7e-12 and 1.7e-13 which are well above the IEEE standards. Many competitor transceivers just barely meet the IEEE specifications illustrate a 1e-13 BER after FEC is computed.

Ixia’s AresONE GUI also monitors the symbol error density (row 10 – 26 below). Mellanox’s 200G SR4 transceivers shows only maximum of 1 corrected symbol on any codeword over 6-7 hours of run time. This is far below the KP4 FEC’s ability to correct up to 15 symbol errors in a code word.

A new parameter, Frame Loss Ratio (FLR), was introduced in the IEEE specifications for 200GE (row 8). FLR is equivalent to number of 64-byte packets lost due to uncorrected FEC symbol errors. We are seeing 0 for FLR because there were 0 uncorrectable FEC code words reported by the test gear (row 27).

This low BER is very important in high-speed networks running at speeds like 200G as errors can quickly pile up and burden the system with correcting up to 15 FEC symbol errors.  This increases the link latency as the host switch or NIC has to calculate the error correction algorithm to correct errors and if errors become excessive, it may force a retransmission. It also can interrupt the state of the switch machine and force other delays.

 

The AresONE colored LEDs indicate the Tx and Rx status, FEC and speed of the links. Blue equals 200G and purple equals 400G.

One 400G DAC cable was damaged during shipping.  See if you can tell which one it is from the display screen!

 

 

Answer: Col E&F showing the error water fall with up to 9 error corrections.

About Mellanox & LinkX:

Mellanox Technologies, Ltd. (NASDAQ: MLNX), a leading supplier of optical transceivers and high-performance, end-to-end smart interconnect solutions for data center servers and storage systems. Mellanox offers a full line of 25/50/100/200Gb/s Ethernet cables and transceivers for hyperscale and datacenter applications and 100Gb/s EDR and 200Gb/s HDR for InfiniBand.

 

About Keysight Technologies:

Keysight Technologies, Inc. (NYSE: KEYS) is a leading technology company that helps enterprises, service providers, and governments accelerate innovation to connect and secure the world.  Keysight’s solutions optimize networks and bring electronic products to market faster and at a lower cost with offerings from design simulation, to prototype validation, to manufacturing test, to optimization in networks and cloud environments. Customers span the worldwide communications ecosystem, aerospace and defense, automotive, energy, semiconductor and general electronics end markets. In April 2017, Keysight acquired Ixia, a leader in network test, visibility, and security. More information is available at

 

Supporting Resources:

Mellanox Wins CIOE Trade Show Technology Innovation Award for 200Gb/s Cables and Transceivers

Mellanox Wins CIOE Trade Show Technology Innovation Award for 200Gb/s Cables and Transceivers 

Mellanox won the Technology Innovation Award at the CIOE 2018 China International Optoelectronic Exposition, in Shenzhen, China last week for our innovations in high-speed interconnect product technologies and achievements of beginning shipments of next-generation 200Gb/s Ethernet and HDR InfiniBand optical transceivers, active optical cables (AOCs) and direct attach copper cables (DAC). While many companies are out talking about 200G and promising 400G, Mellanox has begun shipping products for 200G HDR InfiniBand and 200GbE Ethernet.

CIOE is a huge exposition held in Shenzhen each year with nearly 90,000 people visiting in the 3-day conference and trade show – up more than 20% from last year! Interest in data center systems has skyrocketed!

Low-latency Ethernet and InfiniBand interconnect are the hottest thing in high-speed computing these days, especially for Artificial Intelligence deep learning model development.  Low-loss DAC cables enable reducing or disabling Ethernet FEC entirely saving upwards of 120ns for each direction in critical server links at lengths < 2 meters. InfiniBand standard practice is to run without FEC entirely for minimum latencies throughout the system. Who would think that the lowly DAC cables could have such a large system impact on latency performance at the system level!

At CIOE, we showed displays of our 200G QSFP56-based DAC and AOC cables and SR4 multi-mode transceiver for Ethernet and InfiniBand networks. Also shown was our new low-loss 25/50/100G DAC cables that enable low-latency switching and cable lengths up to 5-meters. These new cables enable support the low-loss IEEE CA-N and CA-L specification.

The 4-channel QSFP56 has more EMI shielding to support 50G PAM4 signaling, a higher thermal dissipation capability of <5Watts compared to 3.5W QSFP28 and is about a 7mm (0.25”) longer.

Key Features

  • Speed selectable
    • 200G cables and transceivers can are speed selectable from 50G PAM4 to 25G and 10G line rates supporting 200G, 100G, and 40G quad channel configurations
  • Split Configurations
    • 200G cables and transceivers can be split into two 100G (2x50G PAM4) ends using DAC and AOC splitter cables or fiber splitters with SR4 transceivers
  • Low-loss cable material enables DACs to:
    • Minimize FEC latencies
    • Maintain the 5-meter (16-feet) lengths for 100G and 3-meters (9.8-feet) for 200G
200G cables and transceivers and low-loss DAC display at CIOE

200G cables and transceivers and low-loss DAC display at CIOE

LinkX™ Sales and Marketing APAC Team at CIOE

LinkX™ Sales and Marketing APAC Team at CIOE

Key Take Aways:

  • 200Gb/s for Ethernet and InfiniBand is here
  • Low-cost, low-loss, 3- and 5-meter DAC cables can save latency and span multiple racks
  • 200G 4x50G PAM4 cables and transceivers are split-able and speed selectable

 

Come See Mellanox LinkX™ cables and transceivers at ECOC!

See Mellanox at ECOC 2018 with Keysight Technologies showing live demonstrations of our 200G SR4 transceiver and 400G DAC cables September 24-26, Rome, Italy, booth 432.

LinkX™ is the interconnect product line of cables and transceivers and part of the Mellanox “end-to-end” ecosystem including our new Spectrum2™ 200GbE Ethernet and Quantum™ InfiniBand HDR systems and ConnectX™-6 network adapters.

 

About Mellanox & LinkX:

Mellanox Technologies, Ltd. (NASDAQ: MLNX), a leading supplier of optical transceivers and high-performance, end-to-end smart interconnect solutions for data center servers and storage systems. Mellanox offers a full line of 25/50/100/200Gb/s Ethernet cables and transceivers for hyperscale and datacenter applications and 100Gb/s EDR and 200Gb/s HDR for InfiniBand.

 

Supporting Resources:

Sign up for free ebook download here: LinkX Cables and Transceivers ebook

See Mellanox LinkX™ on Display at 2 of the 3 Biggest Interconnect Shows of the Year

ECOC 2018 conference September 24-26, Rome, Italy, booth 432.

CIOE 2018 China International Optoelectronic Exposition, Shenzhen, China, September 5-8 booth 1A22-1

Mellanox will showcase the complete LinkX™ product line of 100G, 200G and 400G cables and transceivers products.  We will be showcasing live demos of our 200G SR4 QSFP56 PAM4 transceiver and DAC cables and 400G QSFP-DD DAC cables with Ixia/Keysight Inc. These cables and transceivers are all part of our end-to-end systems offering and kicks off the start of our shipments for 200GbE Ethernet and 200G HDR InfiniBand cables, transceivers and the beginning of our 400GbE cables and transceiver product line.

Designed and manufactured by Mellanox, the 4x50G PAM4 transceiver uses the QSFP56 form-factor and forms the basis for transceivers and AOC products for Mellanox’s upcoming 200G systems. The 200G DACs and AOCs are also being offered in a 1:2 Y-splitter version for breaking out InfiniBand and Ethernet 200G ports into 100G ports. For example, the recently announced 40-port HDR switch can split the ports into a whopping 80-ports of 100G in a 1RU switch form-factor.

On static display at both ECOC and CIOE, will be our full line of 25/50/100Gb/s cables and transceivers for hyperscale and data center applications including:

  • 25G/50G/100G DAC & AOC cables and SR, SR4, PSM4, CWDM4 and LR4 transceivers
  • DAC and AOC dual and quad breakout cables for 100G to 4x25G and 2x50G
  • 200G DACs, AOCs cables and Y-splitters and SR4 multi-mode transceivers

ECOC Speakers Include:

  1. ECOC2018, paper We1H.3, Wednesday Sept 26, 09.30am, “Scaling optical interconnects beyond 400 Gb/s.”  Paper describing work within EC projects plaCMOS and plasmofab. By Paraskevas Bakopoulos
  2. International Symposium on Hybrid PICs, organized at ECOC2018, September 26, 2018

“Addressing datacenter challenges with hybrid PICs” Presentation on optical switching work in EC project 3PEAT. By Paraskevas Bakopoulos

  1. 6th International Symposium for Optical Interconnect in Data Centres, organized at ECOC2018, September 26, 2018, 15:45. “plaCMOS – Wafer-scale, CMOS integration of photonics, plasmonics and electronics devices for mass manufacturing 200Gb/s non-return-to-zero (NRZ) transceivers towards low-cost Terabit connectivity in Data Centers. An overview of plaCMOS project; by Dimitrios Kalavrouziotis

LinkX is the interconnect product line of cables and transceivers and part of the Mellanox “end-to-end” ecosystem including Spectrum2™ 200GbE and Quantum™ InfiniBand HDR systems and ConnectX™-6 network adapters.

Come See Mellanox LinkX™ cables and transceivers!

Supporting Resources:

New Low-Loss DAC Cables Enables Reduced FEC and Low Switch Latency for Server Links

Offering improved shielding, it significantly reduces the 25G signal losses in the cable. This delivers several key benefits:

  • Swapping the Forward Error Correction (FEC) RS-FEC algorithm in the switch to a low-latency FireCode (BASE-R) FEC saving approximately 30% latency or manually turning off FEC entirely saving upward of 100- 120ns latency in each direction.
  • Use of thinner 30AWG wire in place of thicker 28AWG wire making the cable easier to bend and route in crowded racks, with less weight and increased airflow.
  • Maintains the 5-meter reach at 25G and provides better signal integrity to support future 50G PAM4 signaling for 200G and 400G DAC networks
  • Continues offering pre-FEC Bit Error Ratio (BER) rating of 1E-15, which is about 1,000 times better than the IEEE standard specification of 1E-12 post-FEC

 

Green Cables – that are actually Black…Huh?

We made the cable “greener” meaning lead-free, halogen-free and the cable jacket material switched from PVC to Low-Smoke, Zero-Halogen (LSZH), important when things catch fire in data centers. However, they are still black!  In short, PVC is on its way out as a cabling material.

 

New Pull Tab Design

We are changing the pull tabs styles to reduce air flow restrictions and the tab fluttering vibrations that often happens in racks with high air flow.  Also, we are using different colored tabs for break-out link ends for easier identification.

 New Open-Air Flow, Colored Tabs

New Open-Air Flow, Colored Tabs

 

The new cable material is being used in all our InfiniBand and Ethernet 25G and soon 50G PAM4-based DAC cables for 100G, 200G and 400G.

Mellanox 25G NRZ-based Ethernet DAC Cable Product Line

Mellanox 25G NRZ-based Ethernet DAC Cable Product Line

 

What the Heck is FEC?

In the electrical PHY transmitter portion of a switch port, an Error Correction algorithm in run against the data in the packet before transmission. It is then Forwarded with the data and decoded in the receiver to detect and even correct bit errors in the transmitted stream. Hence, name FEC or Forward Error Correction.

However, constructing and deconstructing FEC take approximately 100-120ns per direction, 60ns to initially calculate at the transmitter and 60ns to compute at the receiver and that is for only one direction of a 2-way link.

The new Phase-3, type-N cables (N=No-FEC) enables automatically configuring to a low-loss FEC (BASE-R) or one can manually configuring the switch to turn off the FEC altogether.

A few nanoseconds here and there times 25 billion bits per second, times 10,000 cables, operating at 24/7 for 5 years can really add up as this area is where all the sever data transfer action is!

 

New IEEE Specs

The new specification created three different types of Cable Assemblies (CA-) based on the overall signal losses.

  • Up to 2 meters: CA-N (No FEC) or reduced FEC algorithms (15.5bB max loss)
  • 5 and 3-meters: we offer two versions supporting either CA-25G-N or CA-25G-L
  • 3-to-5 meters: CA-L (Long) with RS-FEC (Reed Solomon) enabled (22.48 dB max loss)

Mellanox uses only two specs as the Type CA-S (16.48 dB max loss) is for only the 4-meter case which is not very popular.

The new cables don’t save latency themselves but when used in the switch-to-adapter link is where the benefit is derived by reducing or eliminating the FEC delay in the switch link.

 

Low BER Rating More Important than Ever

Unlike many optical transceiver and switching system manufacturers, Mellanox designs and manufactures its own DAC cables as well as optics, network adapters and switching systems. This gives Mellanox incredible control over quality control and cable testing.

Mellanox builds its InfiniBand and Ethernet DAC cables to a bit error ratio of 1E-15 without the use of FEC in the switch. That is one-bit error every 10^15th bits transmitted expressed as 1E-15. The IEEE 25G/100G 802.3 Ethernet spec requires the use of FEC with post FEC ratings of a minimum of 1E-12.  Pre-FEC this computes to 1E-5 which is an incredible number of errors for electronics to deal with and can easily be overloaded forcing repetitive data retransmissions.

The IEEE recently introduced the CA-N new standard for short reach DACs that reduces the FEC requirements for less than 2 meter likes to servers and storage. This is where DAC cable quality becomes incredibly important because there is no FEC to correct any bit errors.

Mellanox uses the same DAC manufacturing techniques for both InfiniBand and Ethernet which enables building Ethernet DACs to the 1E-15 BER pre-FEC rating. Over 1,000 times lower bit error rates than competitors DAC cables at 1E-12.  Lastly, as Mellanox also builds switches, we test every cable in actual switching systems under live running conditions found in harsh data center environments and not simulated with benchtop test equipment and manual operators as many competitors do.

This guarantees the highest DAC cable signal integrity and reliability possible. DAC links are when most of the rack traffic take place between servers, switches and storage so reducing latency is very beneficial as this is the highest value-add traffic in the network.

 

Summary

DAC cables are the lowest-cost, lowest power high-speed interconnect for linking switches to network adapters used in high traffic servers and storage subsystems. Now, with the new cable material, DACs enable building the lowest latency switch links, thinner cables, colored pull tabs and LSZH jackets for the best in-rack DAC cabling infrastructure where the highest value-add date traffic takes place.

 

Supporting Resources:

 

Mellanox Announces Linking over One Million 100Gb/s Ports!

Mellanox, a leading supplier of optical transceivers and high-performance, end-to-end smart interconnect solutions for data center servers and storage systems, announced that its volume shipments of LinkX-brand of optical transceivers, Active Optical Cables (AOCs) and Direct Attach Copper Cables (DACs), have surpassed the one million 100Gb/s QSFP28 ports milestone linking switches and network adapter together.

Already offering one of the industry’s largest portfolio of data center-oriented interconnect products, Mellanox’s expanded 100Gb/s LinkX product family now includes:

  • DynamiX™ QSFP-to-SFP port Adapter (QSA28)
  • 100Gb/s DAC copper cables including 1:2 and 1:4 splitters
  • 100Gb/s AOC cables including 1:2 and 1:4 splitters
  • 25G/100Gb/s optical transceivers for multi-mode 850nm SR/SR4
  • Single mode 1310nm PSM4, CWDM4, and LR/LR4.

 

100G Industry Transition Happening Now!

The transition from 10G/40G to 25/50/100G is well underway. Having started less than two years ago, it has spread rapidly across the HPC and hyperscale community now is hitting system OEMs, Web 2.0, tier-2 hyperscale/cloud builders, mainstream enterprise and storage systems.

The silicon and subsystem manufacturing differences between 10G and 25G line-rate based equipment, cables and transceivers is steadily shrinking and with increased 25G volumes 25G prices are converging in on 10G prices.  Offering 2.5X the speed at less than 50% price premium or less is fueling this transition.

 

Servers Move to 25G/100G

Clearly, the next wave of CPU and GPU based servers are sporting lots CPUs and GPUs per server bay with huge amounts of in-server local storage with large DRAM and NVMe DIMM arrays. Internal “express-lane” busses linking CPUs and GPUs are sprouting out all over the place such as CCIX, NVLink, CAPI, QPI.

These busses are not part of the PCIe bus, but separate busses dedicated to synchronizing CPUs, GPUs and memories. All this is making the need to 100G to the server more popular. In addition to our 4x25G SFP28 splitter cables, some of our more popular cables are the 100G-to-Dual 50G splitter DAC and AOC cables which is used to split a 100G port at the Top-of-Rack switches down 50G ports in the servers with lots of on-board memories or NVME FLASH subsystems.

 

Storage Moves to 25G/100G

For storage, “slow is OK” used to be the mantra as 10ms access hard drives just were not in a hurry to do anything! The hard disk drive was invented in 1953 and recent reports show the shipments are rapidly shrinking by 10-15% per quarter and being replaced by much faster SSDs and NVME Flash systems and NVME-over-Fabrics using 100G links.  Full read/write speeds for only 3-4 NVME drives can saturate an entire 100G DAC or AOC link.

For example, Nvidia’s recent announcement of the first 2 Petaflop system, NVIDIA® DGX-2™ artificial intelligence (AI) Deep Learning (DL) system employs eight Mellanox 100G, dual-QSFP28 100G ConnectX-5 network adapters for a total I/O of 800G-1.6T – for a server I/O! The system has sixteen Tesla V100 GPUs linked via the NVLink2 bus and a whopping 30 Tbyes of onboard NVMe memory.

NVIDIA® DGX-2™ 16-GPU system and 8 100G ConnectX-5 100G Network Adapters

 

The Bottom Line

10G/40G is now in the rear-view mirror and the 25/50/100G interconnect based on 25G NRZ signaling line rates is now mainstream and likely to last well into the next decade.

 

LinkX Cables and Transceivers are Designed & Manufactured by Mellanox

Unlike most other suppliers, Mellanox designs and manufactures its own DAC cables, AOC cables, and many of its optical transceivers – including the laser driver and receiver silicon ICs! This close match of transceiver IC electronics with Mellanox switches and network adapters enables maximum performance at the lowest bit error ratio (BER) and power consumption.

See our recent press release on one million ports.

 

Supporting Resources: