Once again, the temperature kicked up another few degrees in the interconnect debate with HPC Wire’s coverage based on information released by Intel on the growth of Omni-Path Architecture (OPA). According to Intel, the company behind OPA, have been seeing steady market traction. We have always expected Intel to win some systems, same as QLogic in the past or even Myricom years back; however, while I read over the article in detail, I couldn’t help but argue some of their points.
On Market Traction
Intel has seen continued delays in Omni-Path’s production release. We are not aware of any company that can buy any OPA offering in the channel, and OEMs have not released anything.
In the article, there are a number of public wins referenced including National Nuclear Security Administration’s Tri Labs (Capacity Technology Systems (CTS-1) program) and the Pittsburgh Supercomputing Center. The latter was built with non-production parts as they could not delay any further, and we have heard from sources that performance is lacking.
The specific Department of Energy deal with NNSA is part of the commodity track of the DoE labs, which is a set of small systems used for commodity work. It is not the DoE leadership systems, and we know that Lawrence Livermore National Laboratory decided to use InfiniBand for their next leader system – under the Coral project. The DoE did grant the previous commodity deal to QLogic TrueScale a few years ago, and QLogic has made the same noise we are hearing today – that they are allegedly gaining momentum over Mellanox.
Additionally, the CTS program (formally TLCC), enables a second tier of companies and helps labs to maintain multiple choices for technologies. The program results in building a small scale of systems that the labs are using for basic work, not for their major and high-scale applications. The previous TLCC was awarded to Appro and QLogic, and the current one to Penguin Computing and Intel OPA.
On A Hybrid Approach
Omni-Path is the same technology as the old technology, “InfiniPath” by Pathscale which was later bought and marketed by QLogic under the name “TrueScale.” Similar to QLogic with TrueScale, we believe any description of Omni-Path as a “hybrid” between off-loading and on-loading is likely not supported by the facts. Read more about it in my latest post for HPC Wire. You can see the system performance difference in various HPC application cases, such as WIEN2K, Quantum Espresso, and LS-DYNA.
Intel chose to highlight message rate performance, stating “Compute data coming out of MPI tends to be very high message rate, relatively small size for each message, and highly latency sensitive. There we do use an on-load method because we found it to be the best way to move data. We keep in memory all of the addressing information for every node, core, and process running that requires this communications.” While previously Intel claimed 160M messages per second with OPA, they recently admitted it is closer to 79-100M. Mellanox delivers a superior solution with 150M messages per second.
Finally, as of today, Intel has not yet provided application performance benchmarks for OPA, that support details of the article, or offer substance to claims regarding its performance versus Mellanox’s InfiniBand. We have a number of case studies to prove the performance of InfiniBand.
We look forward to seeing what Intel comes out with next.