The HPE Apollo 70 System is a 2U Arm-based platform designed to use Socket Direct™ technology for its network interface. It is purpose-built for High Performance Computing clusters where density, scalability and performance matter. Being able to be deployed as a single 2U system and scale up to meet a variety of HPC workloads means customers have the flexibility to grow their cluster as needed. Overall, the Apollo 70 offers 33% more memory bandwidth than today’s industry standard HPC servers.
The architecture of the Apollo 70 is unique in that it offers a dual socket design, paired with a single PCIe x16 and a x8 slot, each connecting to a different socket. The Mellanox ConnectX-5 100Gb Socket Direct™ OCP adapter is fully optimized to maximize this design, unleashing unmatched performance capabilities of standard server interconnects offering dual x8 connectivity. Using one Mellanox Socket Direct capable network adapter improves latency of the CPU by removing the need to pass all data through CPU 1 to reach the network. In addition to boosting CPU 2 performance, this design maximizes PCIe lanes available to memory and GPUs. Another added benefit is the Mellanox Socket Direct™ enabled adapter presents itself to the server management interface as a single interconnect which simplifies network management.
Socket Direct™ at a glance
Mellanox ConnectX-5 with Socket Direct™ provides 100Gb port speed, even to servers without x16 PCIe slots, by splitting the 16-lane PCIe bus into two x8 connectors. For the Apollo 70, Mellanox and HPE worked together to develop a single OCP card that bridges the two sockets together at the PCIe bus. In other cases, Socket Direct™ can be achieved with a PCIe x8 edge connector and a parallel x8 auxiliary PCIe connection card. Mellanox uses Multi-Host™ technology to allow multiple hosts to be connected into a single adapter by separating the PCIe interface into multiple and independent interfaces. Socket Direct also enables GPUDirect® RDMA for all CPU/GPU pairs by ensuring that all GPUs are linked to CPUs closest to the adapter card and boosts performance on both sockets by creating a direct connection between the sockets and the adapter card.
Real Life Scenario Lab Testing
In a real-life scenario, applications that run on a dual-socket server generate data traversing the CPUs (over the inter-processor communication bus). To receive a more realistic measurement of the network performance, we applied a test load on the inter-processor bus, and then measured the effect of this load on the external data traffic of the server. We took these measurements, while comparing the two types of adapters (standard adapter and Socket Direct™). Figure 4 compares the average latency of the two different adapters. The graph shows that when using the Socket Direct™ adapter, latency is reduced by 80% compared to the standard adapter. This latency improvement is a result of the direct path both CPU sockets take to reach the network and the even distribution of TCP streams between the CPUs.
Figure 5 shows CPU utilization. It is evident that direct access to the network using Socket Direct™ also provides a 50% improvement in CPU utilization. Moreover, the even distribution of TCP streams reduces the average cache miss count on both CPUs versus a standard configuration server, which further improves CPU utilization.
When comparing the servers’ external throughput while applying the inter-processor load (Figure 6), it is evident that by implementing Socket Direct™, the throughput is improved by 16%-28% compared to the standard adapter connection.
Why HPE chose Mellanox
Mellanox ConnectX-5 delivers high bandwidth, low latency, and high computation efficiency for high performance, data intensive and scalable compute and storage platforms. As a leader in the HPC interconnect market, Mellanox offers several advantages to accelerate the Apollo 70 System, including Socket Direct™ and Multi-Host™, as well as the multiple offloads provided by ConnectX-5. Our strong partnership with HPE and open standards like OCP made developing a platform and adapter from the ground up to meet specific HPC workloads demands possible.
Whether using the ConnectX-5 for HPC or Ethernet network connectivity, almost all aspects of data center connections are virtualized today, including networks, network devices, and host interfaces. All network connections are defined by software, enabling any server to readily connect to any network, storage, or service. To accomplish this, Connect-X adapters offer a comprehensive set of network and I/O virtualization features:
Adapter Key Features
Mellanox Socket Direct adapters provide the highest performance and most flexible solution for the most demanding applications and markets. Socket Direct extends server performance and utilization with maximum throughput connectivity. The Socket Direct adapter within a dual-socket server, enables both CPUs to connect directly to the network, delivering lower latency, lower CPU utilization and higher network throughput. With added virtualization features, support for multiple protocols and form factors, the Mellanox ConnectX family provides high-performance and the most efficient network infrastructure. With Mellanox, there is no need to compromiser performance, security, or usability in high-performance virtualized environments.
For more information please see the HPE Quickspecs:
For more information on Mellanox ConnectX-5 Socket Direct please see the Product Brief.
For more information on the HPE Apollo 70, please Click Here.