Mellanox Socket Direct™ Technology Accelerates HPE Apollo 70 System

HPE Apollo 70 System  
Uncategorized

The HPE Apollo 70 System is a 2U Arm-based platform designed to use Socket Direct™ technology for its network interface. It is purpose-built for High Performance Computing clusters where density, scalability and performance matter.  Being able to be deployed as a single 2U system and scale up to meet a variety of HPC workloads means customers have the flexibility to grow their cluster as needed. Overall, the Apollo 70 offers 33% more memory bandwidth than today’s industry standard HPC servers.

The architecture of the Apollo 70 is unique in that it offers a dual socket design, paired with a single PCIe x16 and a x8 slot, each connecting to a different socket. The Mellanox ConnectX-5 100Gb Socket Direct™ OCP adapter is fully optimized to maximize this design, unleashing unmatched performance capabilities of standard server interconnects offering dual x8 connectivity. Using one Mellanox Socket Direct capable network adapter improves latency of the CPU by removing the need to pass all data through CPU 1 to reach the network. In addition to boosting CPU 2 performance, this design maximizes PCIe lanes available to memory and GPUs. Another added benefit is the Mellanox Socket Direct™ enabled adapter presents itself to the server management interface as a single interconnect which simplifies network management.

Socket Direct™ at a glance

Mellanox ConnectX-5 with Socket Direct™ provides 100Gb port speed, even to servers without x16 PCIe slots, by splitting the 16-lane PCIe bus into two x8 connectors. For the Apollo 70, Mellanox and HPE worked together to develop a single OCP card that bridges the two sockets together at the PCIe bus. In other cases, Socket Direct™ can be achieved with a PCIe x8 edge connector and a parallel x8 auxiliary PCIe connection card. Mellanox uses Multi-Host™ technology to allow multiple hosts to be connected into a single adapter by separating the PCIe interface into multiple and independent interfaces. Socket Direct also enables GPUDirect® RDMA for all CPU/GPU pairs by ensuring that all GPUs are linked to CPUs closest to the adapter card and boosts performance on both sockets by creating a direct connection between the sockets and the adapter card.

Real Life Scenario Lab Testing

In a real-life scenario, applications that run on a dual-socket server generate data traversing the CPUs (over the inter-processor communication bus). To receive a more realistic measurement of the network performance, we applied a test load on the inter-processor bus, and then measured the effect of this load on the external data traffic of the server. We took these measurements, while comparing the two types of adapters (standard adapter and Socket Direct™). Figure 4 compares the average latency of the two different adapters. The graph shows that when using the Socket Direct™ adapter, latency is reduced by 80% compared to the standard adapter. This latency improvement is a result of the direct path both CPU sockets take to reach the network and the even distribution of TCP streams between the CPUs.

Figure 4

Figure 5 shows CPU utilization. It is evident that direct access to the network using Socket Direct™ also provides a 50% improvement in CPU utilization. Moreover, the even distribution of TCP streams reduces the average cache miss count on both CPUs versus a standard configuration server, which further improves CPU utilization.

Figure 5

When comparing the servers’ external throughput while applying the inter-processor load (Figure 6), it is evident that by implementing Socket Direct™, the throughput is improved by 16%-28% compared to the standard adapter connection.

 

Figure 6


Why HPE chose Mellanox

Mellanox ConnectX-5 delivers high bandwidth, low latency, and high computation efficiency for high performance, data intensive and scalable compute and storage platforms. As a leader in the HPC interconnect market, Mellanox offers several advantages to accelerate the Apollo 70 System, including Socket Direct™ and Multi-Host™, as well as the multiple offloads provided by ConnectX-5. Our strong partnership with HPE and open standards like OCP made developing a platform and adapter from the ground up to meet specific HPC workloads demands possible.

Fully Virtualized for Enterprise Clouds

Whether using the ConnectX-5 for HPC or Ethernet network connectivity, almost all aspects of data center connections are virtualized today, including networks, network devices, and host interfaces. All network connections are defined by software, enabling any server to readily connect to any network, storage, or service. To accomplish this, Connect-X adapters offer a comprehensive set of network and I/O virtualization features:

  • Overlay networks: The adapter incorporates overlay capabilities enabling isolated Ethernet networks to reside within the fabric. These networks operate at the full fabric bandwidth of the adapter (up to 100 Gb/s). Furthermore, the adapter supports InfiniBand, Ethernet, and IP isolation mechanisms such as partitions, VLANs, and subnets.
  • Virtualized server I/O: Virtual machines are presented with virtual InfiniBand, or Ethernet adapters. These virtual network adapters join the fabric through a virtual switch, one virtual switch is supported per physical port.
  • Port virtualization: Both support for SR-IOV and paravirtualization help to provide a seamless hypervisor with Open vSwitch support.
  • RDMA enabled: InfiniBand and Ethernet support RDMA, improving server and storage efficiency and enabling applications to run faster and to be deployed with fewer systems.

 

Adapter Key Features

InfiniBand

  • EDR/FDR/QDR/DDR/SDR
  • Offloads
    • Tag matching and Rendezvous
    • NVMe over Fabric (NVMe-oF)
    • Burst buffer offloads for background checkpointing
    • Adaptive routing on reliable transport
    • vSwitch/vRouter offloads/Open vSwitch (OVS)
    • Erasure Coding
    • T10 DIF
  • PXE boot over Ethernet or Infiniband
  • Virtual Protocol Interconnect (VPI)
  • Mellanox PeerDirectTM RDMA
  • Dynamically Connected Transport (DCT)
  • On demand paging (ODP)
  • Extended Reliable Connected transport (XRC)
  • End-to-end QoS and congestion control

 

Ethernet

  • Virtual Protocol Interconnect (VPI)
  • RoCEv1
  • RoCEv2
  • Mellanox PeerDirectTM RDMA
  • On demand paging (ODP)
  • NVMe over Fabric (NVMf) Target Offloads
  • Enhanced vSwitch / vRouter Offloads
  • Hardware offloads for NVGRE and VXLAN encapsulated traffic
  • End-to-end QoS and congestion control

 

Conclusion

Mellanox Socket Direct adapters provide the highest performance and most flexible solution for the most demanding applications and markets. Socket Direct extends server performance and utilization with maximum throughput connectivity. The Socket Direct adapter within a dual-socket server, enables both CPUs to connect directly to the network, delivering lower latency, lower CPU utilization and higher network throughput. With added virtualization features, support for multiple protocols and form factors, the Mellanox ConnectX family provides high-performance and the most efficient network infrastructure. With Mellanox, there is no need to compromiser performance, security, or usability in high-performance virtualized environments.

For more information please see the HPE Quickspecs:

For more information on Mellanox ConnectX-5 Socket Direct please see the Product Brief.

For more information on the HPE Apollo 70, please Click Here.

About Matthew Hitt

Matt Hitt is a Senior OEM Marketing Manager at Mellanox Technologies. His responsibilities include working with HPE and other OEMs to bring joint Mellanox and OEM solutions to market. Matt Joined Mellanox in 2016, after spending more than seven years at Hewlett Packard Enterprise, in various sales, marketing, and channel roles. His experience provides a unique perspective on what drives the IT industry and customers’ needs in the market, enabling him to better position key technology that solves real world problems.

Comments are closed.