NVMe Over Fabrics on InfiniBand, the World’s Fastest Storage Network

NVMe Over Fabrics

More Vendors support NVMe over Fabrics using Mellanox

G2M Research, an analyst that specializes in solid state storage, just held a webinar on October 24th 2017, about NVMe and NVMe over Fabrics (NVMe-oF). In it, they predicted rapid growth in the NVMe market, including rising demand for specialized network adapters, and they named Mellanox as the “Highest Footprint” vendor with the largest share of these adapters. Back in August 2017 at Flash Memory Summit, IT Brand Pulse readers voted Mellanox as the leading provider of NVMe-oF network adapters. Neither of these is any surprise since Mellanox was first to market with 25, 40, 50, and 100GbE adapters and has been a longtime leader in the Remote Direct Memory Access (RDMA) technology that is currently required for NVMe-oF.

However, while most of the news about NVMe-oF have focused on Ethernet (using RoCE), some well-known storage vendors are supporting NVMe-oF over InfiniBand.

NetApp E-Series Supports NVMe-oF on InfiniBand

In September 2017, NetApp announced their new E-5700 hybrid storage array and EF-570 all-flash arrays, which both support NVMe over Fabrics (NVMe-oF) connections to hosts (servers) using EDR 100Gb/s InfiniBand. This made NetApp the first large enterprise vendor to support NVMe over Fabrics to the host, and first enterprise storage vendor to support EDR 100Gb/s InfiniBand. They are also — as far as I know — the first all-flash and hybrid arrays to support three block storage protocols with RDMA: NVMe-oF, iSER, and SRP. Rob Davis wrote about the NetApp announcement in his recent blog.

Figure 1: NetApp EF-570 supports NVMe-oF, iSER and SRP on EDR 100Gb InfiniBand.

Excelero demonstrates NVMe-oF on InfiniBand at SC17

Excelero has been a fast software-defined storage (SDS) innovator in supporting NVMe-oF as both a disaggregated flash arrays or in a hyperconverged configuration. They support both 100Gb Ethernet and EDR 100Gb InfiniBand, and they are demonstrating their solution using InfiniBand in the Mellanox booth #653 at Supercomputing 2017.

Other Systems Likely to Add NVMe-oF on InfiniBand Support

In addition to the publicly declared demonstrations, there are other vendors who already support InfiniBand front-end (host) connections are could add NVMe-oF support fairly easily. For example, the IBM FlashSystem 900 has a long history of supporting InfiniBand host connections and is known for high performance and low latency, even amongst other all-flash arrays. IBM also has a strong history of delivering HPC and technical computing solutions including storage. So it wouldn’t be much of a surprise if IBM decided to add NVMe-oF support over IB to the FlashSystem 900 in the future.

InfiniBand is the World’s Fastest Storage Network

NVMe-oF allows networked access to NVMe flash devices, which themselves are faster and more efficient than SAS- or SATA- connected SSDs. Because it eliminates the SCSI layer, NVMe-oF is more efficient than iSCSI, Fibre Channel Protocol (FCP), or Fibre Channel over Ethernet (FCoE). But with more efficient storage devices and protocols, the underlying network latency becomes more important.

InfiniBand is appealing because it is the world’s fastest storage networking technology. It supports the highest bandwidth (EDR 100Gb/s shipping since 2014) and the lowest latency of any major fabric technology, <90ns port-to-port, which is far lower than 32Gb Fibre Channel and slightly lower than 100Gb Ethernet.  InfiniBand is a lossless network with credit-based flow control and built-in congestion control and QoS mechanisms.

Since the NetApp E-series arrays are very fast — positioned for “Extreme Performance”— and NetApp is targeting high-performance workloads such as analytics, video processing, high performance computing (HPC), and machine learning, it’s no surprise that the product family has long supported InfiniBand and the newest models support EDR InfiniBand.

Likewise, Excelero positions their NVMesh® to meet the demand of the most demanding enterprise and cloud-scale applications, while the IBM FlashSystem 900 is designed to accelerate demanding applications such as online transaction processing (OLTP), analytics database, virtual desktop infrastructure (VDI), technical computing applications, and cloud environments. With their focus on these applications, it makes sense that they support InfiniBand as a host connection option.

Figure 2: The IBM FlashSystem 900 already supports InfiniBand host connections and IBM has promised to add support soon for NVMe technology.

InfiniBand Supports Multi-Lingual Storage Networking

InfiniBand is a versatile transport for storage. Besides supporting NVMe-oF, it supports iSCSI Extensions for RDMA (iSER) and the SCSI RDMA Protocol (SRP). It can also be used for SMB Direct, NFS over RDMA, Ceph, and most non-RDMA storage protocols that run over TCP/IP (using IP-over-IB). One of the innovative aspects of the new NetApp E-5700 and EF-570 is that they are “trilingual” and support any of the three block storage protocols over EDR (or FDR) InfiniBand. The IBM FlashSystem 900 also supports SRP and will presumably become “bilingual” on InfiniBand storage protocols after adding NVMe-oF.

So, whether you are already using SRP for HPC or want to adopt NVMe-oF as the newest and most efficient block storage protocol (or use iSER with NetApp), Mellanox InfiniBand has you covered.



Figure 3: The Mellanox Switch-IB 2 family supports up to 36 ports at EDR 100Gb/s in a 1U switch or up to 648 ports in a chassis switch, with port-to-port latency below 90 nanoseconds.


Cloud, HPC, Media, and Database Customers Drive Demand for EDR InfiniBand

Now, who exactly needs InfiniBand connections, or any type of 100Gb connection to the storage? If most storage customers have been running on 10Gb Ethernet and 8/16Gb Fibre Channel, what would drive someone to jump to 100Gb networking? It turns out many high performance computing (HPC), cloud, media, and database customers need this high level of storage networking performance to connect to flash arrays.

HPC customers are on the cutting edge of pure performance, wanting either the most bandwidth or the lowest latency, or both. Bandwidth allows them to move more data to where it can be analyzed or used. Low latency lets them compute and share results faster. EDR InfiniBand is the clear winner either way, with the highest bandwidth and lowest latency of any storage networking fabric. Several machine learning (ML) and artificial intelligence (AI) applications also support RDMA and perform better using the super low latency of InfiniBand. And the latest servers from vendors such as Dell EMC, HPE, Lenovo, and Supermicro can all be ordered with FDR 56Gb or EDR 100Gb InfiniBand adapters (based on Mellanox technology).

Cloud customers are on the cutting edge of scale, efficiency, and disaggregation, and whatever lets them support more users, more applications, and more VMs or containers in the most efficient way. As they use virtualization to pack more applications onto each physical host, they need faster and faster networking. And as they disaggregate flash storage (by moving it from individual servers to centralized flash pools) to improve efficiency, they need NVMe-oF to access that flash efficiently. While most cloud customers are running Ethernet, there are some who have built their networks on InfiniBand so want InfiniBand-connected storage arrays to power their cloud operations.

Media and Entertainment customers are scrambling to deal with ultra-high definition (UHD) video at 4K and 8K resolutions. 4K cinema video has almost 4.3 more pixels than standard HD TV (4K TV video is slightly lower, having only 4x more pixels than HD TV). While the initial capture and final broadcast use compression, many of the editing, rendering, and special effects steps used to create your favorite TV shows and movies require dealing with uncompressed video streams, often in real-time. These uncompressed 4K streams cannot fit in an 8Gb or 10Gb pipe, and sometimes even exceed what 16Gb FC can do. This has pushed many media production customers to use FDR (56Gb) or EDR (100Gb) InfiniBand, and they need fast storage to match that.

Figure 4: Adoption of 4K and 8K video is driving media and entertainment companies to adopt high-speed networking for storage, including EDR InfiniBand.

Database over InfiniBand may surprise some of you, but it makes perfect sense because database servers need low latency, both between each other and to the storage. The Oracle Engineered Systems (Exadata, Exalogic, Exalytics) are designed around an InfiniBand fabric and Oracle RAC server clustering software supports InfiniBand. Even in the cloud, a majority of financial or e-commerce transactions end up going through a SQL database in the end, and low latency for the database is critical to ensure a smooth online and e-commerce experience.

Leveraging The World’s Fastest Fabric for Storage

InfiniBand is the world’s fastest fabric with 100Gb/s today and 200Gb/s products announced and coming soon. While most of the world’s networked storage deployments are moving to Ethernet, it’s clear that when the fastest possible storage connections with the lowest latency are needed, InfiniBand is often the best choice. With new flash array support for NVMe over Fabrics, IBM and NetApp are supporting the world’s most efficient block storage protocol on top of the world’s fastest storage networking technology, and I expect the result will be many happy customers enjoying superfast storage performance.

Supporting Resources:



About John F. Kim

John Kim is Director of Storage Marketing at Mellanox Technologies, where he helps storage customers and vendors benefit from high performance interconnects and RDMA (Remote Direct Memory Access). After starting his high tech career in an IT helpdesk, John worked in enterprise software and networked storage, with many years of solution marketing, product management, and alliances at enterprise software companies, followed by 12 years working at NetApp and EMC. Follow him on Twitter: @Tier1Storage

Comments are closed.