All posts by Guest Blogger

Boosting Performance With RDMA – A Case Study

The following is a guest blog by E8 Storage.

E8 Storage recently published the Deploying E8 Storage with IBM Spectrum Scale white paper to demonstrate the simplest and most straightforward way to deploy E8 Storage as the underlying storage for an IBM Spectrum Scale (formerly known as GPFS) cluster.  In that paper, we covered how to enable RDMA between NSD servers and clients to improve the performance of the GPFS cluster with E8 Storage.  Now, we would like to share with our customers what a tremendous improvement a few simple commands can deliver to the performance of your Spectrum Scale cluster.

E8 Storage leverages and greatly benefits from Remote Direct Memory Access (RDMA)– running on InfiniBand or RDMA over Converged Ethernet, aka RoCE–a key feature of our Mellanox ConnectX® adaptors, and the Mellanox switches that we use in our test lab environment. RoCE is supported by all data-center grade Ethernet switches, and is also supported by a wide variety of NICs, primarily from Mellanox. Having a high-speed Ethernet infrastructure already in place enables customers to extract additional value from their hardware and software investments by moving the NSD Server-Client block communication away from the traditional 1GbE networks and on to the fast, reliable and, most importantly, already paid-for RDMA infrastructure. That alone provides a significant performance boost in the form of reduced latency and increased throughput.

It doesn’t stop there. By turning on the support for RDMA using the VERBS API for data transfer between an NSD server and client, customers can further drive the latency down and the throughput up.

The steps to enable the VERBS API can be found in the Deploying E8 Storage with IBM Spectrum Scale whitepaper.  To measure performance, we used a small 3 node cluster consisting of 2 NSD servers and 1 client, each connected at 50GbE to the Mellanox SN2700 network switch via Mellanox ConnectX-4 adaptors. The client node had no local access to E8 Storage volumes and so all I/O had to go through one of the NSD servers. The only I/O load on the cluster came from the I/O generator and performance measuring tool, FIO v3.5.

The testing methodology was simple: run random read jobs against the mounted GPFS file system /e8fs1. The workloads we used for the performance comparison were:

  • 4K, 100% random read, 8 FIO threads, queue depth/thread of 14 for a total queue depth of 112.
  • 128K, 100% random read, 8 FIO threads, queue depth/thread of 7 for a total queue depth of 56.


The results below speak for themselves.  For the client performance we’re talking about

  • Over 5x improvement in small block latency and throughput
  • Over 2x improvement in large block latency and throughput



Note how the large block IO is now able to nearly max out all available bandwidth of the 50GbE connection, which has a potential max throughput of about 5.5GB/s.


As you can see from the results, RDMA delivers a significant performance boost, not only for storage built for NVMe, but for general performance between hosts and servers.  The foundation of this performance is Mellanox’s Ethernet Storage Fabric which dramatically increased the performance of GPFS using RDMA.

Mellanox’s ConnectX-4 NICs delivered high bandwidth and provided a robust implementation of RoCE (RDMA over Converged Ethernet) that is a key component of this NVMe-oF-based system.Mellanox’s SN2700 Ethernet Switch delivered the non-blocking high performance and consistently low latency across all of its ports that was necessary to enable the new benchmark.

E8 Storage customers are uniquely positioned to extract additional value from the fast, reliable and cost-effective solution they already have at their disposal by taking advantage of a few simple steps to enable RDMA within their existing Ethernet or IB networks.

If you want to learn more, contact us and join us for a webinar next week:

E8 Storage uses Mellanox’s Ethernet Storage Fabric in Record Breaking Benchmark!

The following is a guest blog by E8 Storage.

There’s nothing like an established, peer reviewed benchmark such as SPEC SFS®2014 to prove what you’ve been saying all along: E8 Storage delivers high performance and low latency at a fraction of the cost of competing solutions. This means that data-intensive applications can rely on E8 Storage to deliver real results and not just hype.

What is SPEC SFS2014?

SPEC SFS has long been the benchmark of choice for measuring file system storage performance and a strong resource for comparing competitive file storage offerings.  SPEC SFS2014 is the 3rd generation of the benchmark and was a major rewrite to move the focus away from measuring the storage component itself, and instead focuses on the end-to-end storage solution for specific workloads. requires all benchmark results to disclose the hardware, software, and network configuration used, and mandates that only generally-available solutions be used to generate the published results. Results are reviewed by a SPEC committee before publication on the web site.

The benchmark can be run against any of the 5 available workloads which are meant to simulate a wide range of applications needing storage performance. E8 Storage chose to focus on the Software Build (SWBUILD) workload primarily because the results could be applied to various High Performance Computing (HPC) workloads.


Mellanox’s Ethernet Storage Fabric – the foundation for the record-breaking performance

A high-performance storage solution requires a network fabric that is designed to deliver consistently low latency and high bandwidth, with available offloads that remove CPU bottlenecks.  Not every datacenter switch can be used to create an Ethernet Storage Fabric (ESF). An ESF is run on purpose-built switches which are optimized to deliver the highest levels of performance, lowest latencies and zero packet loss, with unique form factors and storage-aware features that provide access to remote data with the same performance as access to local data, from the application’s perspective. This translates into close-to-local predictable latency, line-rate throughput with QoS, and linear scalability to accommodate dynamic, agile data movement between nodes – Mellanox’s Spectrum switches provided the end-to-end network foundation for an ESF and contribute tothe record-breaking performance achieved by E8 Storage.

Mellanox’s ConnectX®-5 NICs were used in each of the 16 hosts to generate storage traffic, and a Mellanox SN2700 100G Ethernet switch was used to connect these hosts to the single E8-D24 Storage Appliance.  The E8-D24 has 8 *100G Ethernet ports (4 ports to each of its 2 controllers) which are powered by ConnectX-5 NICs within. All of the equipment was connected using Mellanox LinkX® DAC cable products. E8 Storage’s multi-patented scale-out software architecture and technologies enabled the NVMe-oF-based storage solution to use Mellanox end-to-end RoCE (RDMA over Converged Ethernet) technology, as shown below, to deliver the record-breaking storage system performance.

The Results

E8 Storage smashed the existing benchmark!   Not only did they complete more builds than any previous results, the solution delivered more throughput and IOPS at a fraction of the latency of previous submissions. E8 Storage achieved the lowest ever latency (overall response time) at the time, and still has one of the only sub-microsecond response times of all submissions to date.


E8 Storage achieved great performance on the SPEC SFS2014_swbuilds benchmark. This chart shows how the response time (latency) of the solution goes up as the number of IOPS increases, until latency becomes unacceptably high. Using the E8 and Mellanox solution, the benchmark achieved:

  • More builds – over 2.5x more than the previous IBM Spectrum Scale test
  • More performance – Almost 300,000 IOPS
  • Lowest latency – a record breaking 0.69ms!


Now, one of the criticisms of standardized benchmarks is that vendors can try to game the system with special performance tuning that isn’t available—or isn’t affordable–to customers.  Not so with our results.  The E8 Storage Controller had no performance tuning; that is no special tweaks and no impractical configurations: just great performance out of the box.

Here’s what we tested:

  • 1 x E8-D24 NVMe Appliance – Our flagship appliance with 24 HGST SN200 NVMe drives and dual controllers for high availability
  • E8 Storage Software version 2.1.1 – Our latest GA software release
  • IBM Spectrum Scale 5.0 – The most up to date software for IBM’s high performance distributed file system
  • 16 x Spectrum Scale Nodes – The Spectrum Scale cluster nodes all connected directly to the shared read / write volume on the E8-D24
  • 16 x Mellanox ConnectX5 NICs – one per Spectrum Scale cluster node
  • 1 x Mellanox SN2700 100GbE Switch – high-performance interconnect
  • 8 x Mellanox LinkX 100GE AOC cables
  • 16 x Mellanox LinkX 100GE to 2*50GE DAC splitter cables.



Mellanox’s Ethernet Storage Fabric and E8’s D24 NVMe Appliance were the foundation for this small, energy-efficient and relatively inexpensive system that smashed the existing SPECsfs benchmark.

Mellanox’s ConnectX-5 NICs delivered high bandwidth and provided a robust implementation of RoCE that is a key component of this NVMe-oF-based system.

Mellanox’s SN2700 Ethernet Switch delivered the non-blocking high performance and consistently low latency across all of its ports that was also necessary to enable the new benchmark.

Finally, here’s what some of the press have been saying about this performance:


Why iSER is the right high speed Ethernet all-flash interconnect today

The following is a guest blog post from Subhojit Roy, a Senior Technical Staff Member working out of IBM India Labs. 

All-flash storage is bringing change throughout the data center to meet the demands of modern workloads. Fiber Channel has traditionally been the preferred interconnect for all-flash storage. However, 21st century data center paradigms like cloud, analytics, software defined storage, etc. are driving a definitive shift towards Ethernet infrastructure that includes Ethernet connectivity for both server and storage. As Ethernet speeds rapidly increase to 25/40/50/100Gb, it becomes more and more lucrative as an interconnect to all-flash storage. While traditional iSCSI has gained significant ground as Ethernet interconnect to storage, inefficiencies in the TCP/IP stack don’t allow it to be the preferred interconnect to all flash storage.

In comes iSER (iSCSI Extensions over RDMA) that maps the iSCSI protocol to RDMA (Remote Direct Memory Access). iSER provides an interconnect that is very capable of rivaling Fiber Channel as the all-flash interconnect of choice. It leaves the administrative framework of iSCSI untouched while mapping the data path over RDMA. As a result, management applications like VMWare vCenter, OpenStack, etc. continue to work as is, while the iSCSI data path gets a speed boost from Remote Direct Memory Access. A move from traditional iSCSI to iSER would thus be a painless affair that doesn’t require any new administrative skills.

iSER retains all the enterprise class capabilities that are expected off Tier 1 shared storage. It also matches or beats Fiber Channel in terms of access latency, bandwidth and IOPS. Capabilities like multipath IO, SCSI Reservations, Compare and Write, vVols support, and offloaded data copy operations like XCOPY/ODX will work from day one on iSER. In addition, iSER benefits from all the SCSI error recovery techniques that have evolved over the years – things like LUN Reset, Target Reset, Abort Task, etc. In essence, all enterprise class applications will continue to work as reliably and seamlessly over iSER as they used to work over iSCSI.

The diagram below shows how iSCSI is involved in the iSER IO path only for the Command and Status phases while the Data Transfer phase is totally taken care of by RDMA transfers directly into application buffers without involving a copy operation. This compares well with NVMeF in terms of latency reduction.

NVMe over Fabrics or NVMeF is a new protocol that promises to take all-flash interconnect technology to the promised land of extreme performance and parallelism and there are a lot of expectations from it. It is a protocol that is still evolving, and therefore not mature enough to meet the requirements of clustered applications running over shared Tier 1 all-flash storage. And it is a quantum jump that not only expects the user to move to high speed Ethernet technology from Fiber Channel but a totally new protocol with a new, unfamiliar administrative model. It is likely that NVMeF will take some time to mature as a protocol before it can be accepted in data centers requiring Tier 1 shared all-flash storage. In addition to that applications must adapt to a new queuing model to exploit the parallelism offered by flash storage.

That leaves iSER as the right technology to bridge the gap and step in as the preferred interconnect for shared all-flash storage today. iSER is ready from day one for latency, IOPS and bandwidth hungry applications that want to exploit high speed Ethernet technology, both as a north-south and east-west interconnect. IO parallelism may not be as high as promised by NVMeF, but it’s sufficient for all practical purposes without requiring applications to be rewritten to fit into a new paradigm.

By implementing iSER today, the move from Fiber Channel to high speed Ethernet can be tried out without ripping out the entire administrative framework or the need to rewrite applications. A gradual move from Fiber Channel to RDMA over Ethernet replaces the layer 2 transport protocol and helps assess the newer protocol in terms of its stability, resiliency and error recovery capabilities that are essential for a SAN storage interconnect. Once proven, the same RDMA technology can then be leveraged to bring in NVMeF which promises more in the future. Since iSER and NVMeF will work equally well on the same hardware, the infrastructure investment made in iSER is protected for the long term.

At IBM we are working toward enabling our customers to move to data center infrastructure that consists purely of Ethernet interconnects with speeds scaling rapidly from 10 – 100Gbps. Built over iSER, this capability is all-flash storage ready from day one. Agnostic of the underlying RDMA capable networking, it is likely to be very attractive to software defined storage infrastructure that is expected to be built from commodity hardware. It enables IBM Spectrum Virtualize products (IBM Storwize and IBM SVC) to be deployed on cloud infrastructure where Ethernet is the only available infrastructure. And in order to get there, we have partnered with multiple hardware and software vendors that are at the forefront of the high speed Ethernet revolution.

So get ready to experience all-flash storage connected over high speed Ethernet from IBM sometime in the near future!

Subhojit is Senior Technical Staff Member working out of IBM India Labs, Pune. He works as development architect for the IBM Spectrum Virtualize product. He has worked for 23 years in Data Storage, Storage Virtualization, Storage Networking etc. across organizations like IBM, Veritas, Brocade & Symantec etc. At IBM he has been driving Ethernet & IP Storage architecture and roadmap for the IBM Spectrum Virtualize products. Currently he is working on high speed Ethernet interconnect for all flash storage including iSER and NVMeF. Prior to IBM he has been responsible for key features for Enterprise Storage products in his earlier organizations. He is Master Inventor and Member Academy of Technology at IBM. He owns significant Intellectual Property in the form of patents and has more than 50 granted and filed patent applications. He can be found on Twitter @sroy_sroy and on LinkedIn at


RDMA enhancements for ISCSI

Guest post by Noam Shendar, VP Business Development, Zadara Storage

Just Punch It:  Accelerating Storage the Easy Way

Architecting storage solutions means being on the hunt for bottleneck after bottleneck and removing them to eliminate latency wherever possible. With the availability of storage media such as Flash and high-performance disk drives, the bottleneck has often moved away from the storage medium itself and onto the interconnect.


When building our enterprise storage as a service offerings, we’ve had to overcome several bottlenecks – from reinventing the proprietary, purpose-built controllers of traditional arrays that caps the speed and expandability of an array, through the performance of the controllers, to using flash cache acceleration, it was clear that to deliver applications with even better efficiency our award-winning Virtual Private Storage Arrays (VPSA) need to take advantage of new technologies for the datapath.


What is iSER technology?

iSER is an interface that uses Ethernet to carry data directly between server memory and storage devices. The protocol eliminates memory copies (a.k.a. memcpy) and the operating system TCP/IP stack, and bypasses the CPU of the target system entirely.  As such, it is lossless, deterministic, with low overhead, and delivers far better CPU efficiency by freeing up resources.

Continue reading

End of an Era: OEM Dominance – Could it Be a Thing of the Past?

Guest Blog post by Giacomo Losio, Head of Technology – ProLabs

Original equipment manufacturers (OEM’s) have long dominated the optical components market but a new study now suggests that, as a result of tighter margins and greater competition, customers are putting quality and price before brand.  Is the era of the big OEM at an end?

When asked their views of the optical transceiver market at the European Conference on Optical Communications (ECOC) in Cannes, over 120 attendees revealed a trend which indicates a paradigm shift in attitudes.

Why do they buy? What they buy? What keeps them up at night? The answers may surprise you:

  • 98% of respondents ranked quality as one of their top three priorities when purchasing fibre optics
  • 89% of respondents placed price in the top three list of priorities
  • Yet only 14% of respondents even considered brand names to be a top three priority – or even a concern

Continue reading

The Benefits of Leaning Into the Big Data

Guest post by Alex Henthorn-Iwane, QualiSystems

Big data is for real, but its places heavy demands on IT teams, who have to pull together and provision cloud infrastructure, then offer big data application deployments with validated performance to meet pressing business decision timelines.  QualiSystems is partnering with Mellanox to simplify big data deployments over any cloud infrastructure, enabling IT teams to meet line of business needs while reducing operational costs.

Quali Systems cutcaster-903282828-Big-data-small

Continue reading

The Storage Fabric of the Future Virtualized Data Center

Guest post by Nelson Nahum, Zadara Storage

It is evident that the future data center will be based on cutting-edge software and virtualization technologies to make the most effective use of hardware, compute power, and storage needs to perform essential analytics and to increase the performance of media-related and advanced web applications. And it turns out that the wires that will connect all this technology together are no less crucial to next-generation data centers and clouds than the software and virtualization layers that run within them.


There are multiple storage fabrics and interconnects available today, including Fibre Channel, Ethernet and SAS. Each has various pros and cons, and fabrics were chosen according to need of performance, compatibility and cost efficiencies.


As an enterprise storage as-a-service provider, delivering a software-based cloud storage solution for public, private and hybrid cloud models based on commodity hardware, Zadara Storage provides storage as-a-service in multiple public cloud and colocation facilities around the globe. Consistency, high availability and predictability are key in supplying the scalable, elastic service our customers expect, regardless of their location, facility or the public cloud they employ. The hardware we use needs to be dependable, pervasive and cost-efficient in order to sustain the performance and cost-level of our service, anywhere and at any scale.


When choosing our fabric, Ethernet came as a clear choice. Ethernet is likely to become the new standard, and boasts several advantages vital to our product:

  • Ethernet’s speed roadmap is aggressive: from 10GbE to 40GbE, and upcoming 100GbE
  • Ethernet is ubiquitous: we can employ it with no complication at any data center or colocation facility around the globe
  • The latency we have found to be more than manageable, specifically as we use advanced techniques such as IO virtualization and data passthrough
  • Ethernet is the most cost effective: an as-a-service company needs to have a competitive pricing edge.

The future of enterprise storage
The future of Enterprise Storage lies in software and a choice of hardware (premium or commodity). Software-defined storage can scale performance more easily and cost effectively than monolithic hardware, and by combining the best of hardware of software, the customer wins. Ethernet is a critical element of our infrastructure, and Mellanox switches offer significant higher performance and consistent dependability that enables our storage fabric and meets our customer’s needs.


Zadara Storage at the Mellanox Booth at VM World 2013
Wednesday, August 28, at 2:15pm
At the Mellanox Booth at VM World 2013, Zadara Storage CEO, Nelson Nahum, will present the Zadara™ Storage Cloud, based on the patent-pending CloudFabric™ architecture, and providing a breakthrough cost structure for data centers. Zadara’s software-defined solution employs standard, off-the-shelf x86 servers, and utilizes Ethernet as its only interconnect to provide performant, reliable, SSD- and spindle-based SAN and NAS as a service.


About Zadara Storage
An Amazon Web Services and Dimension Data Technology Partner and winner of the VentureBeat, Tie50, Under the Radar, and Plug and Play cloud competitions, Zadara Storage offers enterprise-class storage for the cloud in the form of Storage as a Service (STaaS). With Zadara Storage, cloud storage leapfrogs ahead to provide cloud servers with high-performance, fully configurable, highly available, fully private, tiered SAN and NAS as a service. By combining the best of enterprise storage with the best of cloud and cloud block storage, Zadara Storage accelerates the cloud by enabling enterprises to migrate existing mission-critical applications to the Cloud.