Double Your Storage System Efficiency

Enterprise, Storage, , ,

Enable Higher IOPS while Maximizing CPU Utilization

As virtualization is now a standard technology in the modern data center, IT managers are  now seeking ways to increase efficiency by adopting new architectures and technologies that enable faster data processing and execute more jobs over the same infrastructure, thereby lowering the cost per job. Since CPUs and storage systems are the two main contributors to infrastructure cost, using fewer CPU cycles and accelerating access to storage are keys toward achieving higher efficiency.


The ongoing demand to support mobility and real-time analytics of constantly increasing amounts of data demands that new architectures and technologies be used, specifically those with smarter usage of expensive CPU cycles and as a replacement of old storage systems that were very efficient in the past, but that have become hard to manage and extremely expensive to scale in modern virtualized environments.


With an average cost of $2,500 per CPU, about 50% of compute server cost is due to the CPUs.  On the other hand, the I/O controllers cost less than $100. Thus, offloading tasks from the CPU to the I/O controller frees expensive CPU cycles, increasing the overall server efficiency. Other expensive components, such as SSD, will therefore not need to wait the extra cycles for the CPU. This means that using advanced I/O controllers with offload engines results in a much more balanced system that increases the overall infrastructure efficiency.


RDMA (Remote Direct Memory Access) is one of the most effective offload engines to be integrated into Mellanox’s ConnectX® I/O controller family. This engine enables significant I/O acceleration by allowing application software to bypass most layers of software and communicate directly with the hardware. Plus, it enables servers to directly place information into the memory of another computer, thereby reducing the application latency and minimizing the CPU overhead.


As for the storage system, the exponential growth of data has forced the industry to replace traditional SAN with higher capacity, easier-to-scale, and lower cost scale-out systems. This new architecture requires lower latency, which can easily be achieved when RDMA is used to access a distributed cache storage (mostly SSD-based). These new scale-out systems, such as Storage Spaces over SMB Direct, provide the same RAID capabilities as traditional SAN. Moreover, by using I/O adapters with offload engines such as erasure coding, which is often used instead of traditional RAID because of its ability to reduce the time and overhead required to reconstruct data, these scale-out systems can further increase the performance and lower the cost.

In order to meet customer needs, Dell and Mellanox worked together on a joint solution that increases CPU utilization and accelerates access to storage. The solution includes standard off-the-shelf components that reduce the Total Cost of Ownership.  The solution components Included:


  • Client Servers: 2 x Dell PowerEdge R630
  • Switch: Dell S6000 Networking 10GbE Switch
  • File Servers: 2 x Dell PowerEdge R730
  • NIC: Mellanox ConnectX-3 10GbE with RoCE
  • Cables: Mellanox LinkX™ SFP+ Copper Cables


Motti Beck 051115 solution_topology


To compare the efficiency boost that RoCE enables, we compared the IOPS when running different workloads over 4KB and 8KB block sizes. Analysis of the results shows that in all workloads, the number of IOPS when running Storage Spaces over SMB Direct was higher than when running over TCP/IP.


Motti Beck 051115 storage_spaces_IOPs_diagram


To compare the efficiency in CPU utilization, we ran different workloads using a 512KB block size. The results show that when running over SMB Direct, Storage Spaces uses less than 50% of the CPU it uses when running over TCP/IP.

Motti Beck 051115 CPU_utalization_diagram


Faster access to storage with lower CPU usage enables more jobs per second to run over the same infrastructure, thereby maximizing the ROI.


Running Microsoft’s Storage Spaces over Dell PowerEdge servers and Mellanox 10GbE RoCE interconnect provides a much better alternative to TCP/IP-based networking by:


  • Delivering 15% more IOPS
  • Increasing CPU utilization by over 50%
  • Reducing power consumption by achieving more jobs per second over the same infrastructure



About Motti Beck

Motti Beck is Sr. Director Enterprise Market Development at Mellanox Technologies Inc. Before joining Mellanox Motti was a founder of BindKey Technologies an EDC startup that provided deep submicron semiconductors verification solutions and was acquired by DuPont Photomask and Butterfly Communications a pioneering startup provider of Bluetooth solutions that was acquired by Texas Instrument. Prior to that, he was a Business Unit Director at National Semiconductors. Motti hold B.Sc in computer engineering from the Technion – Israel Institute of Technology. Follow Motti on Twitter: @MottiBeck

Comments are closed.