All posts by Tim Lustig

About Tim Lustig

Tim Lustig is the Director of Corporate Ethernet Marketing at Mellanox Technologies, Inc. As a professional in the networking industry, Tim Lustig has been at the forefront of marketing new networking technologies for over two decades. From his start in network and database administration to product marketing and corporate technologist roles, Lustig’s experience includes outbound marketing activities, third party testing/validation, strategic product marketing, market research, and technical writing. Lustig has written many papers and articles on multiple networking technologies, and has been a featured speaker at many industry conferences around the world. Prior to Mellanox Tim held positions at Brocade Communication as Sr. Product Marketing Manager and QLogic Corporation as the Director of Corporate Marketing. Tim Lustig currently sits on the 25 Gigabit Ethernet Consortium marketing committee where he is an industry steward for the promotion of 25, 50 and 100Gb Ethernet. Follow Tim on Twitter: @tlustig

Mellanox Takes the Lead at NVMe Plugfest to Simplify Deployment While Ensuring Interoperability

For two weeks during the last half of June, the University of New Hampshire’s InterOperability Lab (UNH-IOL) hosted this year’s NVMe (Non-Volatile Memory express) and NVMe-oF (NVMe-over Fabrics) plugfests. Each were four day events comprised of approximately ten companies coming together to test over 20 devices including NVMe initiators, drives and storage controllers. Mellanox Spectrum switches (including HPE StoreFabric M-series Storage Switches which are powered by Mellanox Spectrum) connected everything and ensured high-efficiency data movement while supporting the NVMe protocol. The goal for Mellanox, as with others, was to successfully test with as many of the attending companies devices as possible to instill confidence that products will work well in a multi-vendor environment. Ultimately, every vendor seeks to leave with their company’s product being placed on the NVME Integrators List or NVMe-oF Integrators List, which provide a directory of devices that have successfully completed the rigors of the UNH-IOL compliance and interoperability testing.

What is NVMe and NVMe-oF?

NVMe, a storage interface specification, was designed specifically to keep pace with the increasing speeds of solid-state drives (SSD) by allowing faster access. NVMe-oF allows for remote fabric access to flash storage with latency similar to local access over high performance networks such as Ethernet and InfiniBand.   To achieve direct attached levels of performance NVMe-oF is dependent on RDMA technology to provide for a high-throughput, low-latency network. Offering an order of magnitude performance improvement for storage is proving to be a strong driver for the adoption of NVMe. The shift to NVMe is happening now in the enterprise space for applications where top-end performance is necessary, resulting in NVMe becoming the new industry standard for high-performance storage access. With NVMe gaining traction, plugfests like this one provide companies a chance to test hardware and software standards, conformance and interoperability and offer an opportunity to validate product capabilities in a multivendor environment. This provides essential feedback to the vendor community at the event and ensure that when products are released to the public that they are reliable, robust and interoperable with other NVMe products.

Mellanox Attendance at the NVMe Plugfests

This was the 10th annual NVMe and the 9th annual NVMe-oF plugfest hosted by UNH-IOL. Mellanox has been a regular participant and joined the latest events with our ConnectX-5 EN adapters, our newest Spectrum Ethernet switch, the SN2010, and its bigger brother the SN2410. And for the first time, we brought our BlueField SmartNIC a system-on-chip (SoC) network adapter configured to front-end an NVMe-over Fabrics development platform flash array. With the completion of the latest UNH-IOL Plugfests and the past events Mellanox has attended, the entire Mellanox Ethernet portfolio has now been approved for inclusion within the UNH-IOL Integrator’s Lists. Updates to the latest plugfest may not yet be reflected on the Integrator’s Lists at this time, but look for updates to come soon.

Mellanox was also able to test NVMe compliance with our own Mellanox Onyx Network Operating Systems (NOS) which is ideal for Ethernet Storage Fabrics and with Cumulus Networks NOS..

UNH-IOL Test Procedure

To ensure the testing is comprehensive and robust, UNH-IOL has developed their very own IOL INTERACT Test Software. The IOL INTERACT Test Software is built on top of the tNVMe test tool developed for compliance testing (available on tNVMe GitHub). IOL INTERACT Test Software was debuted at the first NVMe plugfest and allows for a wide range of test scenarios to uncover a variety of bugs. Generally speaking, the IOL INTERACT Test Software is a quick, efficient and easy way of testing and is extremely useful for numerous NVMe devices. The test software is incorporated into the Universities NVMe test plans along with the UNH-IOL test scripts which are combined with protocol analyzers to provide interpretation and debug of data traffic. The protocol analyzers allow for fine-grain examination and tracing of commands (i.e., READ, WRITE, etc.), issued from/to the NVMe devices so that the interaction between them can be verified. This allows identification of devices that are not in compliance with the NVMe protocol and for debugging when there is interoperability issues.

PCIe Analyzer with Mixed NVMe Commands


Vendors attending the Plugfest seek to provide confidence in their devices compliance with the NVMe specification and to help foster multi-vendor interoperability. Successful completion of this rigorous testing results in the vendors’ products being listed on the UNH-IOL Integrator’s Lists, and Mellanox is pleased that our latest switches and BlueField SOCs have achieved compliance and interoperability. The Integrator’s Lists are designed to help support data center managers, integrators, CIOs and other IT decision makers with their planned NVMe deployments. They can verify device compliance and interoperability from both documents which are refreshed after each plugfest, including those hosted at UNH-IOL and one hosted in Wuhan, China to increase international exposure. This year’s testing saw an increase in companies attendance as well as devices, a trend that continues and illustrates the continued industry demand for NVMe solution that are capable of higher bandwidth and faster performance.

Mellanox Mitigates Meltdown Mess, Stops Spectre Security Slowdown

Earlier this month, major chip manufacturers announced that vulnerabilities known as Spectre and Meltdown affect processors that are currently deployed in millions of devices. These new security flaws open the door for hackers to access sensitive user data. The flaws are not unique. Instead, they impact chips for everything from phones to servers and may go back more than two decades.  The flaw is the result of a common design practice employed by most modern processors, which has been publicly described as allowing an unprivileged attacker to bypass memory security restrictions and gain read access to privileged memory.  These vulnerabilities allow an unprivileged local attacker to read privileged memory belonging to other processes or memory allocated to the kernel.

The relevant Common Vulnerabilities and Exposure (CVE) items associated with Meltdown and Spectre are recorded under:  Spectre attack: CVE-2017-5753CVE-2017-5715 and Meltdown attack: CVE-2017-5754. Full concise details, including links to the patches issued by OS vendors, are located under or under the Vulnerability Note VU#584653.

Patching Causes Performance Degradation

The nature of the flaws requires fixes to guard against attacks, which have the effect of slowing down computers and can cause a significant decrease in server performance. According to Intel, the worst affected workloads were those “that incorporate a larger number of user/kernel privilege changes and spend a significant amount of time in privileged mode”. While patches are available from major chip and OS vendors, applying these patches can have a major performance impact on workload performance. Performance testing completed by Mellanox before and after the patches were applied, show significant performance degradations as a result of the patch.


Fig. 1 Performance degradation on patched system

Fig 2. Performance degradation on patched system


This can be avoided by deploying SmartNIC technologies that offload transport and memory protection processing from the CPU. Leveraging these offloads the SmartNIC shoulders the burden of processing the TCP/IP data processing and storage network stacks. Mellanox delivers offload technologies such as RDMA and DPDK to remove the burden of processing the TCP/IP and storage stacks from the main system processor, moving it to the network adapter. This provides performance advantages and alleviates performance losses typically seen when patching systems for Spectre and Meltdown.  Testing of data intensive transport processing benchmarks on InfiniBand and RoCE enabled NICs, showed zero performance impact due to the security patches.

Fig. 3 Performance is maintained on systems running Ethernet offload technologies

Fig 4. Performance is maintained on systems running InfiniBand offload technologies


Adapters that deploy network “OnLoad” technologies do exactly the opposite – burdening the CPU with networking functions that rob horsepower from business applications. As a result, many common workloads such as OLTP, database analytics and virtual machines are susceptible to performance impacts on servers that have been patched. Internal testing at Mellanox showed a performance impact of up to 47% on systems reliant on the host processor for TCP/IP and OmniPath data transfers. For further information, read: Mitigating the Performance Penalty of Spectre and Meltdown

Fig. 5 Performance is maintained on systems running offload technologies while a larger degradation of performance is seen in systems running Onload-based adapters.


Impact of the vulnerability on Mellanox Products

Mellanox diligently reviewed any potential security impact of Spectre and Meltdown on all of our relevant products. Below is an update on Mellanox’ action and status with regards to these security breaches. Mellanox testing is still ongoing, and timely updates will continue to be released.

Impact of the vulnerability on Mellanox Network Adapters

Mellanox network adapters are not exploitable by the Spectre and Meltdown CVEs.

Mellanox is currently reviewing all of the released patches issued by OS vendors, and testing them to evaluate potential impact.

  • No functional nor stability impact has been uncovered so far on any of the distributions tested.
  • Performance impact has been observed in some cases, especially in CPU intensive scenarios. However RDMA performance is not impacted, an additional proof that our core technology not only provides better performance, but is also more secure.

Impact of the vulnerability on Mellanox switches (InfiniBand and Ethernet)

Mellanox switch ICs are not affected by the Meltdown or Spectre vulnerabilities.

Mellanox switch systems impact:

  • Externally managed InfiniBand switches and PPC based switch systems are not susceptible to these attacks.
  • Ethernet x86 based switches use Intel CPUs, and as such are potentially susceptible to these attacks. However, exploiting this vulnerability requires running custom code on Mellanox switches.
    • Systems running MLNX_OS
      • When not running virtual machine (VM) / Dockers, the switches are used as closed systems, with no access to shell, and as such are not vulnerable.
      • When running virtual machine (VM) / Dockers, the switches are potentially vulnerable. Customers may avoid exposure by following Mellanox security guidelines, including using only trusted software distributions on VMs and not allowing containers privileged shell access.
      • Mellanox is carefully investigating the released patches, and will release software updates as soon as available
    • Systems running Cumulus
      • Please refer to updates issued by Cumulus Networks.
    • Systems running SONiC
      • Please refer to updates issued by Microsoft/SONiC

Impact of the vulnerability on Mellanox BlueField systems

BlueField includes subsystem multicore Arm A72-based subsystem. Arm has publicly identified Arm cores with Cortex-A72 as not susceptible to Meltdown CVE (CVE-2017-5754), but potentially affected by Spectre CVE variants (CVE-2017-5753 and CVE-2017-5715).

BlueField SmartNIC fully supports secure boot and thus is not susceptible to the Spectre vulnerabilities, as unauthorized software/Users don’t have access to the Arm cores inside BlueField; other BlueField based platforms are potentially susceptible to Spectre vulnerabilities if untrusted code is allowed to run on the device. Mellanox is carefully investigating the released OS patches, and will release software updates as soon as available.

Impact of the vulnerability on Mellanox Management Software

NEO and UFM software are not susceptible to Spectre and Meltdown vulnerabilities.

UFM appliance is a closed system and as such is not susceptible to these vulnerabilities.


Reports of serious workload performance degradation as a result of the Spectre and Meltdown security patches range from 5% – 30%. Internal testing by Mellanox found a similar range from 2% – 47%. This will have a huge impact on large data centers in both OpEx and CapEx as servers will need to be replaced, or additional servers added to compensate for the performance losses. Mellanox ConnectX adapters with offload technologies bypass the kernel and have a proven track record of accelerating performance. If you haven’t looked into Mellanox offload technologies before, this presents a compelling case to deploy Mellanox intelligent network adapters today. Contact your Mellanox representative to learn more about how we can help you alleviate the potential pocket draining performance impacts of Spectre and Meltdown.

Learn more:

Take a Front Row Seat at the 2018 Olympics – Ethernet Networking Provides an Immersive Viewing Experience

It will be the smallest city to host the Winter Olympics since 1994 in Lillehammer, Norway. However, South Korea has big hopes and even bigger technology to ensure the 23rd Winter Olympic Games will offer a viewing experience as never before. The 2018 Winter Olympics in PyeongChang will feature 102 events in 15 sports. From Thursday, February 8th to closing ceremonies on Sunday, February 25, the world will be watching athletes from some 90 countries as new Ethernet technology enables viewers to experience, the thrill of victory and the agony of defeat, as never before.

Technology is nothing new to the Olympics, which has provided a venue and opportunity for broadcast innovation every two years. This Winter Olympics will be the first immersive viewing experience where viewers will be able to experience first-person 360-degree views from athletes via virtual reality from high-definition videos.

Many are familiar with Mellanox for our high-performance connectivity use-cases for High Performance Computing (HPC), but few may know about our involvement with video transportation.

IP based- networks are being used in the 2018 Winter Olympics to help streamline content distribution and management as video will be captured by new methods.

The technological advancements of broadcasting the Games as they have changed throughout the years require considerable networking prowess. Mellanox Ethernet-based networks are proliferating across the entire broadcast and television ecosystems. Higher quality content such as that which will be broadcast at the Olympics (HD, UHD, 4K and beyond) requires a network with more horsepower and bandwidth. Companies such as BBC, FOX and NBC are actively converting to IP infrastructure. This includes live sports, production studios, data transport, content distribution, and even applications for storage. All with the proven and scalable solution comprised of Mellanox Spectrum switches, ConnectX adapters and LinkX cables, while delivering extremely reliable HDR video to viewers.

Mellanox’s leading IP-based technologies allow broadcasters the ability to alter the economics while enriching the video experience by bringing the audience and content closer together. In addition, with the SMTPE standard 2110 established which defines professional media over managed IP networks, broadcasters now have a technology path that accelerate the migration to IP-based networks, which will save time and money. The move from HD to 4K/UHD, 8K and HDR not only illustrates the necessity for absolute bandwidth, but also shows the inability of the current proprietary technology (SDI) & traditional Fibre Channel storage to deliver what customers crave, a more realistic entertainment experience. Driven by their need to have the underlying hardware innovate at their speed, the media & entertainment industry is now transitioning to a converged Ethernet fabric for both their applications and the storage, with an open standard that has been powering datacenters for decades while allowing innovation at a level that has never before been able to be realized.

Mellanox Spectrum switches allow broadcasters to save time and money, and become more innovative. Most operators will continue to separate traffic by priority; however, Mellanox switches have the intelligence to prioritize real-time media streams, critical during the Olympics and other live broadcasts where the action is happening on multiple fronts.

Supporting Resources:

Fueling Science and Research through Machine Perception

The concept of computers being aware of their surroundings and acting upon their perceptions began as fictitious stories around advanced intelligence and consciousness. This, perhaps, was the inspiration that propelled scientists to begin seriously discussing the possibility of designing intelligence into an electronic device. Computers that we are all familiar with have been programmed with a very specific instruction set. Applications are then capable of leveraging these instructions to carry out a wide variety — yet very specific set — of tasks. Until recently, machines have not had the ability to perceive their surroundings and modify actions based on these perceptions, or to increase their accuracy by learning from mistakes. However, this is changing with the arrival of autonomous vehicles; it is through machine perception that we may fuel the next driving force for future innovation.

Machines are already pushing boundaries of human-level performance across the entire spectrum of perceptions, with an ability to surpass even human levels of achievement. For example, we can’t leverage perception through tasks that require systems such as radar, ultrasound, or infrared without the assistance of machines. Computers, on the other hand, have the ability to harness a powerful array of perceptive powers. Great progress has already been made in computers’ ability to harness image recognition, speech analysis, and other such perceptive capabilities. In fact, improvements have been staggering in the past ten years and the pace is sure to continue at an exponential rate over the next decade.

To understand what is possible in the future, let’s first define the terms that are being used today such as, Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL) and how they relate to each other. First, you should understand that AI is the broadest terminology with ML and DL being subsets. Similarly, DL is a subset of ML. Simple definitions of each are listed below:

Artificial Intelligence – performing tasks that require human intelligence to perform.

Machine Learning – a system that can grow its own intelligence.

Deep Learning – inspired by how the brain works and using an artificial neural network of computers.

AI applications are based on the training of computers, which requires complex computations and fast and efficient data delivery. This is accomplished through the use of fast, efficient and lightweight protocols that can streamline the communication and delivery process. Mellanox heritage has been in meeting these needs and is uniquely positioned to provide end-to-end solutions for an organization seeking a competitive advantage for their data. Enabling smart offloading such as RDMA and GPUDirect®, and in-network computing capabilities dramatically improves computer networks and assists in performance attributes required for AI workloads.

If you’ve been following the Mellanox Interconnected Planet campaign, which explores the impact of technology on global issues, you’ve likely read a lot about InfiniBand’s role in AI. Mellanox has proven to be a game-changer in achieving new levels of performance in the race to solve the challenges of scalable DL. This can be seen in the most popular DL frameworks today which utilize multiple GPUs in a server, but have difficulty in scaling throughout multiple servers. IBM Research recently announced an amazing achievement in unprecedented performance and scaling with new distributed DL solutions using Mellanox InfiniBand technologies, which were able to achieve a record 95 percent scaling efficiency on the Caffe deep learning framework. This challenge, in particular, is where Mellanox has been the clear leader as the only interconnect solution able to deliver the performance and offload capabilities required to unlock the power to scale to a 64-node system for DL.


Ethernet for AI Applications

However, Mellanox Ethernet products provide similar capabilities: the most efficient data movement via offloads, accelerators, and RDMA-enabled capabilities all contribute to handling the complex computations and fast and efficient data delivery that enable world-leading AI platforms. Using Ethernet for supercomputing is no novel idea. In 2013, Australia’s National Computational Institute built a world-class, high-end research platform on Mellanox’s interconnect solutions to research coral bleaching in Australia’s Great Barrier Reef.

More recently, Tencent invested in an end-to-end 100 GbE infrastructure for analytics and ML deployments and participated in the TeraSort Benchmark’s Annual Global Computing Competition. The Mellanox Ethernet NIC and Switch solution was a key component enabling them to achieve amazing improvements over previous year’s benchmarks. Achieving more than 1TB/sec of TeraSort performance allowed them to attain record breaking results. It’s an achievement no other company has been able to demonstrate. By leveraging recent advances in CPU, memory, storage, and networking through industry leaders such as IBM and Mellanox, they were able to achieve 5X the performance of the previous record. Solution highlights can be seen below and a solution brief covering the world record results can be viewed here.

Mellanox Ethernet solutions have also been selected by Baidu, the leading Chinese language Internet search provider, for their ML platforms. With the Mellanox Ethernet solution in place, Baidu was able to demonstrate 200 percent improvement in machine learning training times over previous solutions, resulting in faster decision making results. Baidu’s Institute for Deep Learning utilizes a Machine Learning solution to improve voice and image recognition, as well as natural language processing. In return, this helps provide smarter, more useful and more personalized search results. They are also currently using it to research augmented reality, autonomous vehicles and in health care to develop a medical assistant that patients can chat with to receive a reliable diagnosis, similar to a doctor’s visit.


A Promise for Breakthrough Research

AI is all around us and most of us may not even be aware of it. Android and Apple speech recognition on our phones are types of DL that turn speech to text and provide directions through Google maps. The future is here now with many application using AI as a building block that is poised to further intertwine our lives with technology. AI brings the promise of breakthrough research for just about everything, from the way credit card companies detect fraud, to the exploration of space. A prime example of how AI will help research in the future can be seen in health care. Research of the human genome will be combined with an individual’s medical record and provide the capability to access to enormous quantities of clinical studies. AI can then parse through all this data—something that would be impossible for a human to do—to determine an individual’s diagnosis and tailor a specialized treatment.

The amount of data in the health care industry is growing very rapidly; electronic health and imaging records, as well as data from wearable devices and sensors are contributing to a flood of information. While the tsunami of data can be overwhelming to humans, it’s full of possibilities for AI. For health care providers, research could improve patient treatment, simplify regulatory compliance, boost operational efficiencies and increase collaboration between medical teams. Researching the health of large populations will enable AI to recognize potential risks and suggest powerful preventive care. All this will allow us to better address the challenges in taking care of an aging population while allowing much longer independence.



As data volumes continue to increase and processing it becomes more affordable, many more industries are likely to benefit from AI. We can already see this through improved experiences from consumer mobile applications such as Apple’s Siri and Amazon’s Alexa. Many of the top organizations in the world are betting big by investing significant resources in AI including Google, Microsoft, Yahoo, Facebook, Twitter, Tencent and Baidu—and at the center of this investment is Mellanox Technologies. Before long, the ability for machines to achieve perception will be an essential feature in everyday devices, applications and the web. Autonomous cars and voice assistants are only scratching the surface of what AI can do.

Supporting Resources:


Can Technology Help Humanity Reach a Sustainable Environment?

Earth Day, celebrated annually on April 22nd, since 1970, strives to raise awareness and support for improving environmental conditions and the world’s natural habitats for future generations. Increasing greenhouse gasses in the atmosphere, such as carbon dioxide and methane, and the subsequent warming of the planet is only one challenge we are facing. Other issues include the loss of natural habitat and the shrinking of agricultural land which is losing out to global population expansion. Another threatened resource is our supply of fresh water, which needs to increase to support a population that is rising to over 9 billion by 2050. And there are concerns about acidification of the ocean and its shrinking biodiversity, including coral reefs which provide habitat for many marine organisms and protect the mainland from storms and waves. Similarly, there are concerns around the shrinking of the ozone layer, which protects our planet from damaging solar radiation. This is why for the past 47 years, April 22nd has been dedicated to raising awareness around activities that aim to tackle these and other environmental challenges.

Indeed a contributor to our environmental problems is the sheer ingenuity and success of the human race. In fact, our creativity, inventiveness and ingenuity has altered almost every corner of our planet. The challenges we face are only accelerating as each year our planets inhabitants require more food, water, and energy. Could the same human inventiveness that may have placed us in our current state, also provide us with the solutions to our most pressing challenges? It is clear that technology is a powerful tool to help us achieve change and technologies impact on society has, for the most part, improved our quality of life. If applied towards solving Earth’s problems then perhaps we have an answer our challenges.

After all, technological developments in renewable energies are helping to improve the efficiency of renewable energy solution such as solar panels and are providing a way of harnessing the wind without the detrimental effect that long blades have on bird populations. Similarly, efficiency in data centers has the ability to improve application processing, reduce electrical requirements and shrink a data center’s carbon footprint. This is exactly what Mellanox technology is doing for data centers today. Through an end-to-end suite of interconnect products of adapters, switches, cables and optics, Mellanox has been increasing compute efficiency for years. Mellanox solutions enable data centers to achieve the highest efficiencies, subsequently also delivering high-performance, low-latency networks with rich offloads that are capable of accelerating application processing, deliver data faster and unlock system-wide performance.

Another way Mellanox helps with efficiency is by delivering networks that can handle line-rate processing at up to 200 Gb/s speeds and without dropping packets. All this adds to the overall efficiency of a data center and is what Mellanox is referring to when we say ‘Networking Done Right’. This is particularly important within Cloud infrastructure where the opposite can cause system-wide performance degradation. This degradation has the capability of manifesting itself as slow data communication, storage access problems and heightened CPU utilization, all of which can cause a larger hardware footprint that requires more capital expenditure and increases power draw within the data center. Efficiency lessens these factors and thereby reduces the data center’s impact on the environment.

Another aspect where technology is bound to make a difference is the proliferation and access to large quantities of knowledge. Moore’s law tells us that technology will continue to drop in price, which will allow for lower cost computers that can be made readily available to everyone, including those in isolated regions and underdeveloped nations. This will help spread knowledge and give individuals in these countries a better opportunity to use their own ingenuity and access to the World Wide Web to help resolve local or regional challenges. An example of how Moore’s law is dropping prices on technology is the arrival of portable medical devices. This helps not only improve medical care but has the promise of allowing quicker diagnoses. Other emerging technologies such as advancements in quantum computing, nanotechnology, and robotics all provide a range of options for making a difference in the future, from genetic manipulation to end disease to the discovery of alternative energy.

Artificial Intelligence is already making an impact on health care. An example of this is IBM’s Watson, which correctly diagnosed a woman’s rare form of leukemia, something that had stumping medical professionals for some time. The supercomputer read over 20 million cancer research papers, and came up with the proper diagnosis within 10 minutes, suggesting a new treatment that has since been more effective. As computers continue to get faster and smarter, scientist will enlist them in many ways to solve all kinds of challenges. And with the rise of Machine Learning (ML), Artificial Intelligence (AI), and Deep Learning (DL) there is a growing requirement for superior efficiency, low-latency, and high-throughput interconnects. An area where Mellanox happens to excel. The ability to efficiently move data at higher speeds has landed Mellanox Spectrum switches and RDMA-enabled ConnectX-5 adapters as key components for enabling world-leading ML, AI, and DL platforms.

Technology is also causing a major disruption to the consumer market by evolving not just how we produce products but also in how they are being consumed. Cloud Computing, such as Amazon Web Services and Microsoft Azure offer a “Pay for Service” model, where compute is offered as a utility and consumers pay only for what is consumed. The pay for service model is being adopted by many and the disruption it can cause is easy to see in the shake-up of the razor market that subscription service shave clubs have caused. This idea has been expanding to many vertical markets including HVAC filters and is even available with light bulbs where Philips recently introduced “Pay per lux”. For Philips, they maintain ownership of the expensive energy-saving LED bulbs, and sell light as a service. Customers don’t have to worry about replacing bulbs, they buy light and nothing else. Other similar models have taken the mobile world by surprise with sharing services such as Uber and AirBnB. They increase an assets utilization (parked cars and empty beds) and in doing so, lower costs for consumers. All these services have one thing in common, and they share it with Cloud computing and Mellanox, high efficiency. They are better able to meet the demand for products and services without adding waste and unnecessary consumption. If current trends are any indication, we can expect to hear a lot more in years ahead about how technology is helping us make the world a better place to live for the long term.

I have seen a lot of technological advancement in my life, and it was in elementary school in the late 70s when I first started hearing about environmental change. At that time, I envisioned a future society powered by renewable energies and electric cars. While this hasn’t happened yet, I do believe that human innovation with the help of computers will continue to evolve and develop clever new technologies to help us push our planet towards a more favorable, sustainable state. Each year, Earth Day should be a reminder and used as a call to action that encourages us all to lend a helping hand. What can you do? It doesn’t have to be grand and it doesn’t have to be all-or-nothing. Small steps every day can help.  Some things we can do are to turn off lights, leave our cars at home on occasion and walk, bike, or take public transportation, increase our recycling efforts and reduce consumption. So, what are you doing to support and drive significant, positive changes for our planet? Leave us a comment below.

An Evolutionary Jump in Low Latency -Delivering Maximum Benefits to the Application and Network

Addressing Latency with Application Acceleration

Part 3 of the 3-part series

Key decisions facing trading firms and exchanges is the selection of the optimum networking interconnects to achieve the lowest latency. In the first part of this series, I addressed how software improvements developed at the Mellanox Hackathon, and the capabilities of the ConnectX® family of adapters help achieve latencies as low as a 600 nanosecond-well below the current industry standard.  While part two discussed the ultra-low latencies of the Mellanox Spectrum switches-capable of less than 300 nanoseconds of latency. Hopefully, some of you were able to see the live demo by attending the co-sponsored roadshow by Mellanox, Cumulus Linux and CDW, which is just wrapping up. If you missed it, be sure to register for the upcoming webinar Powering the Future of Low Latency Trading Platforms with Intelligent Interconnect on May 24th, 2017.

In this final segment, I’ll wrap up the series with discussions on additional features that should be considered when designing a low-latency solution. I’ll jump into the conversation by mentioning speed/bandwidth, which can have a material effect on latencies. Considering that a car traveling at 60 miles per hour will reach the same destination faster than a car traveling at half the speed. We can apply the same principle to date traversing a network. By simply upgrading from 1 or 10G to 25G or higher, will allows the data to reach its destination faster, cutting into and lowering latencies. This is the number one reason 10GE server connections and networks are being replaced with 25, 50 and 100GE solutions. By upgrading solutions and making a speed jump from 10 to 25GE or greater, organization will see an exponential increase in bandwidth and message rates. 25GbE, provides 2.5x more bandwidth over 10GbE and only a cost increase of about 1.5x. This ensures that the network and connections to servers are not creating a bottleneck that might impede optimum trade execution. Low latency networks provide for better execution rates and allow for more in-line processing for electronic trading and other latency sensitive applications.

Mellanox VMA Message Accelerating Software

Another way an organization will see an immediate drop in latency is through the use of messaging acceleration software, which can enhance the performance of applications by decreasing overall latency and minimize server CPU workloads. This is one of the simplest and most effective ways to increase performance and the result can reduce latency by as much as 300 percent while increasing application throughput by as much as 200 percent (when compared to applications running on standard Ethernet). Mellanox messaging accelerator software, VMA, removes the operating system kernel from the message path, and by doing so, drops latencies. This is accomplished by decreasing the amount of work the CPU needs to do, allowing the system to achieve higher message rates, while decreasing average latency.

VMA is a high performance network offload that is designed to accelerate unicast/multicast traffic and throughput demanding applications. VMA runs on Linux and supports TCP/UDP/IP network protocols and requires no modifications to the applications before implementing. Network processing is offloaded from the server’s CPU by passing the traffic directly from the user-space application to the network adapter, bypassing the kernel and IP stack and thus minimizing the time consuming process of context switching, buffer copies and interrupts. By cutting out all these additional processes, VMA is able to produce extremely low-latencies, and improve overall networking performance.

IP Networks for Broadcast Media

VMA can improve performance of any application that relies heavily on the use of multicast, streaming or requires a high packet-per-second rate, low latency or increased application scalability. This includes a wide array of applications in addition to high frequency trading environments. For instance, it plays well with Next-Gen IP-Studios that are now handling real time, high resolution video where a single 4K uncompressed stream can range from 8.3Gb/s to 20Gb/s. Broadcast, Media and Entertainment is in the process of adopting 4 and 8K video streams.  As this transition occurs, an existing 10GE interface can easily be saturated. Facilitating this transition is also the fact that many studios are moving away from expensive and proprietary Serial Digital Interface (SDI) to off-the-shelf commodity Ethernet products capable of greater speeds and lower costs. Replacing the SDI interface requires a solution that is capable of low jitter and efficient memory utilization, and it also helps to offload the CPU to free up system resources. This is a perfect environment for high-performance Ethernet and VMA message accelerator software.

Low-latency Network Techniques

In the continued search for the ultimate in low latency, there are still other possibilities that need to be discussed. Such as minimizing interrupts on the processor and reducing the processors roll in handling network traffic which can affect overall latency. Through the use of offloads such as RDMA, DPDK and Mellanox’s own ASAP2 technologies, Mellanox ConnectX®-5 adapters are able to achieve performance improvements by performing network processing tasks for the CPU and by bypassing the OS kernel entirely to free system resources. This offers an enormous performance improvement that can be gained over traditional network and transport choices. Simply by moving to a technology stack that supports advanced kernel bypass techniques on a fast underlying high-performance network, the result is a significantly higher data transfer rate at the lowest latency. Together, these offloads can dramatically improve application response time and increase scaling, reducing latencies and increasing performance across a variety of market segments.

Data Plane Development Kit (DPDK)

In case you are not familiar with DPDK, it provides a programming framework that optimizes the data path for applications and in doing so, enables them to process data packets faster. DPDK is beneficial for applications that must handle a substantial amount of Ethernet packet processing or high message rates such as financial applications or in the processing of virtualized network functions.

Let’s take a step back to explain how exactly DPDK is able to increase an applications performance. First, we need to understand that when sending and receiving Ethernet packets from user space, as most applications do, there is a performance penalty that must be paid. These performance penalties occur as a normal process of sending data. First, instructions associated with a task are copied from user space memory into kernel space memory where they can be executed. The CPU is used for this procedure as well as processing the Ethernet protocol stack (i.e. the ISO network layers) for each instruction set from the NIC to the Kernel. All this adds precious time and consumes CPU cycles to make it all happen.

DPDK enables a faster path for applications to communicate with the NIC and bypasses user space to kernel space switching and the processing of the Ethernet protocol stack. Instead of taking the usual route through the network layers with context switching, DPDK bypasses the kernel and network processing entirely. The application is then able to complete the data processing faster and by freeing the CPU from packet processing, and system wide efficiency is increased.

Similarly, ASAP2 enables the majority of data plane switching and packet processing to be offloaded to the Mellanox NIC, while maintaining control plane operations of virtual switches through Open vSwitch (OVS). ASAP2 offloads certain packet forwarding tasks from OVS to the Mellanox adapter also, allowing for delivery of the highest and most deterministic packet performance, with minimal CPU overhead.

Packet Pacing

The final topic I’d like to introduce is Packet Pacing. Packet pacing can be used to optimize data transfers where multiple synchronized streams all send data at the same time. In scenarios like this, often times the sheer quantity of data being sent starts to clash causing overflows on the switch buffers. This effect can be detrimental to content delivery for video streams. Video streaming services must send video to consumers around the world with different access speeds.  Without packet pacing there would be network packet queueing delays, dropped packets, and buffer overflows occurring, each of which can cause a bad viewing experience for the end-user.

Packet pacing regulates the transmission rate based on an interval (which is determined by the round-trip-time (RTT)-the duration corresponding to a previously sent and received packet.) that will not overrun the capabilities of the receiving node. Probably the best example of this is Netflix who utilizes the Mellanox ConnectX-4 adapter with Packet Pacing offload enabled to deliver content to over 100K subscribers from a single streaming box. Packet pacing has been shown to improve performance by 2-4x and eliminates the possible degradation of the overall network transfer speeds. To further enhance media streaming applications, Packet Pacing can be combined with VMA and implemented without the need for any change to the application.


In particular, for High Frequency Trading applications, saving a few microseconds in latency can be worth millions of dollars. For the Broadcast Media and Entertainment market, which has been moving toward higher resolution video, there are struggles to deliver the needed bandwidth that can keep pace with 4 and 8K formats. As these industries try to maintain a competitive advantage and keep pace with technology evolution, Mellanox will continue to evolve IP technology to incorporate a variety of performance and low latency advantages into an end-to-end Ethernet solution. Mellanox switches, adapters, cables and transceivers, and application acceleration software deliver the most advanced networking technologies capable of exceeding low latency requirements for the most challenging environments. They feature the most efficient offerings with the highest density, and lowest jitter in the industry. This ensures that you get investment protection with the ability to start at 25 Gb/s and upgrade speeds as needed with little to no change. And with aggressive pricing, Mellanox allows you to resolve your most challenging concerns at the lowest total cost of ownership for applications that mandate ultra-low latency.

Lowering Latencies to Deliver Maximum Benefits to the Application and Network

Part 2 of a 3-part Series

Financial success in large securities, exchanges and trading houses come when they are able to receive, process, and fill market orders faster than the blink of an eye. Organization with even the smallest advantage can reap huge profits over the competition. This, in-turn, is driving organizations to optimize at the sub-millisecond level. Both the network adapter and switch are critical components in delivering very low latency and therefore, play a vital role in the overall success of latency-sensitive applications. Mellanox, has a heritage in building low-latency and high-performance networks and is the only manufacturer with an end-to-end Ethernet connectivity offering that features switches, adapters and cables. This places Mellanox in a good position to address the growing need for low-latency, high-performance solutions including high-frequency trading and other demanding applications, such as video streaming, virtualization, and big data.

In this second part of my three part series, I’ll be discussing the need for low latency and predictability from the interconnecting network for the Financial Service Industry and how the Mellanox Spectrum switches compare to competing products. This will help you better understand how to implement a competitive advantage. If you missed the first segment discussions on how Mellanox stays at the forefront of providing the lowest latency solutions, you can read it here.

Financial Services Industry

In order to maintain a competitive advantage, financial service organizations must constantly upgrade infrastructure and find new ways to accelerate data analytics, making trading and market data applications one of the most demanding in terms of data center networking requirements. Their goal is to find ways to incorporate the most advanced networking technologies that support ultra-low latency and the highest possible throughput, while maintaining the lowest possible total cost of ownership (TCO). Last year at the HPE Discover conference, Mellanox announced the availability of new 25/100Gb/s Ethernet solutions for ProLiant and Apollo servers that is enabling just that, new levels of networking performance at a new lower TCO (you can also read the blog introducing the joint new product here).

Mellanox has a long list of partners including Cumulus Linux with whom we jointly announced a best in class Network Operating System (NOS) to run on the Spectrum switch platform, (along with others). With the latest release of Cumulus Linux, Mellanox now has the necessary features to play in some of the most advanced data center networks. With support for features such as IP multicasting (PIM) and other added features important to HFTs environments such as hardware based telemetry on packet rates, buffer usage and congestion alerts, we have also implemented slow receiver multicast protection and fine-tuned buffer controls that reduce buffer overruns. To learn more about the advantages, a customer success webinar can be viewed here which showcases an innovative approach to achieving Cloud networking efficiencies of combining Cumulus Linux with Mellanox Spectrum switches.

Mellanox offers one of the most complete solutions for the Financial Services Industry and Broadcast networks and with the unique position of having the only ultra-low latency switches on the market with 25, 50 or 100GE interfaces, the Mellanox solution allows for building the lowest latency networks. The financial service industry is now taking notice and to aid in getting the message out, Mellanox has partnered with Cumulus and CDW to bring a series of roadshows to demo the low latency and rich feature set that Cumulus and Mellanox offer to the High Frequency Trading (HFT) market. The roadshow will be traveling to three different locations.

Latency and Message Rates

Latency is the most important consideration in most financial services applications and is very important in HFT environments. Lower latency means there is less delay receiving market data and less delay in delivering it to applications for processing. It’s also key in cutting delays for executing market orders, this all means that decisions are arrived at quicker and orders can be fulfilled in less time. For these reasons, trading platforms, especially those supporting algorithmic trading, require latencies of less than 5 microseconds with extremely low levels of packet loss. Not only must trading technology solutions deliver low-latency despite a high volume of data that is being moved, but they also must be able to sustain performance through market data traffic spikes, all while minimizing network jitter and properly handling unpredictable messaging outliers. To accomplish all of this requires an end-to-end solution that includes the transport layer of the server and network infrastructure and possibly enterprise messaging software for added acceleration (more on this in part three of this series).

Mellanox Spectrum Switches

The Mellanox Spectrum switch just happens to be designed specifically to handle these type of environments and does so better than any other switch on the market. But don’t take my word for it. I recommend that you download the third party benchmark by The Tolly Group. The Tolly report provides a benchmark of the performance and predictability of the Mellanox Spectrum ASIC compared to Broadcom Tomahawk, which is the merchant silicon chip used in a large percentage of competitive 25/100G Ethernet switches on the market today. The Tolly Group’s testing uncovered a variety of issues with the Broadcom switching ASIC including, frame loss, inconsistency in throughput, higher latencies and unpredictable buffering.

Fig. 1. Only Mellanox is capable of the predictable performance necessary to maintain the lowest latencies for High Frequency Trading environments


The overall performance of latency sensitive applications relies heavily on a switch’s ability to move a frame of data to its destination point. While Mellanox showcased zero frame loss across all frame sizes and testing rates, Broadcom exhibited up to 30 percent frame loss. When both were tested for latency, Mellanox consistently achieved lower latency across all packet sizes – delivering in some cases – as much as a 90 percent lower latency than Broadcom. This report unequivocally establishes the reliability and performance of Mellanox’s Ethernet switch portfolio, specifically demonstrating the capabilities of the Spectrum switches ability to provide data centers with zero-loss frame rate performance while delivering wire rate data transfer at all packet sizes, with better buffering, and the industries only low latency 25GbE or higher switch.

Fig. 2. Mellanox is capable of 4.76B packets per second and running full 100GbE traffic at all packet sizes


Before Mellanox introduced our Spectrum-based switches, there was a gap in the switch market for high-performance, latency sensitive applications due to the low latency switches from Cisco and Arista which are capped at 10GE speeds. Their 25G (or greater) switches average latencies that are ten times higher than their own 10GE switches, making them irrelevant for trading and other such environments where as Mellanox Spectrum switches are capable of less than 300 nanoseconds of latency. As we saw from the Tolly performance report, there is also no silicon on the market besides Mellanox Spectrum that is able to deliver 25, 50 or 100GE throughput in a low latency switch. Mellanox is the only solution the Financial Services Industry has for speeds greater than 10GE due to having 10-20 times lower latency than Cisco’s and Arista’s new 25 to 100GE switches.

In 2017, we will begin to see the first change that will have a drastic impact on latency sensitive environments as 10GE server connections and networks begin to be replaced with 25, 50 and 100GbE connections. This is because 10GE can bottleneck the performance of newer servers and high bandwidth flash-based storage and cannot provide enough bandwidth in ultra-high definition video solutions. In the first two segments of this three part series, I discussed how Mellanox is well positioned to be the adapter and switch of choice for the financial sector as they start to migrate to 25G and greater speeds. In the final segment, I’ll introduce how Mellanox compliments our hardware with acceleration software to significantly enhance application performance through I/O latency improvements, higher bandwidth, and increased message rates, all while reducing CPU utilization. I hope the conversation has keep your interest enough for you to come back to read the final segment of this series.

Lowering Latencies to Deliver Maximum Benefit to the Application and Network (Three Part Series)

Part 1 of a 3-part Series

Low latency has long been an important consideration in many market segments of computing. This includes the Financial Services Industry where a 1-usec advantage has been said to be worth $100 million a year. In the Broadcast, Media and Entertainment segment, increasing video quality video including the latest Ultra High Definition formats require high bandwidth, low jitter (variance in packet delays) networks – so latency must be kept as small as possible.

Applications like these have brought low latency concerns to the forefront as an industry challenge and I’ll be addressing them in this three part blog series. Aristotle said, “The whole is greater than the sum of its parts”, however, when it comes to Latency, there is a variety of approaches that can applied with each helping to reduce overall latency. In this three part series blog we’ll look at several of the technologies Mellanox deploys to offer high performance, low latency Ethernet solutions. In the first segment, I’ll discuss what Mellanox is doing to stay at the forefront of low latency solutions and how we are resolving challenges for the Financial Services Industry. I’ll expand on this topic in the following two segments and discuss the adapter and switch’s role in keeping latencies low. Finishing the series will be discussion around messaging acceleration software and offloads. In the case of Mellanox, we offer several approaches to lowering latencies which can make the overall latency less than the sum of all parts.

Research and Development

Sustaining low latencies takes focus and continued investment in research and development. One of the ways Mellanox places focus on in-house research and development efforts is by hosting and conducting multiple regional internal Hackathons each year. At these events, software development teams are tasked to collaborate and come up with solutions that solve specific industry challenges. This year’s focus was to lower latencies and increase message rates. This was our 8th annual hackathon and we did just that – we set a new standard in reaching low latencies. The starting goal was to break previous latency barriers with new developments that can be implemented in future product and software releases, with the expectation that these solutions will benefit the Financial Services Industry and other markets that rely on low latency solutions. Designing a low latency infrastructure is complex undertaking, especially when the goal is to avoid trading some degree of efficiency for improved latency gains. The teams were up for the challenge and had one advantage, designed around our newest Mellanox ConnectX®-5 Network Adapters which are capable of up to 100 Gb/s Ethernet speeds and 200 million messages per second.

Hackathon Results

The results of this year’s Hackathon were greater than what could have ever been expected. Latencies were reduced against the latest competitive devices at all data points. Furthermore, by thinking outside the box, the winning team demonstrated a whole new way of handling traffic and managed to cut latency down to the hundreds of nanoseconds. By comparison, current products on the market today measure latency in microseconds.

In the Hackathon, our teams demonstrated two drastic latency improvements over competing offerings. One team made use of on-chip buffering, a newly introduced technology available on the ConnectX-5 network adaptors, to cut latency by as much as 33 percent. This improvement will be rolled into our upcoming software release in June of 2017. Second, by using some creative concepts, another team was able to reduce latencies to around 600 nSec. This superior performance demonstrated our ability to lower latency by as much as 70 percent compared to all other NICs on the market. The idea for this breakthrough was first leveraged by Mellanox for use within the High Performance Computing (HPC) market and our InfiniBand products. The concept is based on extending and advancing the in-network computing capabilities of the Mellanox NICs, a new feature within our ConnectX-5 adapters. And while this feature is not yet available to the general market, it will be available later this year.

By placing the focus of the Hackathon on achieving greater performance through lowering latencies, the Mellanox teams were able to design specially tailored performance enhancements that directly impact latency and ultimately improve performance for applications that mandate ultra-low latency. Through a creative and innovative approach, each team was able to reduce latencies and demonstrate performance gains against competitive solutions. With these improvements, organizations have an opportunity to design a network that can easily exceed performance and latency requirements – better than anything on the market today. These enhancement are especially important for time-critical decision making application such as the financial market segment which can use these to their advantage in Exchanges, Trading Platforms and High Frequency Trading (HFT) environments. Ultimately, these factors translate into higher profits for these financial organizations.

Continued Latency Reductions Throughout 2017

As the Financial Service Industry races to the finish line, intent on completing analytical algorithms faster than their competitors, the innovative ideas from the Mellanox Hackathon will continue to emerge throughout 2017 in future software releases and will help to reach new levels of latencies reductions for these firms. This is sure to drive Mellanox even further as a leader in the high-performance market that continues to break existing latency barriers.

In the next blog segment from this three part series, I’ll discuss how Mellanox is further addressing latency and message rates with advancements in switching and adapters specifically for the Financial Services Industry. Following that segment, I’ll discuss the Mellanox software acceleration solution and how this is being used in other market segments too like Media and Entertainment where technology struggles to keep pace with increasing video resolution. I encourage you to check back and read the final two parts of this three part series.


25/50 and 100Gb Ethernet Soon to be Most Deployed Ethernet Bandwidth

Results from the first ever 25, 50 and 100Gb Ethernet plugfest, driven by the 25 Gigabit Ethernet Consortium, are in and the multi-vendor interoperability event has yielded some very promising results that showcase the level of maturity that the technology is currently at.  Before discussing the vendor interoperability 1testing and results, it is important to understand what is driving it and what it took to get us where we are at today. According to Crehan Research1, in only three short years, 25/50 and 100Gb Ethernet will surpass all other Ethernet solutions as the most deployed Ethernet bandwidth. This trend is being driven by mounting demands for host-side bandwidth as data center densities increase and pressure grows for switching capacities to keep pace.  It was only a short time ago when data centers were exploring the need for 10GbE, and in a little over two years, they are now looking toward 25GbE and greater speeds to answer their bandwidth concerns.  More than just bandwidth, 25G technology is also helping to drive better cost efficiencies in capital and operating expenses when compared to 40G, which is driving cloud providers and enterprise data centers to migrate to higher speeds. These efficiencies, in concert with the endorsement of 25G by large cloud providers such as Google and Microsoft, appears to be what will  cause 25G to surpass 40G as the most deployed server access speed.

In an effort to remove barriers to the rapid adoption of 25/50 and 100Gb Ethernet deployments, members of the 25G Ethernet Consortium, an open organization of third parties developers working together to enable and promote the standardization of 25 and 50 Gbps Ethernet, recently gathered to jointly test the interoperability of their networking products in an industry Plugfest. Plugfests are a mainstay of high tech industries for the purpose of verifying interoperability between products from different manufacturers and to help accelerate the growth of the entire industry. This, in turn, helps to build confidence that products will, “Plug and Play”, ultimately benefiting end users because they will feel secure in the knowledge that a multitude of product combinations have all be tested to work together.

The first ever 25/50G Consortium Plugfest was held at the University of New Hampshire InterOperability Lab (UNH-IOL) in scenic Durham, New Hampshire. Founded in 1988, UNH-IOL is one of the most renowned independent providers of broad-based testing and standards conformance services for the networking and data communications industry. The industry’s top manufacturers work with the laboratory to test, collect feedback on and improve their technology portfolios.

Over the years, the lab has developed testing criteria, tools and methodologies that manufactures have come to rely on to validate interoperability with each other. In today’s day and age, no one can go solo as a player in technology. As the industry moves away from proprietary solutions and towards open standards, products have to be able to play well with others. As such, UNH-IOL is no stranger to the open computing revolution. The lab currently hosts events for the Open Compute Project (OCP), Open Networking Foundation (ONF), the Open Networking User Group (ONUG), and Open Platform for NFV (OPNFV).


2The 25G Consortium member companies, along with UNH students, gather in a 27,000+ square foot facility, which houses a multi-million dollar array of test equipment and the latest devices from UNH-IOL member companies, including many of the 25G Consortium members. The testing provides a forum where member companies can collaborate and ensure interoperability of feature sets and configurability between products. The week-long event, which is scheduled to be held annually, includes a wide variety of test scenarios including auto-negotiation, stressed conditions and the use of comprehensive benchmarking, data integrity, and stress-test tools in an effort  to uncover a wide range of possible interoperability, data corruption, and protocol interpretation issues that might obstruct solution deployments in the field. The results of the testing demonstrate a high degree of multivendor interoperability in open environments and that proved protocol conformance is in line with industry requirements and expectations from standard, ubiquitous Ethernet-based solutions. The results were publicly announced and are being made available for download on the 25G Consortium website or can be viewed here.


The UNH-IOL Plugfest testing helps 25G Consortium member companies to develop higher-quality products and aids in bringing open networking solutions to market faster. Furthermore, it provides confidence in the ability to deploy best-of-breed products that can coexist so IT professionals feel self-assured in choosing 25GbE solutions for upgrading existing or building out new data centers. The results of the Plugfest will also be summarized into an, ‘Integrators List’, which will be posted shortly on the Consortiums web site at If you are looking to expand your data center with 25G or greater Ethernet solutions or even looking into this for future projects, I encourage you to download the Integrator’s List when it’s available.


  1. Nov 2016, Server-class Adapter/LOM Market Overview, Crehan Research, Inc.