Mellanox Spectrum Linux Switch Powered by SwitchDev

 
Uncategorized

Spectrum Linux Switch enables users to natively install and use any standard Linux distribution as the switch operating system on the Open Ethernet Mellanox Spectrum™ switch platforms and ASIC.

The Spectrum Linux switch is enabled by Switchdev, a Linux kernel driver model for Ethernet switches. It breaks the dependency of using vendor-specific, closed-source software development kits (SDK).

The open-source Linux driver is developed and maintained in the Linux kernel, replacing proprietary APIs with standard Linux kernel interfaces to control the switch hardware. This allows off-the-shelf Linux-based networking applications to operate on the Spectrum switch, including L2 switching, L3 routing, and IP tables (ACLs) at hardware-accelerated speeds.

On top of the above Switchdev enables native control over Temperature, LED and Fans directly thought the Linux user interface.

The combination of the Open Ethernet Spectrum switch and Switchdev driver provides users with the flexibility to choose the best hardware platform and software solution for their needs, resulting in optimized data center performance, lower cost of ownership and higher return on investment.

Installing network switches with a standard Linux distribution turns them into yet another server in the data center. This greatly reduces management efforts, as the same configuration and monitoring tools can be used for both servers and switches.

 

Linux application, OS and Kernal driver on Mellanox Spectrum

 

The Mellanox Spectrum ASIC based Switches

The Mellanox Open Ethernet Switch portfolio is fully based on the Spectrum ASIC, providing the lowest latency for 25G/100G in the market, Zero Packet Loss and a fully shared buffer. The ideal combination for Cloud Networking demands.

The Mellanox Spectrum switch systems are an ideal Spine and Top of Rack solution, allowing flexibility, with port speeds ranging from 10Gb/s to 100Gb/s per port, and port density that enables full rack connectivity to every server at any speed.

Check out this report and get details about our unmatched ASIC performance generated by the Tolly group. Read it to Understand the fundamental differences between Mellanox Spectrum and Broadcom Tomahawk based switches.

By using the Mellanox switches as your building blocks, you will be able to build a high performing leaf/spine data center.

The Mellanox Spectrum switch systems

 

Read more about leaf/spine designs best practices.

The use of the Linux Switch aligns well with Linux Based servers and Containerized workloads deployments

The drive for Containers in the data center requires technologies such as RoH (Routing on the Host), meaning that the same routing stack can run on both the Servers and the Switches by using SwitchDev.

Mellanox Technologies is the first hardware vendor to use the Switchdev API to offload the kernel’s forwarding plane to a real ASIC, allowing full line rate performance of Bridge, Router, ACLs, Tunnels and OVS without traffic going via the Kernel (CPU).

As an example, 2 servers are connected to ports Eth1 and Eth2, L2 connectivity is needed between the Servers, a Bridge will be created in the Linux user space on VLAN 10. In the image bellow we can see the difference when rules are offloaded and when they are not.

The difference between Linux user space with offload with SwitchDev and Spectrum and without

 

Mellanox’s current switchdev-based solution is focused on the 100Gb/s Spectrum ASIC switches (SN2000 Series).

This is achieved by using an upstream driver in the Linux kernel. A user can simply buy a switch, install Linux on it like any other server and benefit from the underlying hardware.

Linux kernel

SwitchDev offloaded features on Mellanox Spectrum Switches:

Visibility and Maintainability Protocols (L2/L3) ACL
  • [ER]SPAN
  • Temperature
  • Fans
  • LED Control
  • ethtool (port counter, FW version, transceiver data)
  • Resource queries
  • RIF counters
  • sFlow

 

  • Bridge – 802.1D
  • VLAN   – 802.1Q
  • LAG
  • LLDP
  • IGMP snooping
  • Unicast v4/v6 router
  • ECMP
  • DCB
  • QoS
  • IGMP flood control
  • 256 VRFs
  • GRE tunnelling
  • Multicast v4/v6 router
  • IPv4/IPv6 weighted ECMP
  • VRRP
  • VxLAN
  • ECN: RED and PRIO
  • OVS

 

  • tc-flower offload
  • Actions: Drop, Forward, Counters, Trap, TC_ACT_OK
  • TC chain template
  • Keys: Port, DMAC, SMAC, Ethertype, IP proto, SIP DIP (IPv4/6), TCP/UDP, L4 port, VLAN-ID, PCP, DCSP, VLAN valid, TCP flags

 

Detailed configuration guide for Linux based protocols can be found here.

 

As an example, by using Free Range Routing (FRR), you will be able to run a full routing stack on top of SwitchDev.

FRR is an IP routing protocol suite for Linux and Unix platforms which
includes protocol daemons for BGP, OSPF, PIM, and many other protocols.

FRR’s seamless integration with the native Linux/Unix IP networking stacks
makes it applicable to a wide variety of use cases including connecting
hosts, VMs and containers to the network, advertising network services, LAN
switching and routing, Internet access routers, and Internet peering.

For detailed documentation of FRR go here!

About Stav Sitnikov

Stav Sitnikov is a Networking Specialist at Mellanox Technologies. Stav has been in the networking industry for over 10 years. For the past 7 years he has worked at Mellanox in various roles focusing on the Ethernet switch product line.

Comments are closed.