Switching to Linux for… Switches?

Like many folks out there, I’m following the rise of “whitebox switching”, and am interested to see if (and where) it takes off. There’s many players out there who are trying to pitch disassociating the software from the hardware, and quite a few hardware manufacturers that are offering various hardware platforms on which to run the switching/routing OS’s that are being put forward to run on top of them.

The hardware vendors are all pretty similar at this point — the ones that I’ve seen are all offering 1U (“top-of-rack” form-factor) switches built around the Broadcom “Trident” chipset family that offer some combination of 1Gbps/10Gbps/40Gbps ports. These include the following:

One the software side, it’s a bit more diverse… There’s companies like Big Switch Networks that are offering their own software (“Switch Light”, which is a lightweight switch OS that provides an API for OpenFlow control, such as by their “Big Network Controller” SDN controller software.) There’s companies that are “splitting the difference”, such as Pica8, which offers a “dual-boot” implementation of their own traditional Linux-based L2/L3 switch OS, as well as an hardware implementation of Open vSwitch, which can be used in a SDN scenario.

The last company I follow in this space has perhaps the most interesting software offering – not just a Linux-based switch OS, but an actual full Linux distribution itself, optimized for network switches. Of course, I am referring to Cumulus Networks and their Cumulus Linux offering. Their special sauce is basically a device driver that takes the forwarding elements that usually live in kernel space, and push them down into the hardware ASICs so that packets get switched at wire speed. Other than that, and a few tweaks to the open-source Quagga routing daemon (to make the config commands non-modal), it’s “just Linux” (basically Debian 7.x “Wheezy”, so I’m told.)

But if you’re like me, even though I’d used Linux to implement routing and firewalling devices, I hadn’t ever used it to implement a switch. I began to think, how would one configure a typical VLAN-ed switch using straight Linux? Time to read up about Linux bridging, and fire up the lab!

I started by taking an old Dell box I had, and stuffing two more NICs into it (I would have loved to use dual-NIC or even quad-NIC PCI cards, but we were fresh out, and three NICs were the minimum for the lab I had in mind.) My lab plan was as follows:

Lab whiteboard drawing

lab plan

On the base install of Debian Wheezy (I used 7.2, and then ran apt-get update && apt-get upgrade after the install was complete to install the latest patches) I had to install two extra packages, namely vlan to provide the vconfig command, and bridge-utils to provide the brctl command, which are used to manage VLANs and bridging, respectively. (To install packages in Debian, the syntax is apt-get install packagename – you’ll need to do this as root, or with root permissions via the sudo command.)

Now, using VLANs on bridges in Linux is a bit unintuitive if you are used to working with your typical VLAN-capable switch. Instead of creating a bridge and then using a command to set the VLAN of a given bridge port, you instead create a VLAN sub-interface[1] on the interface you’d like to use as your trunk link, then create a bridge per VLAN, and add the relevant VLAN sub-interface as well as the other physical interfaces you want to be in that VLAN to the given bridge. (The best analogy I can think of for these per-VLAN bridges is that they are like port-groups in VMware vSwitches.) Also, when you create a bridge in Linux, it automatically adds an interface (named the same as the bridge) that can be used like any other physical interface in the system. For example, to have the equivalent of an SVI (in Cisco-speak), all you would have to do is assign the bridge interface with an IP address, which would put the bridge (VLAN SVI) into the local routing table. If you’re confused with all of this (and I certainly was initially!) hopefully the picture above and the config below will make it clear. (This article in Linux Journal also helped greatly in my understanding.)

So having the three physical interfaces eth0, eth1 and eth2 in Linux, the first thing I had to do was create two tagged vlan interfaces for the trunk link to the connected Cisco switch. Having selected the eth2 interface for this link, the commands were:

modprobe 8021q  # this command makes sure the VLAN kernel module is loaded
vconfig add eth2 2  # add a tagged sub-interface for VLAN 2 on eth2
vconfig add eth2 3  # add a tagged sub-interface for VLAN 3 on eth2
ip link set eth2.2 up  # bring the eth2.2 interface up
ip link set eth2.3 up  # bring the eth2.3 interface up

So now that we have the two tagged interfaces up, we can move to creating bridge instances (one per VLAN) and adding the relevant interfaces to them:

brctl addbr br-vlan2  # instantiate a bridge named 'br-vlan2'
ip link set br-vlan2 up  # bring the bridge (interface) up
brctl addif br-vlan2 eth2.2  # add the 'eth2.2' interface to the bridge
brctl addif br-vlan2 eth0  # add the 'eth0' interface to the bridge
brctl stp br-vlan2 on  # enable STP (802.1D) on the bridge

brctl addbr br-vlan3  # instantiate another bridge named 'br-vlan3'
ip link set br-vlan3 up  # bring the bridge (interface) up
brctl addif br-vlan3 eth2.3  # add the 'eth2.3' interface to the bridge
brctl addif br-vlan3 eth1  # add the 'eth0' interface to the bridge
brctl stp br-vlan3 on  # enable STP (802.1D) on the bridge

Now that we have our bridges established, we can look at a few things. First, let’s list the bridges and their assigned ports:

root@debian:~# brctl show
bridge name       bridge id          STP enabled    interfaces
br-vlan2          8000.001aa00079f7  yes            eth0
br-vlan3          8000.001aa00079f7  yes            eth1

One interesting thing that we notice right away is that both bridges have the same bridge-id; this is because the bridges inherit their bridge-id from the lowest MAC address on all their member interfaces; since the MAC of the eth2 NIC is the lowest of the three physical NICs, and the VLAN interfaces for the eth2 NIC inherit the MAC from the base physical NIC, the bridges both use the MAC address of eth2.X in their bridge ID. Here is the listing of MAC addresses of all the system interfaces showing this:

root@debian:~# ifconfig | grep HWaddr
br-vlan2  Link encap:Ethernet  HWaddr 00:1a:a0:00:79:f7  
br-vlan3  Link encap:Ethernet  HWaddr 00:1a:a0:00:79:f7  
eth0      Link encap:Ethernet  HWaddr 00:40:05:41:78:99  
eth1      Link encap:Ethernet  HWaddr 00:40:05:43:1c:f2  
eth2      Link encap:Ethernet  HWaddr 00:1a:a0:00:79:f7   
eth2.2    Link encap:Ethernet  HWaddr 00:1a:a0:00:79:f7  
eth2.3    Link encap:Ethernet  HWaddr 00:1a:a0:00:79:f7

We can also show the MAC address table per bridge with the following command:

root@debian:~# brctl showmacs br-vlan2
port no	mac addr		is local?	ageing timer
  2	00:0d:bd:13:a8:8c	no		   1.20
  2	00:16:c8:bb:ca:c1	no		   0.00
  2	00:1a:a0:00:79:f7	yes		   0.00
  1	00:24:e8:96:38:70	no		   0.00
  1	00:40:05:41:78:99	yes		   0.00

And we can show the STP port forwarding state per bridge with this command:

root@debian:~# brctl showstp br-vlan2
 bridge id		8000.001aa00079f7
 designated root	8000.001aa00079f7
 root port		   0			path cost		   0
 max age		  20.00			bridge max age		  20.00
 hello time		   2.00			bridge hello time	   2.00
 forward delay		  15.00			bridge forward delay	  15.00
 ageing time		 300.01
 hello timer		   1.56			tcn timer		   0.00
 topology change timer	   0.00			gc timer		  88.14

eth0 (1)
 port id		8001			state		     forwarding
 designated root	8000.001aa00079f7	path cost		 100
 designated bridge	8000.001aa00079f7	message age timer	   0.00
 designated port	8001			forward delay timer	   0.00
 designated cost	   0			hold timer		   0.55

eth2.2 (2)
 port id		8002			state		     forwarding
 designated root	8000.001aa00079f7	path cost		  19
 designated bridge	8000.001aa00079f7	message age timer	   0.00
 designated port	8002			forward delay timer	   0.00
 designated cost	   0			hold timer		   0.55

Testing with a couple of PCs attached to both the Linux “switch ports” (i.e., eth0 and eth1) and the Cisco switch the trunk link ran to proved that there was connectivity between machines on the same VLAN, and also that there was indeed separation between VLANs. I went a step further and made some SVIs on the Cisco, and after setting the default gateway on the PCs to the relevant IP of the SVI for the given VLAN they were attached to, I was able to route traffic between the PCs attached to the Linux box on VLAN 2 (eth0) and VLAN 3 (eth1).

So, Linux as a switch OS? Not as crazy as it once sounded… And I’ve just scratched the surface of what can be done with Linux on a switch. Now to continue my experiments, and hopefully someday I can get a real switch running Cumulus to play with ;-)

[1] Not really a sub-interface (more like a virtual interface) but I thought that most network engineers would be able to grasp the concept better if I explained it like this…

Will Dennis

Will Dennis

Will Dennis has been a systems and network administrator since 1989, and is currently the Network Administrator for NEC Laboratories America, located in Princeton NJ. He enjoys the constant learning it takes to keep up with the field of network and systems administration, and is currently pursuing the Cisco CCNP-R/S certification. He can be found on the Twitters as @willarddennis, and on Google Plus.
Will Dennis

Latest posts by Will Dennis (see all)

  • Jason

    Nice overview of software white box switching. I understand some of the advantages especially when it comes to server virtualisation. One of the key benefits I see touted is cost. However a number of manufacturers such as super micro are producing some very cost effective switching solutions. Do we really need to go to this level of integration on linux. It was not that long ago that the ASIC was king, now the CPU has been allocated a ton of new tasks.

    Also to orchestrate a consistent install of your Linux switch could prove challenging. This is why tools such as puppet could be used to provide consistency. Despite this I don’t think that a Linux switch is really that up to snuff in production unless you are Facebook or google with the staff and resource

  • Matt Simmons

    Great writeup, Will.

    I just don’t think it’ll ever really jive with me that the command you use to administer layer two virtual devices is ‘ip’, you know?

    Thanks for doing this!


    • Willard Dennis

      Well, I guess you could still use ‘ifconfig’ but I’m trying to get with the “new” iproute2 program here ;-)

  • theNetWorks

    You missed the biggest vendor in this space Arista Networks. While they build their own hardware there is no reason it can not run on any commodity switch that is a business discussion not a technical one. The nice part with Arista is they have a process which emmulates the industry standard IOS CLI making traditional switch management easy while giving complete bash access on an unmodified Linux subsystem, based on the Fedora core distro.

    • Willard Dennis

      I would hardly put Arista in the “whitebox OS” category… Many vendors (even now Cisco) base their switch OS on a Linux (or *BSD) core. What I like about Cumulus (and what is also I think their biggest challenge) is that they are pitching straight Linux (not Linux-based with a custom CLI shell) as a switch OS. One of the reasons I labbed it up and then thought to write this post is that many (most?) network engineers would never think to use straight Linux as a network device OS (well, for switches anyways) and I wanted to find out what it would be like…

  • Michael

    Nice overview. I really like the idea of having a real linux on a switch. I’m working since ages on IOS, JUNOS, VCS quiet some time on NX-OS and other vendors.
    But as well I heavily work on linux and I like the idea to use tools like tcpdump on a the switch. (I know that it’s possible on an JUNOS and with a different syntax on NX-OS)

    I’m working in a full automated environment using opscode chef. So a linux based switch would also allow us to fire up new machines and to configure the network accordingly. Using chef or puppet you’ll have a consistent installation over the whole infrastructure and if you’re using for a code management tools GitHub you have your config management.

    I think if you spend some time with thinking about all the stuff which can be done with a linux based switch, you’ll find a lot of pros and you’ll save money ;-)

    Next time I need to purchase a switch, I’ll definitely consider Cumulus.

    • oldcreek

      It sounds nice that you can use git, chef/puppet to manage network devices the way you manage servers, but in reality, network most of time is just a dumb pipe to applications, ie, network device configurations do not change as frequently as servers, and if a bad receip or manifest is pushed unintentionally to those devices, the consequencies can be catastrophic.

  • What Lies Beneath

    Great Article Willard. I’m wondering how this would look and work as a VM routing appliance rather than on a white box switch. This will certainly help my research, once it starts… Thanks

  • Willard Dennis

    Here’s a good article on Cumulus and Pica8 that explains their positioning against CSCO, Juniper, Arista and the rest of the incumbent HW+NOS vendors out there…


  • https://twitter.com/kiliasov Kanat Iliasov

    interesting test. kinda clunky, but i’m sure cumulus linux will have more elegant cli syntax.
    It’s funny how people tend to forget that we’ve been using *nix based systems for network gear for quite some time now. Most HW appliances are x86 based anyway, it only makes sense to commoditize switching/routing HW. The bigger question is – will it really drive the price down and deliver on promise of democratizing networking? Or will it just push the price load towards the SW?

    • mstone7699

      Cumulus does not have a more elegant CLI – which is part of its beauty/power.

  • d katoola

    What’s the point other than farting around at home. We buy switches for hardware/asics not the OS. Switch hardware and software is highly specialized. I also don’t understand this talk of linux “democratizing” networking, that’s the whole purpose of protocol standards and interoperability. If the vendor that controls the hardware decides to port their code to linux, we will be running a locked cisco linux build. That is not democratizing anything. Your still going to run a specific distribution designed around those asics you need, and pay for the specific licenses necessary. If you dont need the speed, go to Staples, you will get a cheaper switch then building your own out of X86 or a (very cool, but ultimately irrelevant) board from soekris engineering.