Like many folks out there, I’m following the rise of “whitebox switching”, and am interested to see if (and where) it takes off. There’s many players out there who are trying to pitch disassociating the software from the hardware, and quite a few hardware manufacturers that are offering various hardware platforms on which to run the switching/routing OS’s that are being put forward to run on top of them.
The hardware vendors are all pretty similar at this point — the ones that I’ve seen are all offering 1U (“top-of-rack” form-factor) switches built around the Broadcom “Trident” chipset family that offer some combination of 1Gbps/10Gbps/40Gbps ports. These include the following:
- Edge-Core Networks (subsidiary of Accton Technology Group), Hsinchu Taiwan
- Penguin Computing, Fremont CA
- Pica8, Inc., Palo Alto CA
- Quanta Cloud Technology (QCT), Fremont CA
One the software side, it’s a bit more diverse… There’s companies like Big Switch Networks that are offering their own software (“Switch Light”, which is a lightweight switch OS that provides an API for OpenFlow control, such as by their “Big Network Controller” SDN controller software.) There’s companies that are “splitting the difference”, such as Pica8, which offers a “dual-boot” implementation of their own traditional Linux-based L2/L3 switch OS, as well as an hardware implementation of Open vSwitch, which can be used in a SDN scenario.
The last company I follow in this space has perhaps the most interesting software offering – not just a Linux-based switch OS, but an actual full Linux distribution itself, optimized for network switches. Of course, I am referring to Cumulus Networks and their Cumulus Linux offering. Their special sauce is basically a device driver that takes the forwarding elements that usually live in kernel space, and push them down into the hardware ASICs so that packets get switched at wire speed. Other than that, and a few tweaks to the open-source Quagga routing daemon (to make the config commands non-modal), it’s “just Linux” (basically Debian 7.x “Wheezy”, so I’m told.)
But if you’re like me, even though I’d used Linux to implement routing and firewalling devices, I hadn’t ever used it to implement a switch. I began to think, how would one configure a typical VLAN-ed switch using straight Linux? Time to read up about Linux bridging, and fire up the lab!
I started by taking an old Dell box I had, and stuffing two more NICs into it (I would have loved to use dual-NIC or even quad-NIC PCI cards, but we were fresh out, and three NICs were the minimum for the lab I had in mind.) My lab plan was as follows:
On the base install of Debian Wheezy (I used 7.2, and then ran apt-get update && apt-get upgrade after the install was complete to install the latest patches) I had to install two extra packages, namely vlan to provide the vconfig command, and bridge-utils to provide the brctl command, which are used to manage VLANs and bridging, respectively. (To install packages in Debian, the syntax is apt-get install packagename – you’ll need to do this as root, or with root permissions via the sudo command.)
Now, using VLANs on bridges in Linux is a bit unintuitive if you are used to working with your typical VLAN-capable switch. Instead of creating a bridge and then using a command to set the VLAN of a given bridge port, you instead create a VLAN sub-interface on the interface you’d like to use as your trunk link, then create a bridge per VLAN, and add the relevant VLAN sub-interface as well as the other physical interfaces you want to be in that VLAN to the given bridge. (The best analogy I can think of for these per-VLAN bridges is that they are like port-groups in VMware vSwitches.) Also, when you create a bridge in Linux, it automatically adds an interface (named the same as the bridge) that can be used like any other physical interface in the system. For example, to have the equivalent of an SVI (in Cisco-speak), all you would have to do is assign the bridge interface with an IP address, which would put the bridge (VLAN SVI) into the local routing table. If you’re confused with all of this (and I certainly was initially!) hopefully the picture above and the config below will make it clear. (This article in Linux Journal also helped greatly in my understanding.)
So having the three physical interfaces eth0, eth1 and eth2 in Linux, the first thing I had to do was create two tagged vlan interfaces for the trunk link to the connected Cisco switch. Having selected the eth2 interface for this link, the commands were:
modprobe 8021q # this command makes sure the VLAN kernel module is loaded vconfig add eth2 2 # add a tagged sub-interface for VLAN 2 on eth2 vconfig add eth2 3 # add a tagged sub-interface for VLAN 3 on eth2 ip link set eth2.2 up # bring the eth2.2 interface up ip link set eth2.3 up # bring the eth2.3 interface up
So now that we have the two tagged interfaces up, we can move to creating bridge instances (one per VLAN) and adding the relevant interfaces to them:
brctl addbr br-vlan2 # instantiate a bridge named 'br-vlan2' ip link set br-vlan2 up # bring the bridge (interface) up brctl addif br-vlan2 eth2.2 # add the 'eth2.2' interface to the bridge brctl addif br-vlan2 eth0 # add the 'eth0' interface to the bridge brctl stp br-vlan2 on # enable STP (802.1D) on the bridge brctl addbr br-vlan3 # instantiate another bridge named 'br-vlan3' ip link set br-vlan3 up # bring the bridge (interface) up brctl addif br-vlan3 eth2.3 # add the 'eth2.3' interface to the bridge brctl addif br-vlan3 eth1 # add the 'eth0' interface to the bridge brctl stp br-vlan3 on # enable STP (802.1D) on the bridge
Now that we have our bridges established, we can look at a few things. First, let’s list the bridges and their assigned ports:
root@debian:~# brctl show bridge name bridge id STP enabled interfaces br-vlan2 8000.001aa00079f7 yes eth0 eth2.2 br-vlan3 8000.001aa00079f7 yes eth1 eth2.3
One interesting thing that we notice right away is that both bridges have the same bridge-id; this is because the bridges inherit their bridge-id from the lowest MAC address on all their member interfaces; since the MAC of the eth2 NIC is the lowest of the three physical NICs, and the VLAN interfaces for the eth2 NIC inherit the MAC from the base physical NIC, the bridges both use the MAC address of eth2.X in their bridge ID. Here is the listing of MAC addresses of all the system interfaces showing this:
root@debian:~# ifconfig | grep HWaddr br-vlan2 Link encap:Ethernet HWaddr 00:1a:a0:00:79:f7 br-vlan3 Link encap:Ethernet HWaddr 00:1a:a0:00:79:f7 eth0 Link encap:Ethernet HWaddr 00:40:05:41:78:99 eth1 Link encap:Ethernet HWaddr 00:40:05:43:1c:f2 eth2 Link encap:Ethernet HWaddr 00:1a:a0:00:79:f7 eth2.2 Link encap:Ethernet HWaddr 00:1a:a0:00:79:f7 eth2.3 Link encap:Ethernet HWaddr 00:1a:a0:00:79:f7
We can also show the MAC address table per bridge with the following command:
root@debian:~# brctl showmacs br-vlan2 port no mac addr is local? ageing timer 2 00:0d:bd:13:a8:8c no 1.20 2 00:16:c8:bb:ca:c1 no 0.00 2 00:1a:a0:00:79:f7 yes 0.00 1 00:24:e8:96:38:70 no 0.00 1 00:40:05:41:78:99 yes 0.00
And we can show the STP port forwarding state per bridge with this command:
root@debian:~# brctl showstp br-vlan2 br-vlan2 bridge id 8000.001aa00079f7 designated root 8000.001aa00079f7 root port 0 path cost 0 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.01 hello timer 1.56 tcn timer 0.00 topology change timer 0.00 gc timer 88.14 flags eth0 (1) port id 8001 state forwarding designated root 8000.001aa00079f7 path cost 100 designated bridge 8000.001aa00079f7 message age timer 0.00 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.55 flags eth2.2 (2) port id 8002 state forwarding designated root 8000.001aa00079f7 path cost 19 designated bridge 8000.001aa00079f7 message age timer 0.00 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.55 flags
Testing with a couple of PCs attached to both the Linux “switch ports” (i.e., eth0 and eth1) and the Cisco switch the trunk link ran to proved that there was connectivity between machines on the same VLAN, and also that there was indeed separation between VLANs. I went a step further and made some SVIs on the Cisco, and after setting the default gateway on the PCs to the relevant IP of the SVI for the given VLAN they were attached to, I was able to route traffic between the PCs attached to the Linux box on VLAN 2 (eth0) and VLAN 3 (eth1).
So, Linux as a switch OS? Not as crazy as it once sounded… And I’ve just scratched the surface of what can be done with Linux on a switch. Now to continue my experiments, and hopefully someday I can get a real switch running Cumulus to play with
 Not really a sub-interface (more like a virtual interface) but I thought that most network engineers would be able to grasp the concept better if I explained it like this…