I had the misfortune to work on Cisco FEX solution in the early days of that product and had poor experiences with UCS servers/UCS 6xxx FI’s and also Nexus 5k/2k proprietary 802.1BR implementations. While Cisco has solved most of these issues over the last seven years, the experience left me quite cynical that IEEE8021.BR standard for port extension.
It seem clear that vendors find its implementation beyond their competency to write and maintain software for this feature.
From memory, I cannot recall any other vendors implementing Port Extenders but I have an idea Brocade came out with some. HP Networking was a key contributor to early IEEE drafts. Quick search seems to confirm.
So it was a surprise to see Marvell repeatedly blogging about Port Extenders (links in References below) and claiming that this was a way to scale up Ethernet in the data centre.
The Idea Behind Port Extension
My take on the principles behind Port Extenders (PEX) is:
1. That all traffic must flow through a hardware device for control, policy and visibility.
1. that ASICs are expensive (in multiple dimensions) and should be conserved.
2. That bandwidth is under-utilised and can be maximised using sharing/extension
2. Port extenders hardware are cheaper than Ethernet switches because they don’t perform forwarding
A hardware port extender device looks like 1RU switch with 48 front ports and some number of ASICs but no forwarding between the ports occurs on the device. Software port extenders were to be implemented in vSwitch’s of various types.
Port Extenders have other problems:
1. no reductions reduce the cabling or costs of SFP modules (which often make up 50% the cost of switch solution)
2. Software interactions to handle the remote configuration of PEX is difficult since PEX configuraton must come of the switch not the PEX
3. Engineers don’t understand them and this requires costly training & support to create a market.
4. ASICs are cheap enough.
Switching ASICs have a strange dynamic. In the 1990s & 2000s, it was relatively simple to have many different types of ASIC for diffrent needs. As we move into the 16nm production, the cost per ASIC remains substantial but an order of magnitude less than a few years. Bandwidth is cheap and ASIC I/O have followed this principle to rapid reductions in cost.
One solid use case for PEX was to provide lower speed Ethernet ports such as 1G or 10G when using 40G/100G switches respectively. However the SFP+ market has produced breakout cables that share the abundance of I/O channels on the ASIC. That is, ASICs have a large numbers of 10G or 25G inputs and these can be mixed into 10/40 or 25/50/100. Rememeber that 100G is 4 x 25G lanes.
The EtherealMind View
When I first learned about 802.1BR and port extensions in 2010 it made sense. The only place for enforcement was in the physical silicon. Virtual switches were just starting out and performance of them was a major concern. Over time, vSwitches of all types have been optimised and hardware accelerated to the point where 10Gbps is simple and speeds greater than 40Gbps are common.
The cost of Ethernet switches has rapidly declined and now the cost of the SFP+ modules is likely more than 50% of the total solution price. Its not the ASIC thats expensive and thats what port extension was intended to minimise. And whitebox switches take this even further.
The complexity of software in a port extender is substantial. Thats another cost / headache for customers.
Its time to return to all switched and ditch 802.1BR. Lets move on.
Extending the Lifecycle of 3.2T Switch-Based Architecture « Blog – Marvell Retrieved 22 Sep, 2017
Rightsizing Ethernet « Blog – Marvell Retrieved 22 Sep, 2017
Modular Networks Drive Cost Efficiencies in Data Center Upgrades « Blog – Marvell Retrieved 22 Sep, 2017