It’s been way too long since this Texan has blogged. During my hiatus, I’ve been caching topics and will be shotgunning a few over the coming weeks.
Sit back, strap in, and enjoy.
25G and 100G switch news was all the buzz for the last several news. This vendor introduced 100G ASICs, followed by this vendor; that vendor is the first 100G switch followed by this other one. Essentially, it was all a race to be first: to the market but more importantly, to shareholders/investors.
But none of them want to claim to be the first to work in an interoperable fashion. Why? Well, because that didn’t happen until recently (like within the last year or two) with the distinction going to Mellanox. (I would have said Cisco but their supported pluggables for their 25G switch doesn’t include optics at the time of publishing this post.)
When I was still working for Cumulus, I wrote a “gentle” blog post about what Cumulus, along with the OCP L1 Interop Program folks, have gone through to make 25/100G switches interoperable based on the IEEE standards. The post described all the warts associated with 25/100G adoption and is the basis for the remainder for this post.
So…if you haven’t read that post or don’t have a good understanding for 25/100G standards, open that post in a new tab and read it. (Keep this one open 😉 ). It’s cool, I’ll wait till you finish.
All done? Cool.
Now let’s get down to brass tacks: the new 28GHz standards (which 25G and 100G are based on) allow for great improvement and even backwards compatibility to 12GHz based standards (1/10/40G). However, in order to achieve these faster speeds, the standard required features such as Auto-Negotiation (AN) and Forward Error Correction (FEC) need to be set depending on the media type.
This proved to be a big problem for folks wanting to deliver interoperable 25G switches.
Let’s get specific: Broadcom’s Tomahawk ASIC was the first commercially available ASIC for 25/100G and was out before the standards where adopted. Luckily, Tomahawk had all the necessary bits to be interoperable for 100G and we saw the first Tomahawk switches listed on the OCP L1 Interop Program’s Integrators List (https://www.iol.unh.edu/registry/opennetworking) in Q3 last year.
Unfortunately, this is not true for Tomahawk 25G switches. You see, for optics to work in 25G switches, FEC, in-particular Reed-Soloman (RS) FEC, needs to be enabled in every channel/lane. It’s not. Tomahawk was able to do RS-FEC for each 100G port (or group of 4x25G channels). The silver lining is that for DACs (copper cables), the standard mandates that Auto Negotiation is enabled and through this process, the appropriate FEC settings are chosen dynamically which can allow Base-R FEC or no FEC depending on the end point’s capability.
This is not the case with Mellanox’s Spectrum ASIC (the second commercially available ASIC to market, a few months after Broadcom). Those extra few months of chip debugging allowed Mellanox to include all the necessary bits in their ASIC to meet the 25/100G standards for interoperability and as such, were the first to market for a 25G Open Networking Switch last year. We saw the first 25G solutions make it to the OCP L1 Integrator’s List (https://www.iol.unh.edu/registry/opennetworking) in December.
I would expect that Cavium’s XPliant and Barefoot’s Tofino to meet the 25/100G standards since they were introduced after Broadcom and Mellanox and after the 25/100G standards were ratified respectfully. (I don’t have access to hardware with those ASICs so I’m going on a hunch.)
Today, we see several Open Networking vendors putting out Tomahawk-based 25G switches such as EdgeCore AS7312 (an OCP contribution) and Quanta IX2. Luckily, through the nature of open source collaboration, the EdgeCore box is being re-fitted with a new ASIC from Broadcom called Tomahawk+ (Tomahawk plus a few fixes like proper RS-FEC per channel) before being made generally available to customers. Much props to EdgeCore and the OCP Networking community for putting the customer first.
We also now hear that more vendors will be moving towards Tomahawk+ for 25G switches and expect those to be available starting later this year such as Alpha Network’s OCP contribution for a 25G switch, the Tomahawk+ based SNC-60×0-486F, code named “Mont Blanc“.
What’s interesting though is when you hear Dell announce on May 8 to have the first 25G Open Networking Switch available in Q3 this year, it’s safe to say they are either going to use Tomahawk+ from Broadcom or another ASIC vendor like Cavium.
That’s right, I mentioned Cavium as a possibility because:
- They are not going to use Cisco’s Custom Avago ASIC for a 25G switch
- I don’t see Dell re-branding the Mellanox 2410 25G switch (though, they probably should)
- This is the ASIC that Arista uses for their 25G switches
Given all of this “news” and “history”, I would expect to start seeing 25G ToR adoption to pick up.
Hopefully for the next generation of 200/400G speeds (read: 56GHz and 112GHz lane speeds) which is coming soon (next couple years), we’ll learn the 25/100G lesson and focus on interoperability and the first interoperable products at those speeds based on ratified standards and not who was first to market.