BGP in the data center? And MPLS? Are you insane? Well, maybe, yes. But then again, I’ve been known to do a lot of crazy things in my time. Isn’t MPLS a core and edge service provider technology, while VXLAN is an enterprise data center technology?
But let’s begin with this idea that technologies are suited to one environment and not another. We hear this all the time with enterprise and service provider networks — “they use different technologies, and different design principles, and…” One of the great untold secrets of the network architecture world is that large scale enterprise and service provider networks often face the same problems in terms of services and scale. Using a different set of tools in one or the other just because the box says, “enterprise,” is crazier than, well… Using BGP and MPLS in a data center. So, first things first, forget everything you’ve ever heard about what you’re supposed to use where.
Once we’ve done that, let’s think about what we really want in a data center fabric. We want a single transport over which we can carry all our traffic, including layer 2 (Ethernet), IPv4, and IPv6. We want something that will allow us to separate traffic into multiple virtual topologies so we can support multi-tenancy. We want something that can stack forwarding information to support service chaining. For the control plane, we want something that allows us to separate the overlay reachability information from the infrastructure reachability information, will scale to millions of reachable destinations, and gives us a lot of policy knobs to play with.
Reread this list. Is any of it beginning to sound a little familiar? Doesn’t it sound a lot like the requirements a service provider has for creating multiple layer 2 and layer 3 virtual circuits over a single physical network? Yes, in fact, it does. And what set of tools do service providers use to create networks with these sorts of characteristics? Oh, that’s right — BGP and MPLS.
So let’s say we’ve set out to build a data center using BPG and MPLS; what set of technologies would you use?
IS-IS or OSPFv3 for the fabric control plane. This provides IPv6 (yes, right, IPv6, not IPv4) reachability through the fabric (top of rack to spine to top of rack). This also provides label distribution through the fabric.
BGP for the overlay control plane. This provides multi-address family reachability information for the overlay control plane, overlay control plane separation, and lots of policy knobs.
eVPN for layer 2 VPNs, which essentially just carries Ethernet addresses in a BGP address family.
L3VPNs for carrying IPv4 and IPv6 traffic over the fabric, top of rack to top of rack, switched through the spine.
This might seem like a complex set of technologies, but consider the alternative — VXLAN with layer 3 riding in layer 2 VLANs? A mix of IPv4 and IPv6 on the fabric to support various services? Playing with route metrics and other odds and ends to manage traffic flow patterns? Sometimes you find yourself trying to move a refrigerator with a smart car.
Sometimes it’s just simpler, in the end, to move to the bigger tool.
Over the next couple of posts, in between talking about how the Internet really works, I’ll be thinking through some of the details of such a configuration. No, I’m not going to give you the configurations — I doubt my configs would work — but I will be thinking through some of the various ideas and snags you might run into with such a design.
Of course, it helps that this is precisely what I’m working on for Greenland right now… Yes, even seals need cell phones.