Last week I was at a Cisco users group meeting where some sales engineers were giving a presentation on the new Application Centric Infrastructure (ACI) architecture and Nexus 9000 products. It was a very high-level overview, but it was interesting. I had assumed when Cisco made the ACI announcement that it would be based on OpenFlow. As the SEs were going through some of the slides naming different protocols, something dawned on me; There was no mention of OpenFlow! So I asked the question, “Is this not based on OpenFlow?” The speaker paused for a second and replied, “It uses VXLAN, ISIS and LISP.” I tried to ask more targeted questions after that, but he declined to give more detail.
Then a light bulb clicked on in my head about how those 3 protocols may work together in an infrastructure like this. In a way it seems not much different than MPLS VPN in theory.
Disclaimer: This is a theoretical brainstorming exercise and it does not contain any official Cisco information about ACI. It is pure speculation of how ACI COULD possibly work based upon the only known information that ACI uses VXLAN, ISIS and LISP.
MPLS VPN and any other SDN technology needs to achieve 3 primary goals to function –
- Tenant Isolation – The ability to virtually separate multiple organizations to prevent unauthorized access to traffic, data or virtual infrastructure. Without this no one would trust the “cloud” and life as know it would cease to exist.
- Location Services – The ability to know where in the network a tenants’ resource is located. It wouldn’t do much good to have customers connected to different points in the network if it didn’t know the destination.
- Path Selection (routing) Protocol – The ability to send the traffic in the right direction toward its destination.
While these basics would be common to SDN and MPLS VPN, SDN would add to these building blocks with an added orchestration layer that would centrally configure and manage the whole network.
Before we get into ACI, let us review MPLS VPN from a high level. MPLS VPN provides a highly scalable and secure multi-tenant infrastructure that, compared to its predecessors ATM and frame relay, is a pretty simple design and has great operational efficiency (at least from a carrier’s perspective).
MPLS VPN works by adding one or more labels to the packet between the layer 2 and layer 3 header. The typical MPLS VPN configuration uses two labels in the packet. The first, or “top” label is used by the IGP to identify the destination provider edge (PE) router that the packet needs to go to. The second, or “bottom” label is the VPN label which is used to identify the VRF that the packet belongs to. A comparison of a standard Ethernet frame and an MPLS VPN frame is below.
MPLS VPN achieves the our goals in the following fashion –
MPLS VPN isolates tenants inside the PE and across the network with a combination of some of the following elements –
- Virtual Routing and Forwarding (VRF) – VRFs isolates tenants on the PE router by keeping their routing tables separated from each other.
- Route Distinguisher (RD) – The RD is the identifier that is unique across the network. This is the main ID for the tenant.
- Labels – The labels “isolate” the packet as it travels along the network by identifying where they are supposed to go.
Location information of tenant resources can be pseudo-centralized through the use of MP-BGP route reflectors. An MP-BGP route reflector keeps the customer prefixes, RD, next hop, VPN labels and other attributes in the BGP RIB. Thinking of the tenants IP prefix/RD combination as the tenant resource, the next hop for the prefix (which is the loopback address for the PE router) would be the “location” of that resource.
Path Selection protocol
Now that you have the location of the destination resource, you need a path to forward your packet along. With MPLS, you run an IGP for that. OSPF or ISIS are the 2 of the IGPs that can be used in MPLS along with Label Distribution Protocol (LDP). Without getting into too much detail of how MPLS forwarding works, let’s simplify the explanation by saying that MPLS backbone routers (commonly known P routers) don’t use the destination IP to forward the traffic as it travels along the network (except to look up the label on the first PE router). Rather, it looks at the input label to make routing decisions, decides what to do to the label on the outbound side and sends it to the next hop. Basically the P routers don’t care about the tenant’s prefixes, RDs, VRFs, etc. are; they only care about getting that packet to and from the “locations” (the source PE router loopback to the destination PE router loopback).
An example topology of MPLS VPN is shown below –
Before we cover ACI let’s take a look at one of the more popular physical topologies for an SDN network just to give us some background. The topology I have heard talked about the most for SDN networks is the “Leaf and Spine” architecture. In this design the 2 or 3 layer tiered networks based on HA switch pairs is replaced by edge switches called “leafs” which are attached in a mesh like fashion to two or more big spine switches. One of the big benefits to this is that East-West traffic from moving big data loads such as VMotion will have more bandwidth and less hops. This is a rough example of a leaf and spine network –
ACI, using VXLAN,LISP & ISIS, could achieve our theoretical goals. Since many of the details about ACI are not known yet, this theory will be at a high level. A lot of these are new protocols to me but I will try my best to explain the basics. Here is a possible theory of ACI –
Using VXLAN (Virtual eXtensible Local Area Network), tenant isolation can be achieved. VXLAN is a technology in the IETF draft stage that identifies tenants with a 24 bit number known as a VXLAN Network Identifier (VNI). While you could almost think of this as a fancy VLAN, the 24 bit VNI allows up to 16 million tenants whereas VLANs are limited to 4094 (and good luck trying to find a switch that allows you to run that many active VLANs). VXLANs start and end on a Virtual Tunnel End Points (VTEPs). To travel across the network, VTEPs encapsulate most of the original Layer 2 frame from the source VTEP in a VXLAN UDP packet. This encapsulation could also allow a VXLAN packet to be sent across a standard L2 or L3 network (aka an overlay model). This UDP packet includes the VXLAN ID in a header just after the UDP header but before the original Ethernet header as shown below.
ACI may use other things in addition to VXLAN for tenant isolation, but it seems to me that the use of VXLAN makes the most sense.
Now that we have a way to keep tenants isolated and encapsulate their packets across a network, the source leaf switch would need to know where to send a packet. This is where Locator/ID Separation Protocol (LISP) could come in.
In a conventional network the destination IP is also the location. When you send a packet to another endpoint, the destination IP is used to make the routing decisions. If the endpoint moves to a different network, the IP must be changed. The LISP protocol separates the location and end point addressing by defining a location as a Routing Locator (RLOC) and the endpoint address as an Endpoint ID (EID). An endpoint can move from one network to another but would still retain the same IP address. LISP accomplishes this by encapsulating packets from the Ingress Tunnel Router (ITR) and sending it to the Egress Tunnel Router (ETR).
While it remains to be seen whether ACI would use the tunneling from LISP (since VXLAN can already do that) I would think the most useful part of LISP would be the location services. To locate an EID, the LISP ITR would have to know which ETR the EID is at. After all it couldn’t just send a packet to a random place. A mapping server provides the location information in the form of EID-to-RLOC map records. To send traffic, you would query one of these servers to find the right RLOC for the EID you wish to communicate with. This is similar to a DNS lookup (as a matter of fact the RFC compares this to DNS). In the case of an SDN data center network , the RLOCs could be the leaf switches and the EIDs could be the VMs or other endpoints. Since the end point VMs can be located on any leaf switch in the network and can also migrate to a different leaf switch at any time, LISP could work well to provide location data. Now the leaf would be able to know where to send a VXLAN packet to.
Path Selection Protocol
With the destination leaf becoming known though LISP, the VETP can encapsulate the packet and the network can use ISIS to get it to the right destination leaf node. There are 2 reasons I suspect that ISIS was chosen –
- ISIS works at layer 2 – While ISIS and OSPF both use the Dijkstra algorithm for calculating paths, ISIS can route non-IP protocols as it was originally designed for DECnet. ISIS has been used in TRILL (and Cisco Fabric Path), 802.1aq SPB, and OTV implementations to more or less “route” MAC addresses. This way ISIS could carry some other information that ACI might use for path selection such as VNI, MACs or maybe even latency.
- Familiarity – Because ISIS is used in Fabric Path and OTV, it would be well known technology for Nexus developers. All they would have to do is make a few tweaks to the protocol.
It still remains to be seen just exactly how ACI works but I suspect as we get closer to the date when the controller starts shipping we will find out more. While it would be really cool if my theory was close, there is probably a better chance that I am way off. But hey, that is what us nerds do; think up stuff all day just for pure enjoyment. Sometimes we also get paid for it.
Luc De Ghein ; “MPLS Fundamentals”; ISBN-13: 978-1587051975
IETF; VXLAN Draft
IETF; LISP; RFC 6830
IETF; LISP Map-Server Interface; RFC 6833
Wikipedia; 802.1aq SPB
IETF; “IS-IS Extensions to support OTV” Draft
IETF; “TRILL Use of IS-IS” Draft