Packet Pushers sponsor Pluribus Networks sent along Robert Drost to bring us this blog post. He’s a pretty interesting guy.
Robert Drost was a Sr. Distinguished Engineer and Director of Advanced Hardware at Sun Microsystems. Robert has extensive hardware experience, including over 90 patents and a 17 year career in high-performance computing systems. Among other recognitions, Robert has been cited by MIT’s Technology Review TR35 as most innovative computing technologist under age 35, and by the Wall Street Journal with a Gold Medal for Innovation in Computing Systems.
Here’s Robert’s post.
In the next few blog posts, I’ll share my perspective on the state of SDN, the concept of network virtualization and hypervisors, and an attempt to create an ‘SDN Landscape’ to put the various approaches into perspective with a goal of better understanding the potential advantages of one approach over another. The goal is to up-level the conversation from overlays, underlays, and above the fray regarding where in the data center the intelligence should reside, while at the same time showing how it relates to the physical network infrastructure – the switches and routers – are well as orchestration approaches such as OpenStack.
Battle-lines between software- and server-centric approaches vs network- and hardware-centric approaches.
First off, as we know, SDN is sometimes a vague term, and can really mean any kind of programmability added to a network. This is its promise and its Achilles heel. For example, it could apply to data center switches, to server hypervisors, to network virtualization, and even to Network Function Virtualization (L4-7 virtual appliances) and WANs (including Layer 3 and even MPLS). Here I’ll focus on the first three domains.
To really understand SDN, a taxonomy helps, depicting where we run the various components required to program the network. We have the SDN Management Plane (aka orchestration), responsible for policies and deployment goals, the SDN Control Plane, which interacts with the management plane and provides real-time control, and the SDN Data Plane, were we actually forward traffic.
When we talk about SDN Controllers, we are referring to the control plane, responsible for figuring out all the configuration changes to make the data plane capable of implementing the high level policies and goals. Then there is the data plane, consisting of both physical switches adapted to SDN control (or operating in both SDN and ‘classical’ modes at once – hybrid operation) as well as virtual switches (vSwitches), carrying out the actual packet processing. To complete the picture, ‘southbound’ interfaces are between the control plane and the data plane, while ‘northbound’ interfaces interact with the management plane. This is also where OpenStack would tie into the deployment.
With this understanding, let me present an SDN Landscape. The first approach is server-hosted SDN, which I’ll refer to as ‘Type 2.’ This was the first type of SDN popularized, heavily promoted by Nicera, now VMware’s NSX. It’s really a fairly pure software play on SDN, adding the data plane into the hypervisor as a software virtual switch. It is promising because it works with existing networks—no rip and replace! But, and this is a fairly substantial BUT, it has two key issues.
One is performance: standard x86 CPUs are great, don’t get me wrong, but using your x86 processors in servers to tunnel, encapsulate, de-encapsulate, and SSL all your data packets is kind of a waste of good compute silicon. After all Switch and NPU chips designed for packet processing are about 100x (!!!) more cost and power-efficient at doing so. Not chump change or Watts at all.
Perhaps even more troubling for this SDN architecture is that it willfully splits the network into the real, physical network–that CCNA, CCIE Cisco-certified network administrators run–and separately the overlay, or virtual network, that a server or DevOps admin would typically be in charge of. Like ship passings in the night, who can trouble-shoot problems and take responsibility for the combined network? Life is already hard for CIOs.
The second approach is network-hosted SDN, which I call ‘Type 1.’ With network-hosted SDN, the data plane is moved out from the hypervisor into the network hardware and chips themselves. Makes sense, doesn’t it? And there is a parallel in history.
At first VMware popularized virtual machines (VMs) without any help from the CPUs guys. In fact, the CPU vendors were fairly luke-warm to server virtualization. After all, consolidating 20 servers onto one CPU doesn’t exactly strike most Harvard and Stanford MBAs as a way to sell more x86 chips!!
But over time the tidal wave of virtualization took hold and CPU vendors like Intel had bigger fights with the likes of AMD (remember Opteron versus Itanium?) and Intel added hardware primitives right into the x86 CPU chip to hardware accelerate some of the tricky and slow parts of VMs. And the rapid growth of the web and clouds has outstripped the consolidation anyway.
In this second phase of virtual machines, what was originally proved out as software emulation of a virtual machine now became HW accelerated while maintaining all the flexibility of software emulation. Hardware acceleration of virtual machine primitives improved the performance of VMs while still maintaining their software flexibility. And it was just in time for VMware and VMs mass adoption throughout the industry.
This shift was coupled with hypervisors moving from software emulation (Type-2) to bare-metal KVM (Type-1). And I use this terminology here as well to highlight the difference of software emulated SDN (Type-2) versus network-hosted and hence hardware accelerated SDN (Type-1).
Note that my impression so far of Cisco’s SDN offering, ACI, is closest to this architecture. It has some data plane processing in HW–actually via tagging of packets with an additional header. However, the SDN control plane is still largely centralized and separated from the SDN data plane.
Having outlined the differences between Type-1 and Type-2 SDN, let’s look at the management and control planes in greater detail. I lumped these into the ‘management’ box component of the data center hardware. This is because there is (unfortunately) a ‘centralized controller’ mentality pervasive in the SDN field. This idea of there being a central-truth and central-control of a distributed system has been seen before in other large-scale datacenter elements, for example parallel large-scale high-performance computing. Or, early attempts at building computer networks.
What was found out is that it is really hard to know everything while at the same time not knowing if your actual data (state) collection has become out-of-date, thereby making the whole exercise useless for making good control decisions. This is one of the reasons Memcached is very popular and scales great across hundreds of racks of servers, but shared coherent memory does not.
SDN Controller Challenges
This central control dilemma has been referred to in computer science as a kind of manifestation of the Heisenburg uncertainty principle: you can know some information in a timely manner or all information in an untimely manner, but you can have both. Now it is possible to work hard and patch a central controller architecture with a master-slave hierarchy and redundancy, etc. to get them to work at some scales, but the physics are against you.
Large scale server topologies solved the problem with distributed and clustered control plane. I’ve seen that in storage scale-out control planes the same distributed control ideas have been successfully applied. The diagram below simply shows that choice of bringing the SDN control plane from a centralized to a distributed layer right next to the data plane, a more optimized and scalable architecture.
Note that that the SDN Management plane is left up in the external Management boxes (or management VMs in some cases). The idea is that Cloud management, ala OpenStack, and SDN policy management and distribution should be accomplished on a global scale. Just the fast local control decisions made against those policies can be advantageously made in locally near the action of the data plane.
Last but not least, in parallel to a more distributed SDN architecture, another major development will change the way we look at networking, and thus the demands placed on management and control, within the rack.
Evolving ‘microserver’ architectures support upwards of 800 low-cost servers (CPU chips) and 6400 cores within a single rack. Power and footprint savings are substantial. At this scale, it becomes prohibitive to run 800+ Ethernet ports from individual NICs to the TOR or end-of-row switches. The solution, adopted by Supermicro and others, is to place the TOR entity directly in the server chassis. Here, there may be more than 20 integrated switching chips where there used to be a comparable number of NICs.
The ultimate evolution of this architecture is to integrate the 10G NICs directly into the switch chips, and even into the CPU itself. This architecture mandates a fully-distributed SDN control plane by distributing switching and therefore the SDN data plane ‘inRack’ and creating scalability requirements beyond that of either centralized controllers or Type-2 approaches.
Now that we have examined the differences between server-based (Type-2) and network-based (Type-1) SDN, and proposed the advantages of a distributed control plane, the real question still remains… what practical needs does SDN address? We talk about programmability and control, but network managers and CIOs don’t deploy or fund a ‘control.’
More practically, we can look at three types of capabilities – network virtualization (aka micro-segmentation), analytics and control, and Network Function Virtualization including virtual appliances, in-network IP services, and what is known as service-chaining. Here I’ll focus on the first.
In its most basic form, network virtualization is an application that runs in and over the network and that uses the SDN infrastructure for programming control and visibility. As I previously described, there are two fundamentally different SDN deployment models, and therefore, two ways of implementing virtualization. The first is via-software emulation within the hypervisor’s vSwitch (ala NSX), and the second is via support on the actual network switches. The real value of network virtualization is that servers and storage are never again in the wrong place. It frees up the infrastructure management to use resources wherever they are in the network, with services turn-up time and troubleshooting greatly enhanced.
Now, what defines virtualization – how do you know it when you see it? A while back, Popek and Goldberg developed a virtual machine litmus test – “A virtual machine should exhibit a behavior essentially identical to that demonstrated when running on an equivalent machine.” We can apply the same test to network virtualization.
My network virtualization litmus test – “A virtual network should exhibit a behavior essentially identical to that demonstrated when running on an equivalent network directly.” Basically, it must guarantee compatibility to existing servers, storage, and applications. They must have a ‘network experience’ that looks and acts like a physical network.
Note that not only semantics and isolation around IP, MAC, VLAN, etc. address spaces are important, but if you want to avoid having greedy or malfunctioning VMs in one virtual network slow down or stop another virtual network, then resource controls, like QoS, congestion control, rule limits, bandwidth guarantees are critical. Some cloud providers have been noticing that such SLAs around virtual networks can be a valuable commodity to offer.
And a graphic may be the best way to understand this – the parallels between compute virtualization via VMs and network virtualization. We have the underlying hardware, either CPUs or switching silicon, a clustering – resource pool layer that allocates resources, and the actual ‘zones’ reflect unique slices of the physical network infrastructure. Also note that in a network-centric Type 1 SDN deployment, the network hypervisor handles both the L2/L3 switching – the SDN data plane – as well as the network virtualization. There are no separate switch and hypervisor operating systems.
Lets put this all together into a simple way of looking at the relationship between the physical and the virtual, and as they relate to both the servers and the network. Interestingly, the conventional approach has been to work our way from physical machines (pre-virtualization) and physical networks (PM+PN) to a world of virtual machines and physical networks (VM+PN) and then in the last few years to virtual machines and virtual networks (VM+VN). But what was missing was the fourth and final quadrant (PM+VN), where we can combine at-will legacy and evolving software stacks and operating systems. Where can deploy any combination of physical and virtual in conjunction with any network architecture. This is the promise of Type 1 SDN, and the basis for the truly elastic data center.
For more on the Pluribus approach to network hypervisors, please see Netvisor: the Pluribus Network Hypervisor Deconstructed.