More Open Flow Symposium Notes…

1. OpenFlow and standards.

“Laws, decrees, edicts, ordinances, writs will shower down upon the poor people like hailstones.” -Pierre-Joseph Proudhon

Part of the appeal of any potential new order is that it always promise things will be simpler, however in time as reality turns in its vote, the new order becomes an institution with beauracracy. Its body of laws will grow and soon the promise of simplicity will start to fade. Any body (i.e., legislative body, standards committee) responsible for producing abstract guidelines for behavioral conformance suffers from this. You would imagine that “Do not @#!%ing kill your neighbor” is pretty clear, but as it turns out… its not. Standards and laws evolve as technology and people evolve. Even when guidelines are passed they are usually eight-sided things and reality is then a nine-sided thing so there are gaps at the outset. This is how these things work.

Rob Sherwood of Big Switch Networks rightly points out that after so much time the IETF has amassed thousands of RFCs. IP was such a simple idea in the beginning.  Each subsequent RFC patches up some shortcoming in a previous RFC or it introduces some new protocol. They mostly deal with nodes with distributed control-plane intelligence and how to get those nodes to behave in various ways. Rob is correct, making a standards conforming packet-switching device is difficult today. There is much to cram into a device that has a relatively little CPU.  Add to this that existing network OS’s have been around for a while and have some baggage in them.  These things make it hard to produce new features quickly. Igor Gashinsky from Yahoo! made a great slide explaining the the impact of this: If you need a new feature, you pay up. Or you threaten the vendor that you’ll leave. The vendor may or may not adequately follow through.

Igor also made it clear he thinks OpenFlow is relatively simple and that it just works. Simple is a relative term. Right now, OpenFlow seems relatively simple. However it now has its own institution: the Open Networking Foundation. They have a roadmap for different versions of the OpenFlow standard. They have much internal disagreement (as any healthy body of this nature would). There is probably some splintering within the ONF. As the standard grows and introduces new features so the complexity of the software (and firmware) supporting the standard grows. Throw in the need for differentiation amongst vendors and the support of vendor TLVs and things start to get even hairier. Nicira’s extensions (such as for NAT) are distributed with the OpenFlow reference implementation itself but are not part of the standard. Now they are speaking of a potential capabilities negotiation. SIP anyone? Its true that in the future OpenFlow v2.0 nodes will be still be simpler than todays routers and switches, but like any technology.. the standard will grow, non-standard extensions will proliferate, and in general.. it will get more complicated from a field engineer’s perspective.

The complexity of programming distributed packet-switched nodes must move somewhere if you are pulling the intelligence out of the nodes themselves. The proposed systems to do this are not necessarily simpler. In Yahoo!’s case, they propose maintaining a database with 100% accuracy of millions of devices with their IPs, MAC addresses, VLANs, and ports that they are attached to. They intend to hook this up to servers with OpenFlow controllers and software that will compute the paths in the network. There are other databases that they will also use: databases with circuit costs for instance. Someone has to ensure the accuracy and integrity of these databases and they must also ensure that the interfaces between those various databases and the controller software is functioning. Someone must ensure the stability of the controller itself and the associated software modules. Its not going to work perfectly all the time, so someone must also troubleshoot it.

For instance, have a look at the functional diagram of Apache’s RouteFlow:

Is that necessarily simpler? At the top of this diagram there is a virtual-machine for each routed node you wish to create.  Have you managed a linux server before? This guy has. I actually love linux, but I must concede that running into dependency and compile time issues is extremely frustrating and *common*. I hope that mandatory patch doesn’t take out your network. Look, the complexity has moved somewhere else. The SDN body of standards and APIs will grow to encompass APIs and interfaces at the controller and application layers. People will grow tired of the databases they manage and software packages with APIs will appear to ease the pain. Applications with various APIs such as Quantum will appear.

Implicitly, none of this will just work. If it did, we wouldn’t need all those standards would we? There will be OpenFlow nodes that do not support version “x”, but they do support version “y.” Or they might just support some features from version “y.” What about vendor extensions? Thats just the OpenFlow part. What about everything else? Not everyone has a team of programmers to support the network. What if some package I need has dependencies that conflict with some other package I am using? This *will* happen. Imagine installing software on your controller(s) in such a way that now your whole network is impacted. There will be some growing pains here folks… its bound to happen. Commercial packages will not be perfect either.  And there will be smart folks on the ready to produce more APIs and standards and best practices to smooth the way.

Ideas are simple. Implementations often are not. All this may sound like I’m being overly negative, but thats not true.  I think there is great potential for OpenFlow, but I want to be realistic about it.  The Symposium was a conversation that was grounded in reality. Real network people are talking about and working on OpenFlow.   Many of the issues I previously blogged about are being addressed. This is a good thing. In order for OpenFlow to reach its potential,  we need brilliant people actively developing the standard and hashing these things out. By the looks of the #OFS11 panel, its clear the right people are involved.  The good news is that there are products shipping today such as NEC’s that actually work. OpenFlow is off to a good start.

2. OpenFlow and the x86 instruction set

I was a programmer before I was a network engineer. I know something about assembly language. OpenFlow is not the “x86 instruction set” of networking. Everyone needs to stop saying that. Here is the most amazing thing you will see all day:

1. JMP LABELA;
2. ADD REGB, 4;
3. MOV REGC, 0×8000;
4. LABELA:
5. SUB REGD, REGC;
6. MOV REGA, 5;

Beautiful isn’t it? I miss programming in assembly. That is actual code for the EZ-Chip NP-1 NPU. OpenFlow, on the other hand, is really just a protocol. There is a thin OpenFlow processing layer on the forwarding node itself (the firmware) that will receive these OpenFlow add/mod instructions and then do something appropriate with them as required by the underlying hardware.. such as generating microcode for an NPU to instantiate flow entries.

3. Juniper’s OpenFlow announcement

I got a chance to sit with David Ward from Juniper at lunch and talk to him about OpenFlow support in Juniper’s SDK. This was quite impressive. Ivan Pepelnjak and myself asked about support for existing routing protocols in an OpenFlow network during the symposium. Igor from Yahoo said “Just use static routes to point into the OpenFlow network.” Er.. no. However, NEC indicated they think another year or perhaps 18 months and we should start seeing more software packages for controllers that run traditional protocols. OK, sounds nice…

David/Juniper had a more compelling answer. With their SDK there could be one or more OVS switches residing on an MX. Each OVS switch could correspond to a tenant/security-zone if you wanted to do that. Interfaces from VRFs (@tonhe: VEEEERRRRRFs) or virtual-switch instances can be mapped to each OVS switch instance. A controller can insert a flow entry into a VRF and have that entry (assuming its destination IP-based)  redistributed into BGP in that VRF instance. You could have an OpenFlow infrastructure with an MX sitting at the top of it to integrate with an existing MPLS network. Umm… Juniper… its hard not to be a fan these days.

Dave did mention the possibility of integrating with BGP Flow Spec, but did not indicate if that was supported now in conjunction with OpenFlow in the SDK. Also it looks like JUNOS does not support the ability to match on Source IP/Prefix or Source Port (TCP/UDP) with the Flow-Spec feature hmmm…. it is in the RFC though. At any rate, I sense an article in the future that fully explores this intriguing new addition to Juniper’s SDK.

4. Some OpenFlow controllers use MAC addresses kind of like MPLS labels or DLCIs.. but they should probably just use MPLS labels now.

One question that has bugged me since the beginning of the OpenFlow hype boom is, well, how does an OpenFlow network… work? If you are building virtual-networks with an OpenFlow v1.0 based setup, how do you forward traffic in the core? Also, Ivan P has repeatedly brought up the issue of state explosion in the core. So really the solution must solve both things: Be able to distinguish between virtual networks and also prevent state explosion. Without MPLS label support in OpenFlow v1.0, I guessed a couple weeks back that the solution to this problem would be the use of locally-administered MAC addresses. It turns out, I was right: Some vendors are indeed doing this. Let me describe how this might work:

1. Ingress ethernet frame with src-mac A and dst-mac B.
2. Rewrite destination MAC to a LA-MAC address “D” that is a “transport” MAC (kind of like outer label in MPLS) corresponding to some egress switch.
3. Rewrite source MAC to a LA-MAC address “C” that the egress node matches on then re-writes the original “A” and “B” mac addresses back onto the frame.  This is like the inner label in MPLS.
4. Intermediate nodes match on address “D” to get frame to egress switch.
5. Egress switch matches on address “C” and then puts original MAC addresses “A” and “B” back onto frame and transmits out port.

Now that MPLS is supported, I think they should probably just use that. Do not rewrite the MAC addresses. Prepend an inner and outer MPLS label. PH-Pop the outer label on the second-to-last OpenFlow node, then the egress node will match on the remaining (previously inner) label and simply forward out an interface. Why do this? Well its less stuff to do in the network and its less state to keep track of.

Also, it opens up the possibility to smoothly integrate with MPLS. Imagine being able to add an external port to an OpenFlow network via Kompelli pseudowire? Its kind of like OpenFlow with tentacles. OpenTentacleFlow. Integrating with MPLS in this fashion also means that you might be able to extend your OpenFlow network through an intermediate MPLS infrastructure natively and place edge OpenFlow Nodes (or a Juniper MX OVS switch/VRF) on the other side of the MPLS network. I really see OpenFlow and MPLS coming together in a big way in the future…

[edit: If you want to see why myself and others are so excited about OpenFlow/SDN then check out the Network Field Day NEC presentation.. techies can skip to the 25 minute mark.]