There’s been a lot of noise about SDN recently, not least due to well-timed announcements from Cisco. Almost every major player seems to have started laying out their position. Some vendors have laid out quite a bit of detail, others have given us pieces, but we don’t yet see the full vision. That’s not necessarily a bad thing that we don’t see the full position from everyone – it’s still very much a moving target.
I’ve been trying to make sense of what it all means for my career, and those around me. I’m reaching the conclusion that the network engineer’s role in 10 years time will look quite different, which is no surprise. But what I am concerned about is that I think that many current network engineers will be unable to make that transition, and worse, they will hold back industry development. They will do this by failing to demand more of their network vendors. Many companies spend time and money on focus groups and market research, trying to give us what we think we want. Brave companies don’t worry about that – instead they give us what we didn’t even realise we needed. How many enterprise equipment vendors would you classify as brave? Will we push them to deliver more, or will we just accept what we’re given?
My Corner of the World
To give you some background, much of my career has been working in New Zealand. I’ve worked in various organizations, large and small. Currently I’m consulting, and working mostly with medium sized businesses, trying to help them with the typical problems they face. I don’t work with the hyper-scale data centres. Yes, you may well work with a service provider with 10 CCIEs on staff – but most people I meet don’t. The network teams are small, often only a handful of people trying their best to cope with the day to day demands of managing a network. They’re not dealing with a few problems on a grand scale, ala Google or Facebook. Instead they’re dealing with a lot of small problems. It’s probably a lot tougher.
Frequently they’re quite stressed out, seemingly running just to stand still, barely keeping up with keeping the lights on. They don’t have large numbers of staff, and have to make do with what they have. So you’d think they’d be looking for anything they could to make their lives easier, and automate the drudgery. Yet when I watch what people do, I see a lot of time at the CLI, managing individual configs by hand. There’s a bit of cut/paste going on, but there’s a lot of manual work. Even simple scripting isn’t done. Instead it’s small changes, repeated hundreds of times. Configurations start out from a template, but drift over time. Yet this just seems to be accepted. Why?
Currently, I’m seeing many medium-sized businesses putting effort into automating and orchestrating their server and application configurations. Historically, this was only done by large enterprises, if at all, but the systems explosion through virtualization, coupled with the availability of free, practical tools like Puppet has made it a “must-have.” Changes are now managed centrally, configuration is easily pushed out across all systems, and auditing is vastly simplified. Those companies that haven’t made that step yet are going through a lot of pain, and are trying to work out how to get automation implemented. (Note that it’s often political, rather than technical reasons for holding up implementation.) But these are server admins making these moves towards centralized control and orchestration. Unix admins have long been masters at scripting mundane tasks, so for them, the step to centrally managing configs is no great leap. Windows admins know the power of Group Policies.
Why Aren’t We All Managing Our Networks Centrally?
Before I go on, I’m sure there will be people saying “But we do have central management/orchestration for our network. We have great provisioning systems, and very little, if any config is done directly on the router/switch.” That’s fantastic, but my guess would be that you’re working with a service provider, or other very large network operator. You’ve also probably only got a limited number of devices/vendors in your network. I’m just not seeing that sort of automation in small-medium sized networks.
So servers are centrally managed, and it lets those admins get their estate under control. Now they can scale up, without needing more headcount. Life is good. Yet many network engineers just carry on doing their day to day work, and never seem to think about that level of control, that mindset, that way of thinking about managing a network. Why is that? Here’s a few reasons I can think of, or have heard:
- The tools don’t exist to do what we want, even though maybe we haven’t looked at options in years.
- We’ve tried the tools before, and got burnt. Normally this is in reference to CiscoWorks.
- Huh? Central orchestration? Why would I want to do that? What’s wrong with doing it the way we’ve always done it? Look how quickly my hands move across the keyboard punching in switchport configs!
I’ll come back to that last point in a minute. @etherealmind, having a rant about vendors and SDN, says that “Vendors haven’t developed tools or solutions that keep the complexity of networking under control,” and that they lack vision, not being able to see valid use cases for SDN. I certainly see his point, although I don’t entirely agree – I am seeing tools becoming available that do things like graphical VLAN management. They’re not perfect, but I think they’re heading in the right direction. Virtual Application Networks is a move towards “Mak[ing] Network Orchestration concurrent with application deployment.” I like IMC, and I’ll have more posts on it in future, along with a full disclaimer (short version: HP gave me free tickets to HP Discover. No other inducements to write nice things though). I think Greg is right in general though – things are changing now, but vendors haven’t really given us much in that space over the last 10 years.
I’m sure anyone that’s used CiscoWorks over the last decade has been incredibly frustrated with it at times. I have heard it’s better now, although I can’t personally verify that. So engineers tried doing things with CiscoWorks in 2003 and it was buggy, so they gave up and went back to their PuTTY terminal. And they never looked back to see what might have changed in the meantime. Well, maybe it’s time we looked again. No, we won’t find all the tools ready now – but maybe we’ll be able to see a future, and we’ll be able to demand solutions from our vendors.
Is It You or Is It Me?
This gets back to the last bullet point – is the problem not that the vendors aren’t providing the vision, but that we, as network engineers, are not demanding more? Are we the problem? For Unix admins and application developers, who have that scripting/programming background, the concept of central orchestration is obvious to them. But look around at your networking peers – how many of them have programming, or at least scripting, in their backgrounds? Many of them came from being cabling engineers. There’s nothing wrong with that, but if you’ve never been exposed to alternative methods of doing things, or alternative ways of thinking, then you may never have cause to re-evaluate the things you do every day. We just take them for granted, and never challenge the status quo. Are vendors able to continue to hold back on real change, because we, the customers, are not jumping up and down about it? We’re just happy to tug the forelock, touch the hem of the cloak, oh look, this shiny new router is 8% faster than the last one. Look, we can grow incrementally, and my skills will still be totally relevant.
If all you’ve ever done is manual CLI configuration, then the thought of writing scripts, or trusting applications to push out complex configs can be scary. How will these engineers make the leap to a brave new world of SDN? Sure, if you’re someone like @cloudtoad, then you’ve got a background in programming. But what about everyone else? How are they going to make the transition? Are we, as network engineers, going to hold ourselves back?
Am I Completely Off-Base?
Sometimes you get in your little bubble, and you assume that because you spend 20+ hours a week studying, and reading all you can about industry developments, that you assume everyone else does the same. But then you get a reality check – e.g. recently I was speaking to someone who is involved in OpenFlow marketing, and they said that at the recent Interop conference, only maybe 5-10% of the attendees were interested in OpenFlow. If the sort of people that attend Interop are not making noise about SDN, who is? Are we, the readers of websites such as this, a real minority? Sure, we talk of the need for change. But does anyone else care?
As a wider group, do network engineers lack the vision, and the broader skillset, to want to change? What will prevent change? Will it be lack of technical capability, lack of solutions from vendors, or lack of funding because businesses don’t see the need for change? Or will it be because too many of us are complacent or scared of change?
Or have I got it all wrong? Am I just talking to the wrong people? Enlighten me, please.