Will Engineers Hold Networking Back?

23 June 2012 by 22 Comments

There’s been a lot of noise about SDN recently, not least due to well-timed announcements from Cisco. Almost every major player seems to have started laying out their position. Some vendors have laid out quite a bit of detail, others have given us pieces, but we don’t yet see the full vision. That’s not necessarily a bad thing that we don’t see the full position from everyone – it’s still very much a moving target.

I’ve been trying to make sense of what it all means for my career, and those around me. I’m reaching the conclusion that the network engineer’s role in 10 years time will look quite different, which is no surprise. But what I am concerned about is that I think that many current network engineers will be unable to make that transition, and worse, they will hold back industry development. They will do this by failing to demand more of their network vendors. Many companies spend time and money on focus groups and market research, trying to give us what we think we want. Brave companies don’t worry about that – instead they give us what we didn’t even realise we needed. How many enterprise equipment vendors would you classify as brave? Will we push them to deliver more, or will we just accept what we’re given?

My Corner of the World

To give you some background, much of my career has been working in New Zealand. I’ve worked in various organizations, large and small. Currently I’m consulting, and working mostly with medium sized businesses, trying to help them with the typical problems they face. I don’t work with the hyper-scale data centres. Yes, you may well work with a service provider with 10 CCIEs on staff – but most people I meet don’t. The network teams are small, often only a handful of people trying their best to cope with the day to day demands of managing a network. They’re not dealing with a few problems on a grand scale, ala Google or Facebook. Instead they’re dealing with a lot of small problems. It’s probably a lot tougher.

Frequently they’re quite stressed out, seemingly running just to stand still, barely keeping up with keeping the lights on. They don’t have large numbers of staff, and have to make do with what they have. So you’d think they’d be looking for anything they could to make their lives easier, and automate the drudgery. Yet when I watch what people do, I see a lot of time at the CLI, managing individual configs by hand. There’s a bit of cut/paste going on, but there’s a lot of manual work. Even simple scripting isn’t done. Instead it’s small changes, repeated hundreds of times. Configurations start out from a template, but drift over time. Yet this just seems to be accepted. Why?

Server Automation/Orchestration

Currently, I’m seeing many medium-sized businesses putting effort into automating and orchestrating their server and application configurations. Historically, this was only done by large enterprises, if at all, but the systems explosion through virtualization, coupled with the availability of free, practical tools like Puppet has made it a “must-have.” Changes are now managed centrally, configuration is easily pushed out across all systems, and auditing is vastly simplified. Those companies that haven’t made that step yet are going through a lot of pain, and are trying to work out how to get automation implemented. (Note that it’s often political, rather than technical reasons for holding up implementation.) But these are server admins making these moves towards centralized control and orchestration. Unix admins have long been masters at scripting mundane tasks, so for them, the step to centrally managing configs is no great leap. Windows admins know the power of Group Policies.

Why Aren’t We All Managing Our Networks Centrally?

Before I go on, I’m sure there will be people saying “But we do have central management/orchestration for our network. We have great provisioning systems, and very little, if any config is done directly on the router/switch.” That’s fantastic, but my guess would be that you’re working with a service provider, or other very large network operator. You’ve also probably only got a limited number of devices/vendors in your network. I’m just not seeing that sort of automation in small-medium sized networks.

So servers are centrally managed, and it lets those admins get their estate under control. Now they can scale up, without needing more headcount. Life is good. Yet many network engineers just carry on doing their day to day work, and never seem to think about that level of control, that mindset, that way of thinking about managing a network. Why is that? Here’s a few reasons I can think of, or have heard:

  • The tools don’t exist to do what we want, even though maybe we haven’t looked at options in years.
  • We’ve tried the tools before, and got burnt. Normally this is in reference to CiscoWorks.
  • Huh? Central orchestration? Why would I want to do that? What’s wrong with doing it the way we’ve always done it? Look how quickly my hands move across the keyboard punching in switchport configs!

I’ll come back to that last point in a minute. @etherealmind, having a rant about vendors and SDN, says that “Vendors haven’t developed tools or solutions that keep the complexity of networking under control,” and that they lack vision, not being able to see valid use cases for SDN. I certainly see his point, although I don’t entirely agree – I am seeing tools becoming available that do things like graphical VLAN management. They’re not perfect, but I think they’re heading in the right direction. Virtual Application Networks is a move towards “Mak[ing] Network Orchestration concurrent with application deployment.” I like IMC, and I’ll have more posts on it in future, along with a full disclaimer (short version: HP gave me free tickets to HP Discover. No other inducements to write nice things though). I think Greg is right in general though – things are changing now, but vendors haven’t really given us much in that space over the last 10 years.

I’m sure anyone that’s used CiscoWorks over the last decade has been incredibly frustrated with it at times. I have heard it’s better now, although I can’t personally verify that. So engineers tried doing things with CiscoWorks in 2003 and it was buggy, so they gave up and went back to their PuTTY terminal. And they never looked back to see what might have changed in the meantime. Well, maybe it’s time we looked again. No, we won’t find all the tools ready now – but maybe we’ll be able to see a future, and we’ll be able to demand solutions from our vendors.

Is It You or Is It Me?

This gets back to the last bullet point – is the problem not that the vendors aren’t providing the vision, but that we, as network engineers, are not demanding more? Are we the problem? For Unix admins and application developers, who have that scripting/programming background, the concept of central orchestration is obvious to them. But look around at your networking peers – how many of them have programming, or at least scripting, in their backgrounds? Many of them came from being cabling engineers. There’s nothing wrong with that, but if you’ve never been exposed to alternative methods of doing things, or alternative ways of thinking, then you may never have cause to re-evaluate the things you do every day. We just take them for granted, and never challenge the status quo. Are vendors able to continue to hold back on real change, because we, the customers, are not jumping up and down about it? We’re just happy to tug the forelock, touch the hem of the cloak, oh look, this shiny new router is 8% faster than the last one. Look, we can grow incrementally, and my skills will still be totally relevant.

If all you’ve ever done is manual CLI configuration, then the thought of writing scripts, or trusting applications to push out complex configs can be scary. How will these engineers make the leap to a brave new world of SDN? Sure, if you’re someone like @cloudtoad, then you’ve got a background in programming. But what about everyone else? How are they going to make the transition? Are we, as network engineers, going to hold ourselves back?

Am I Completely Off-Base?

Sometimes you get in your little bubble, and you assume that because you spend 20+ hours a week studying, and reading all you can about industry developments, that you assume everyone else does the same. But then you get a reality check – e.g. recently I was speaking to someone who is involved in OpenFlow marketing, and they said that at the recent Interop conference, only maybe 5-10% of the attendees were interested in OpenFlow. If the sort of people that attend Interop are not making noise about SDN, who is? Are we, the readers of websites such as this, a real minority? Sure, we talk of the need for change. But does anyone else care?

As a wider group, do network engineers lack the vision, and the broader skillset, to want to change? What will prevent change? Will it be lack of technical capability, lack of solutions from vendors, or lack of funding because businesses don’t see the need for change? Or will it be because too many of us are complacent or scared of change?

Or have I got it all wrong? Am I just talking to the wrong people? Enlighten me, please.

Filed Under: Blogs, SDN, Work Life
About Lindsay Hill

Lindsay (@northlandboy) is a network management consultant, with experience across networks, servers, applications and security. He is CCIE #36708, RHCE, CISSP and HP MASE. More of his own content is at lkhill.com.

  • Bernie

    I see this column as being spot on. I’m far from a programmer, but I’ve used Perl scripts on more than one occasion for network engineering. At one job, I modified a custom scripting tool to perform network auditing and briefed my team of 6 coworkers on how it worked, then created a HowTo doc and made it all freely available. No one used it, not even after we got audited and they saw that I finished my work in hours versus days. I guess that’s one reason I’ve gone on to bigger, better jobs, though!

    • http://twitter.com/northlandboy Lindsay Hill

      Yeah, this is the sort of thing I’m getting at – even when someone’s done the legwork, people still seem to resist it. Perhaps your comment about moving on is getting at the heart of it – it’s maybe an institutional thing, with some organisations being more resistant, while others embrace those sorts of ideas.

  • http://www.facebook.com/paull Paul A. Leroux

    HI Lindsay. Excellent article and I can couldn’t agree with you more. But if you are looking for a company that is going down the route of centralized network management “properly” you need to look at Avaya*. As I am sure you are aware, Avaya is a champion of SPB. And the Avaya centralized management tools to configure and manage the virtualized environments is astounding. It is literally, point and click, drag and drop to manipulate VM’s and SPB routes.

    *No, I don’t work for Avaya.

    • http://twitter.com/northlandboy Lindsay Hill

      Thanks. Sounds like I need to check out Avaya more. Will their tools work (reasonably) well in a multi-vendor environment, or are they more aimed at all-Avaya shops?

      • http://www.facebook.com/paull Paul A. Leroux

        thanks for the reply. the avaya network management tools are vendor agnostic. But like all nms solutions that come from vendors, they do work best with their gear.

  • http://packetpushers.net/author/ecbanks Ethan Banks

    I’ve been thinking of dusting off my programming skills, reviewing C, and maybe learning Python. I built some sites with LAMP a few years back, so there’s some PHP in my recent past. And of course, I’ve done some perl, like everyone else. The larger point being that I’ve had many ideas about building a simple web interface to be used for simple provisioning tasks, but customized for the environment I work in.

    To your point about tools, they’ve been so poor for so long, that I have indeed shied away from them. OpenView? Too cumbersome. And expensive. And ugly. Ionix? A beast, and untrustworthy back in some of the earlier days of its life as Voyence. SolarWinds? I use it today and mostly like it, but it’s barely a configuration tool at this point. Much stronger as a monitoring tool, at least as long as the database stays intact. CiscoWorks? A Prime eval is in my future, but past CW experience was awful. Terrible GUI riding on CPU- & memory-hungry daemons, no matter how small your installation was. Plus not all CW components worked well with other ones.

    Do I think tools are worth looking at again? Sure. But I also think that no matter how good an out-of-the-box tool is, the real magic is going to be in customized solutions that help a business get IT tasks done quickly. My money is on APIs helping to facilitate that. And yes, I think network engineers will, in large part, be the ones that are writing code.

    There’s plenty of community examples for us out there on how to get such a thing done, and share our code. I’m curious to see how Cisco does with ONE and onePK. If they make the API accessible to those of us in the trenches (and not just pure developers), that will drive adoption of this programmability notion. But whatever Cisco does, I think there’s room for a modular open-source provisioning framework to rise up and see use in the enterprise, modules, scripts, and other bolt-ons to be written by network operators. And I’m all for it. I’m not married to the CLI, and the longer I work in networking, the more tedious I find the CLI-way of doing things.

    • http://twitter.com/northlandboy Lindsay Hill

      Yes, I’m very interested in how ONE works out too. The ultimate success of it will come if it gets picked up by the broader community, and we share tools/plugins/frameworks/etc. I think if it ends up being used by a few isolated, expensive, proprietary systems, then it risks remaining relatively niche.

      PS C? Hard man. My guess would be that it’s maybe easier/better to go with a higher-level language like Python.

  • joe

    Service providers have more engineers (yes sometimes as many as 10 CCIE and sometimes many many more) but that is because their networks are larger. Don’t assume that because Service Providers have more engineers that you have it harder in the enterprise/IT environment. Service provider engineers have very demanding jobs that require a lot of late night work in addition to the daytime job (we’re talking on a weekly basis, not some rare dc migration late night work). The ROI for automation tools is very clear because they have so much more gear. The reason SP’s have implemented more automation is because of the clear ROI, if that can be proven in the enterprise you’ll see it moe and more there as well. I don’t think engineers will fight the automation, engineers are typically ones that want new tools, they want to improve the network, it is the suits that need to see the ROI in it all to pay for those tools.

    • http://twitter.com/northlandboy Lindsay Hill

      I’ve done plenty of my share of late nights too. Mainly when I worked in Telcos. Minimum 1 night/week then. There’s a good reason I don’t do night work or on call anymore! It is certainly easier at Enterprises that have a defined business day. Doing work at 5pm is so much easier than 3am. That said, I’ve got customers that are 24×7 Enterprises, where they need to replace failed switches within 30 minutes, any time of day/night. They have no real defined service window, all they can do is find a time when fewer people will be impacted. All depends on the nature of the business.

      That’s a good point about needing a certain level of gear to make the ROI on automation pay back. Regarding your comment that “…engineers are typically ones that want new tools, they want to improve the network…”, I would like to think that it was true of all engineers. Unfortunately I see a lot of complacency. I see engineers that do their best to constantly improve the network and themselves, and I see others that just drift along. Perhaps Bernie’s comment throws more light on it too – that it can be an organisational culture thing.

  • http://twitter.com/cjinfantino CJ Infantino

    Great post. I feel the exact same way, as matter of fact I just wrote a post about pretty much the same thing right before I read this.

    I believe the landscape of the industry will change in the very near future. I started to delve into running Openflow controllers and switches to see what it is all about, to get hands on time with it. I can honestly say, I feel like it will change everything. To be able to program a switch to do exactly what we want has such potential.

    I don’t believe this will be for large shops either. I believe Openflow and SDN is for every business large and small. There are endless use cases for Openflow and SDN, things we didn’t even think of yet.

    I for one am excited, and I have begun dusting off my programming skills.

    Excellent post!

  • Phil Ashman

    It always amazes me to read about the same issues we were facing 10 years ago are still a thorn in everyone’s side. I worked as the network architect at a mid sized university before moving into the teaching side of things. Back then we were dealing with OpenView, Cisco Works and Optivity, yet I remember quite clearly banging my head against the desk and looking to the ceiling in frustration as I tried to bend these apps to my needs. I ended up using CLI in combination with the individual proprietary GUI tools for whatever vendor’s equipment I was configuring.

    It appears we haven’t come much further since then, but I think we all feel and see a major shift to the industry not felt since the late 90′s when VLANS came about.Most network engineers I talk to are pushing a burnout pace balancing between the day to day firefighting, proactive management of their infrastructure, and professional development. As you say, most are barely able to keep the ship afloat let alone challenge the vendors to give them the tools they need. Now, with a lot of our core networking infrastructure integrating with the virtualized world and technology to centralize and manipulate layer 2 & 3 processing in different ways, network engineers are simply going to be forced to make that next step whether they like it or not.

    I’m a little more optimistic that most network engineers tend to rise to the challenge when that next hump presents itself. We often drive forward regardless of the political or bureaucratic constraints even to our own detriment. It’s always an exciting time in our industry, but I think we are entering that next major architectural transition toward which all will slowly evolve; some faster than others.

  • http://About.Me/RonnyLam Ronny Lam

    Hi Lindsay, I couldn’t agree with you, and the other commenters, more. Programming networkdevices, as I call it, is still stuck in the eighties. One of the coreproblems IMO is that it is in the interest of both the vendors and the engineers to keep it that way. Vendors don’t want to be commoditized and for engineers it’s a part of their status or jobsecurity so you will. “What’s wrong with doing it the way we’ve always done it? Look how quickly my hands move across the keyboard punching in switchport configs!”

    I just joined a small start-up building vendor-agnostic automation tools, just to solve this problem. I can tell you that in selling, or implementing, the product our challenge is not with C-level or senior-management, not even with architects. Our challenge is with the people that are going to work differently. Not touch the CLI anymore, not needing to change the configurations of multiple devices in order to configure a VPN for example, but instead do it in one click.

    It is as you say a combination of not wanting to leave the CLI and not trusting that an automation tool can produce the right config. For troubleshooting of course you still need that CLI, no doubt, but only a a matter of last resort.

    Once we overcome those challenges we see that the more expensive, skilled, engineers and architects are working on models, templates and scripts in the automation tool. They don’t have to do repetitive work anymore, just automate that. Boy, have I seen, and done, in my career too expensive people provisioning vlan’s and switchports not trusting cheaper personnel to do it. When they don’t have to do that work anymore they have more time for strategic work, and have to leave that of course for advanced CLI-troubleshooting once in a while. Although that also gets less because of the implemented automation. In the end they say that their work is is more fun and their skills are still very much needed.

    • http://twitter.com/northlandboy Lindsay Hill

      You’re right – once people see that they don’t have to do the drudgery, their job is far more interesting. I also know what you mean about not trusting cheaper personnel to do a job – I’ve been guilty of this myself. But with the right tools, hopefully we can stop the simple config errors that plague us.

  • WiseWun

    Great read. From my perspective, I think the demand just isn’t there, network engineers are far too comfortable.

  • Cristian

    I am a network engineer, and did a bit of programming/scripting to make my life easier. The scripts are for relatively simple but otherwise time consuming tasks, like recording the interfaces/networks defined in each network device and smarter syslog grepping.
    The problem with the tools today is that they are expensive and complex. They are expensive to purchase initially and to maintain every day. It takes a small army people to deploy, configure, and maintain HP Openview or EMC Smarts; some databases are always out of synch in our CiscoWorks deployment. It’s funny that the tool we trust the most is WhatsUp, a simple application running in Windows that does one thing – pings devices and sends out messages via pager or email when a device stops responding to pings. Mr Vendor, give me something that “just works” and is reasonably priced (ideally on a multi-vendor deployment) and you have my attention.
    The unfortunate effect of this lack of tools is a lot of manual work, and a reactive approach to network management.
    As a programmer, the biggest challenge is how difficult is to push or get information from network devices. Expect-like script that emulate CLI commands are very limited in what they can do, and difficult to use. The new SDN-like push from vendors will make a difference. The APIs will allow the management servers to interact with the network devices in a programmatic way. There a need for this kind of tools, the vendors will move in to fill this void (and sell more product).

    • http://twitter.com/northlandboy Lindsay Hill

      I have to be careful about what I say about OpenView (nb that name has been deprecated since 2007, it’s NNMi now) – part of my income is from maintaining HP Software! But you’re right, it does require a fair amount of maintenance. In my experience, network management software tends to rot quickly if unmaintained. Part of that is simply because networks change over time, and you need to ensure that the management tools are aware of those changes. Vendors make a big deal out of auto-discovery, etc., but ultimately the tools are not mind readers.

      I think you’re right about the APIs changing things – script-based approaches are just too flaky, and too prone to error. We’ve tried using the CLI as a form of API, and it just doesn’t work – it needs to be done properly.

  • http://twitter.com/cloudtoad Derick Winkworth

    I think network engineers are generally pretty smart people. I can easily see them picking up programming skills. I don’t imagine they are going to write fully-fledged totally custom applications that control everything. I do suspect they will write scripts or integrate open source tools in different ways. Certainly if a market place develops, they will deploy software to manage aspects of their network.

    Programming really isn’t that hard. I disagree that C is hard. Have you looked at Java code lately? Lets all stick needles in our eyeballs. Thats Java code. C, at this point, is much simpler. Python and Ruby are probably two good ones. Sadly, there are so many good network-related modules for Perl… Cisco should probably support Perl with ONE. I can think of at least one competitor of theirs that has C, Java, and Perl libraries for different platforms.

    Back to your question though, some people will adapt and some won’t. I don’t think anyone can hold back the tide..

  • Massimo

    Nice article Lindsay!

  • Craig D

    great article, and some positive comments too , means you’ve touch on a very valid point. This article could easily be widened to encompass other areas of the IT industry (servers/virtualization/storage etc) . As the old saying goes “If you do it more than once, script it”.

  • ed

    The issue with SDN solutions is that it requires a “perfect” network structured environment to start with. It requires all your network equipment being on the same page at the same time, something I’ve never witnessed in my career. All these SDN solutions are fine in labs and on white paper, however in the real world there a real world snags that always find ways to trip up predetermined scripts or config pushes.
    And assuming you do get all your 200 3560′s and dozen 4500′s and the few 6500′s and the Nexus on the level you then face the challenges of the learning curve of the SDN. In the past CicsoWrx required a PHD to use it.

    IMO there are some areas of the network environment that can use an SDN as a tool, however as a whole the network is too dynamic and too cryptic to be controlled by a single GUI. Network changes can be quite granular, and most of these granular changes are not system wide changes.
    Comparing the a network to a server farm is apples and oranges. Especially when servers are going VM. Networks are branching off into the dark recesses of the business environment all the time. It’s a living organism with dynamic and sometimes can be “touchy”.
    There’s a reason CLI is the most used. It’s because it’s the most trusted. There is nothing worse than a SDN execution gone wrong. It takes many network engineers to undo a misguided SDN error.

    We also have the cost, and upper management doesn’t want to pay for cost for something the network engineers don’t have it’s 100% backing.
    I’m not against SDN, but I do have a history with it. And like the old saying, fool me once.

    • http://twitter.com/northlandboy Lindsay Hill

      “Comparing the a network to a server farm is apples and oranges”
      Why? In a Google-type business, they are running thousands of near-identical systems, but a typical enterprise is running a much wider variety of applications. It doesn’t mean that they don’t benefit from automation of their server configurations. It doesn’t mean automating every single configuration on every server – but there’s many commonalities, and drudgery tasks – e.g. patching, /etc/resolv.conf config, user management, that are better automated.

      It doesn’t mean you should centralise/automate all forwarding table decisions ala OpenFlow – I’m not yet certain that’s the right way to go – but we do need to better automate our standard configuration entries. Things like firmware should be standardised too. You’re right that our networks need to all be on the same page, and that right now, we just don’t have that, with all kinds of odd configurations in those dark corners of the network.

      But whose fault is that? A lot of places I see have a lot of variation in their configurations for no real reason, other than A) the engineers don’t have time to fix it, or B) the engineers don’t know and don’t care. If we don’t have the time to fix those things, we need to work out how we’re going to try and do things in a better way. If the engineers don’t know/care, well, that’s obviously a different story!

      For the near-medium future, I’d like to see better use of tools for doing away with drudgery. Doesn’t replace CLI for troubleshooting, but it does replace CLI for typical tasks. Consider the post that Ethan wrote a while back on configuring a standard host port (http://packetpushers.net/good-habits-for-basic-ethernet-switchport-provisioning-in-a-cisco-ios-environment/). That’s a lot of steps, and you have to look at that, and wonder “Is that a good use of my time?”

      Completely understand your comment about “history” though. But sometimes we need to re-evaluate things we thought we knew. I first used a Gnome Desktop around v1.0, and it was rubbish. KDE was far superior at the time, and Gnome should not have been branded 1.0. Took me years to even consider trying it again. But when I did, I was surprised at how things had improved. Admittedly, they’ve taken a turn in a different direction now, but that’s getting way OT.

      Lindsay

      PS Good to hear you’re in a place that only buys tools that get 100% engineer backing. I see plenty of places trying to use tools that have 100% mgmt backing, and engineers only find out after the P.O. has been signed…

  • Greg

    I agree with the spirit of this post. I am frustrated with the persistence of device-by-device CLI management, when GUIs have been around for 25+ years; we should’ve developed a standard GUI for more holistic network management by now. Instead, we configure/script device-by-device using a crude/arcane/arbitrary CLIs. Add a different platform to your network (even same vendor, e.g. IOS, IOS-XR, etc), and the education and development starts over, because none of the commands/outputs to accomplish the same ends are the same. Vendor-proprietary user interfaces for network management will stifle progress, and we need to stop looking to network vendors for this solution. If we want network management UI development to progress at a rate independent of how many UI software engineers a vendor employs, then network equipment needs to be reduced to an open instruction set and standard management interface. Cisco ONE is not the right solution, because it’s Cisco only; I don’t care about the logo on a device, I only care whether it satisfies my requirements. I shouldn’t be stuck with one vendor’s equipment because the vendor has forced me to build up a bunch of management tools around that vendor’s proprietary interface(s). I don’t know if OpenFlow will be the final solution, but IMHO it’s definitely the right direction for the industry.