Things Network Engineers Hate

An industry friend of mine was recently asked these questions.

What are the actions that network engineers hate to be asked to do? What are network engineers asked to do which they would rather not?

He had his answers, but was curious about my take based on the interaction we have with the community here at Packet Pushers Heavy Industries. My take is that we’ve got a couple of significant complaints as network engineers.

1. Contextless provisioning requests.
2. Stupid network tricks.

Both of these points are closely related, and I feel get at the root of some of network engineering’s greatest practical challenges.

Contextless Provisioning Requests

A contextless request for provisioning is that request that comes through the system with no explanation of why something is needed. In some orgs, that might be okay. Perhaps you’re a little far down the line, and it’s not your role to second-guess requests that come through. But in many orgs, you’re the person who does practically all of the networking in the company.

What technology is driving a request? What are the business requirements driving the request? Any network engineer who’s been around for a while will instinctively want to know these details.

Let’s take a classic example of a request to provision a series of ports with VLAN assignments. That seems simple, but it belies the potential complexity inherent in any connectivity request. Every system has specific availability and performance requirements that should inform the network engineer how best to fulfill the request. Ideally, the network engineer has been involved from the beginning of the project, and is not expected to work magic as an afterthought.

IT silos should never throw requests over the wall to the networking silo if networking hasn’t been involved in the project. A network engineer should have the opportunity to understand the project and create a network design that would meet specific objectives for performance and system availability.

Stupid Network Tricks

The corollary to contextless provisioning requests is another great networking annoyance–the expectation that the network must fix bad application design. Stupid network tricks.

My personal favorite example of this is stretched layer 2. Yes, there are networking technologies that facilitate the deeply foolish idea of putting the same IP address space in more than one physical data center. Yes, there are movements within networking that might ultimately make separation of endpoint identifier and location a typical network design. But until then, stretched L2 technologies, such as OTV, are band-aids for applications engineered with dependencies on IP addresses.

I’m not faulting OTV as a technology. OTV solves the problems that come with stretched L2 as well as can be expected, but that OTV exists at all points to a massive problem in the practice of IT. The wrong silos are solving the wrong problems.

Another so-called use case for stretched L2 is vMotion between data centers. That’s a great whiteboard idea that’s sometimes intended as a disaster recovery technique. However, the amount of data required to successfully complete a vMotion make a VM move between geographically distributed DCs impractical due to bandwidth and latency constraints. And that’s just a single VM. What happens if you need to drain a DC entirely? Imagine moving a beach from one place to another, and all you’ve got is a plastic toy shovel…

What I’m calling stupid network tricks might seem to a manager like technical tools to put a system into production. However, there’s a tradeoff in that a long-term commitment to a silly design introduces technical debt, infrastructure fragility, and long-term opex. Poor infrastructure designs chosen now to complete a project quickly have a way of being problematic later.

The View From The Hot Aisle

Another challenge network engineers face is that of time. That is, there never seems to be enough of it. There is too much work piled onto too few people, and no amount of reading The Phoenix Project seems to have resolved that problem in organizations who view their technology teams as cost centers.

There are ways that IT teams can become more efficient, but IT engineers are always overloaded. Taking time out to master a new process seems untenable when requests are backing up.

“Uh, boss, I can’t stand up the new closet switch right now. I’m learning Python and researching this vendor’s half-baked API.” Right. Not gonna fly. This, perhaps more than any other reason, is why CLI wizardry persists. And why requests get fulfilled without the right questions being asked. And why stupid network tricks get implemented over and over again. What was done in the past is the path of least resistance in the future.

When an org is unwilling to allot time and invest money into their staff to learn new technology, they are left with operational inertia. Engineers don’t have time to learn anything new because they are overloaded. Orgs don’t create the time required to learn anything new because they are too cheap to invest in their IT staff.

What does operational inertia look like? People doing the same things over and over again, because it’s all they know how to do. They throw requests over the silo wall in an attempt to get it done as quickly as possible. Engineers, right or wrong, provision the request as quickly as they can with the tools they are the most familiar with–probably the CLI. And then everyone moves on to the next project as quickly as possible, because there’s six more after that.

That’s not the way forward for technology-centric companies, but the problem is as much about organizational practice as it is about network engineers. Yes, there are things that network engineers hate, but making the changes required to make those things go away comes from how an org thinks about technology. Driving that thinking takes leadership and a willingness to take on personal risk for the greater good.

Hard, that.