Packet Pushers

Where Too Much Technology Would Be Barely Enough

  • HOME
  • Podcasts
    • Heavy Networking
    • Priority Queue
    • Network Break
    • Briefings In Brief
    • Datanauts
    • Full Stack
    • IPv6 Buzz
    • Community
  • News
  • Hosts
  • Subscribe
  • Sponsor
  • Contact

IGNITION MEMBERSMEMBER LOGIN

You are here: Home / Blogs / Things Network Engineers Hate

Things Network Engineers Hate

Ethan Banks October 10, 2017

An industry friend of mine was recently asked these questions.

What are the actions that network engineers hate to be asked to do? What are network engineers asked to do which they would rather not?

He had his answers, but was curious about my take based on the interaction we have with the community here at Packet Pushers Heavy Industries. My take is that we’ve got a couple of significant complaints as network engineers.

  1. Contextless provisioning requests.
  2. Stupid network tricks.

Both of these points are closely related, and I feel get at the root of some of network engineering’s greatest practical challenges.

Contextless Provisioning Requests

A contextless request for provisioning is that request that comes through the system with no explanation of why something is needed. In some orgs, that might be okay. Perhaps you’re a little far down the line, and it’s not your role to second-guess requests that come through. But in many orgs, you’re the person who does practically all of the networking in the company.

What technology is driving a request? What are the business requirements driving the request? Any network engineer who’s been around for a while will instinctively want to know these details.

Let’s take a classic example of a request to provision a series of ports with VLAN assignments. That seems simple, but it belies the potential complexity inherent in any connectivity request. Every system has specific availability and performance requirements that should inform the network engineer how best to fulfill the request. Ideally, the network engineer has been involved from the beginning of the project, and is not expected to work magic as an afterthought.

IT silos should never throw requests over the wall to the networking silo if networking hasn’t been involved in the project. A network engineer should have the opportunity to understand the project and create a network design that would meet specific objectives for performance and system availability.

Stupid Network Tricks

The corollary to contextless provisioning requests is another great networking annoyance–the expectation that the network must fix bad application design. Stupid network tricks.

My personal favorite example of this is stretched layer 2. Yes, there are networking technologies that facilitate the deeply foolish idea of putting the same IP address space in more than one physical data center. Yes, there are movements within networking that might ultimately make separation of endpoint identifier and location a typical network design. But until then, stretched L2 technologies, such as OTV, are band-aids for applications engineered with dependencies on IP addresses.

I’m not faulting OTV as a technology. OTV solves the problems that come with stretched L2 as well as can be expected, but that OTV exists at all points to a massive problem in the practice of IT. The wrong silos are solving the wrong problems.

Another so-called use case for stretched L2 is vMotion between data centers. That’s a great whiteboard idea that’s sometimes intended as a disaster recovery technique. However, the amount of data required to successfully complete a vMotion make a VM move between geographically distributed DCs impractical due to bandwidth and latency constraints. And that’s just a single VM. What happens if you need to drain a DC entirely? Imagine moving a beach from one place to another, and all you’ve got is a plastic toy shovel…

What I’m calling stupid network tricks might seem to a manager like technical tools to put a system into production. However, there’s a tradeoff in that a long-term commitment to a silly design introduces technical debt, infrastructure fragility, and long-term opex. Poor infrastructure designs chosen now to complete a project quickly have a way of being problematic later.

The View From The Hot Aisle

Another challenge network engineers face is that of time. That is, there never seems to be enough of it. There is too much work piled onto too few people, and no amount of reading The Phoenix Project seems to have resolved that problem in organizations who view their technology teams as cost centers.

There are ways that IT teams can become more efficient, but IT engineers are always overloaded. Taking time out to master a new process seems untenable when requests are backing up.

“Uh, boss, I can’t stand up the new closet switch right now. I’m learning Python and researching this vendor’s half-baked API.” Right. Not gonna fly. This, perhaps more than any other reason, is why CLI wizardry persists. And why requests get fulfilled without the right questions being asked. And why stupid network tricks get implemented over and over again. What was done in the past is the path of least resistance in the future.

When an org is unwilling to allot time and invest money into their staff to learn new technology, they are left with operational inertia. Engineers don’t have time to learn anything new because they are overloaded. Orgs don’t create the time required to learn anything new because they are too cheap to invest in their IT staff.

What does operational inertia look like? People doing the same things over and over again, because it’s all they know how to do. They throw requests over the silo wall in an attempt to get it done as quickly as possible. Engineers, right or wrong, provision the request as quickly as they can with the tools they are the most familiar with–probably the CLI. And then everyone moves on to the next project as quickly as possible, because there’s six more after that.

That’s not the way forward for technology-centric companies, but the problem is as much about organizational practice as it is about network engineers. Yes, there are things that network engineers hate, but making the changes required to make those things go away comes from how an org thinks about technology. Driving that thinking takes leadership and a willingness to take on personal risk for the greater good.

Hard, that.

6 Comments

About Ethan Banks

Co-founder of Packet Pushers Interactive. Writer, podcaster, and speaker covering enterprise IT. Deep nerdening for hands-on professionals. Find out more at ethancbanks.com/about.

Comments

  1. S. Phillip Simonds says

    October 11, 2017 at 5:01 pm

    Great blog Ethan! Hit the nail on the head. I appreciated your Phoenix Project reference :D. Thanks for all you guys do!

    Reply
    • Ethan Banks says

      October 15, 2017 at 9:14 pm

      Thanks, man. We appreciate that.

      Reply
  2. Brandon Hunter says

    October 14, 2017 at 2:51 pm

    I sat in a cloud networking vendor presentation earlier this week and heard another so-called stretched L2 use-case of “re-hosting” VMs into a public cloud environment (in direct partnership with AWS mind you). When asked, they confirmed its a bit like LISP, with their magic boxes in path, coordinating packet forwarding, tunneling and/or translations necessary to orchestrate the affair. Kudos to them for pulling it off, but I secretly cringed violently and managed to politely restrain myself from pulling out Thor’s Mjöllnir and smashing this thing into oblivion. Look for it at the next re-invent, I’d be curious on your thoughts there as well ?

    Reply
    • Ethan Banks says

      October 15, 2017 at 9:14 pm

      Ugh. Sounds like it has all the usual issues. Would need a directory service, a way to handle BUM traffic, tromboning problems. “Tunneling and/or translations,” yay. Sigh. Lots of complexity just to port an IP to a remote location, but we know there are apps that seem to need such a thing. So we keep hearing.

      Although…wow. If we’re talking about re-hosting as opposed to live migration, that seems like you could to a proper re-IP of the app and update DNS. No? I must be missing the specifics of the use case.

      Re-invent. Not in 2017. Too much to do between now and end of the year. Maybe 2018? I would like to pick one event next year where I can focus on the content being delivered. That’d be a nice change from what usually happens to us at conferences. 😉

      Reply
  3. Toivo Voll says

    October 15, 2017 at 6:28 pm

    This speaks incredibly well to the exact issues I’m seeing in the industry, and my own frustrations. I had the benefit of being involved in the front-end of projects, with time to do the research, and not only did it bring a lot of value in picking the right solutions, asking the right questions during the marketing meetings, it had a vast improvement in how meaningful my contribution seemed. Instead of fulfilling mystery requests, or implementing stupid tricks, I could use my knowledge and experience to help make smart decisions for the organization at the first step.

    Reply
    • Ethan Banks says

      October 15, 2017 at 9:15 pm

      Completely agree. These have been my experiences as well, both good and bad.

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Email
  • Facebook
  • LinkedIn
  • RSS
  • Twitter
  • YouTube

RSS YouTube

  • What Does Digital Transformation Mean To Me February 12, 2019

RSS The Weekly Show

  • Heavy Networking 430: The Future Of Networking With Guido Appenzeller February 15, 2019

RSS Priority Queue

  • PQ 161: Inside Juniper’s Programmable Silicon (Sponsored) December 13, 2018

RSS Network Break

  • Network Break 221: Cisco Calls Privacy A Human Right; VMware Revamps Recertification February 11, 2019

RSS Briefings In Brief

  • BiB 071: SnapRoute CN-NOS For Whitebox Focuses On Operators February 12, 2019

RSS Datanauts

  • Datanauts 158: Creating, Operating, And Collaborating On Open Source February 13, 2019

RSS Full Stack Journey

  • Full Stack Journey 028: Turning The Mic On Scott Lowe December 18, 2018

RSS IPv6 Buzz

  • IPv6 Buzz 019: IPv6 And Broadband Internet Cable Providers February 7, 2019

RSS The Community Show

  • Day Two Cloud 002: How To Do Cloud Right February 6, 2019

Recent Comments

  • Ethan Banks on BiB 071: SnapRoute CN-NOS For Whitebox Focuses On Operators
  • Glenn Sullivan on BiB 071: SnapRoute CN-NOS For Whitebox Focuses On Operators
  • Ethan Banks on BiB 071: SnapRoute CN-NOS For Whitebox Focuses On Operators
  • Ethan Banks on BiB 071: SnapRoute CN-NOS For Whitebox Focuses On Operators
  • michael marrione on BiB 071: SnapRoute CN-NOS For Whitebox Focuses On Operators
  • Jay on Cisco AnyConnect and IPv6

PacketPushers Podcast

  • Heavy Networking
  • Network Break
  • Priority Queue
  • Briefings In Brief
  • Datanauts
  • Full Stack Journey
  • IPv6 Buzz
  • Community Podcast

PacketPushers Articles

  • All the News & Blogs
  • Only the Latest News
  • Only the Community Blogs
  • Virtual Toolbox

Search

Website Information

  • Frequently Asked Questions
  • Subscribe
  • Sponsorship
  • How To Pitch Us
  • Meet the Hosts
  • Terms & Conditions
  • Privacy Policy

Connect

  • Contact PacketPushers
  • Ask Me Anything
  • Subscribe to Podcasts
  • Sponsorship
  • Facebook
  • LinkedIn
  • RSS
  • Twitter
  • YouTube

© Copyright 2019 Packet Pushers Interactive, LLC · All Rights Reserved