Matthew Norwood writes about IT myths on The Network Therapy Blog. He makes the excellent point (among several) that experience gained working on a large enterprise network isn’t what makes one engineer better than another. I’ve managed the enterprise network gamut now, having worked on everything from tiny office networks to multi-data center networks spread over thousands of miles. I’ve worked for small companies with dozens of employees and large companies with tens and even hundreds of thousands of employees.
I know some network engineers crave the chance to work on A Really Big Network. Alternatively, they in fact do work on A Really Big Network and gain some sort of personal identity or ego boost from it.
Been there, done that. I won’t tell you that really big networks aren’t fun, rewarding, or interesting. Certainly they can be all of those things. But to think that you have to work on some sprawling monstrosity of a network to truly grasp what networking all about is to not quite see big networks for what they really are: groups of similar, smaller networks glued together into a whole. Sure, so you have 100 or 1,000 or more switches in a data center. But what is a data center network other than a series of racks copied in form and function row upon row? Need more servers? Build another row, throw in a few more access switches, and uplink them to the aggregation or core switches. Got a new remote office to add to the 500 you already manage? Great. Order the same circuit, firewall, router, and switch you’ve ordered for all the others, ship them, walk the on-site person through the turn up process. Big whoop.
A balanced view of really big networks is that while providing no lack of interesting work for scads of engineers, they are also a pain in the butt. Working on a large enterprise network isn’t always a win.
- You must feed all your baby birds. And by feed, I mean monitor health, react to events, maintain operating systems, perform security audits, etc. There are tools, tools, and more tools aimed at helping enterprises manage their gear, including network gear. Most networking companies offer umbrella managers to help you manage all of their own stuff you can’t seem to stop buying. With rare exception, all of these tools are junk. They are hard to use, designed by people who’ve never managed a network, expensive, unreliable, and untrustworthy. And I’d rather pour bleach in my eyes than deal with most of their user interfaces. Managing large networks is longest running fail in the industry.
- Standardization cuts both ways. On really big networks, you need to keep hardware, code, and operating system revisions you support the same as much as possible. You’ll go bonkers if you are troubleshooting a connectivity or performance issue, but can’t predict what the hardware platform, OS rev, or device configuration is. Standardization brings a certain amount of predictability that is wonderful…most of the time. The rest of the time, you hate your networking vendor (and maybe your life) when you discover that your standard switch and OS rev has a catastrophic memory leak, security vulnerability, or other business-impacting problem you have to face. Time to patch! And oh, by the way, don’t take down your really big network while you do it.
- Welcome to the silo. Big enterprises tend to segment network responsibilities by technology across their engineering staffs. So while you might dig security, wireless, route/switch, IPT/VoIP, load-balancing, and WAN optimization, you might or might not get to work on all those technologies. More likely, you’ll be a part of a team that does a smaller subset (or even one) of those things. You need a firewall rule opened up, but you’re not on the security team? Put in a request; you’re just another customer who has to put up with the process. Want some utilization reports from the network management station? Helps to have a buddy in the NOC, because you might not have direct access to the tools.
- An inconvenient window. Now, certainly late night maintenances happen no matter what size the network is you’re working on. But in my experience, the larger the network and therefore the more employees or customers impacted, the greater the likelihood that your maintenance windows will be small, infrequent, and at times that don’t take your family into consideration. You do your best work on no sleep? Yeah…me neither.
- It’s the process, stupid. Really large networks tend to have similarly large groups of people running them, for obvious reasons. Therefore, if you want to get anything done, there’s very probably a formal process you have to follow. Tickets to submit. Forms to fill out. Signatures to obtain. Workflow boxes to check. Inscrutable web pages to navigate. Interoffice envelopes to mail. And then you wait for your request to be fulfilled. If you “know a guy”, then maybe he can grease the skids and move things along. If you don’t, you wait in line for your request to be serviced in accordance with the process. Don’t be in a hurry, either. The bigger the corporation, the less likely they’ll care about actually resolving your issue in a timely way, and the more likely they’ll be a slave to the process…probably a process no one remembers writing, and with no clear logic behind it.
So am I saying there’s no good reasons to work on a big network? Not at all – there’s lots of reasons, not the least of which is that you get to play with all the cool toys. Big networks need big iron, big circuits, big data centers, and big budgets. Big networks usually have big problems to solve. You get to be a part of some big projects that might not be happening on smaller networks. But does the sun rise and set in the data centers of global corporations? Nah. It’s a trade off…it’s all in what you’re looking for.