Dropping Down a Gear

As I have written previously, I have started work after many months of long service leave, and evaluating where I am going with my career. I am about a month into a six month contract working at a non-profit with three or four hundred employees, focusing on social services. The organization previously outsourced all of its ICT functions, but for various reasons the outsourced organization was let go, and three IT staff were hired. I was hired on a contract as a network engineer to evaluate their current network, and to design and implement a replacement which will better serve current needs, plus cater for future amalgamation of some related non-profits.

One month in, and I am loving the job. It is low-pressure, low-stress, and perhaps due to the nature of the work they do, or maybe out of appreciation for the better service our little team in providing, the clients are nice people with whom it is a real pleasure to work. In fact, were it an option to continue in the role on an ongoing basis, I would jump at it.

Which leads me to the point of this post. The last project I worked on in my previous job was the design and implementation (although I left before implementation began) of a University Data Center deployment, using Nexus 7010s, Nexus 5020s and Nexus 2K FEXes and the deployment of new Catalyst 6509 core switches. Multiple 10-Gig links over dark fibre, VDC, VSS, VPC and many other new and interesting acronyms. In my new role, the existing network consists of seven main and 20-odd subsidiary sites with ADSL links, using Draytek routers connected via PPTP tunnels. Many sites of up to 40 or 50 users only have ADSL services with 384 kbps upload speeds. The switches across the sites are unmanaged (or have never had any management configured) and a chaotic mix of Netgear, ASUS, D-Link and more; all stuff you can pick up from the home electronics shop. Knowledge of traffic patterns and requirements were totally non-existent. Having led a relatively sheltered life in higher education, where I thought I knew what tight budgets were, I was initially dismayed by what I surveyed before me.

From this...

... to this.

 

However, I am now starting to enjoy the challenge of designing a replacement network. Everything from racks and cables to active equipment and carrier services are in the remit. One of the concerns, of course is the low budget, both capital and operational, so I am looking at LAN LITE Cisco L2 switches and other (relatively) low spec equipment to keep costs down. As there is not going to be a network engineer on staff after I complete my contract, front and center in my mind is producing an easily manageable network. I am talking with carriers to provide better bandwidth options, private network options to eliminate the inter-site VPNs, and for the carrier to manage the network including the CPE routers at each site, meaning that only L2 needs to be managed by support staff in the ICT team on an ongoing basis. And I need to document the current and future network for non-networking people.

I know that this experience may be run-of-the-mill for many of you reading, especially those of you who do contracting or consulting, but for me it has been an exercise not so much in changing gear, as mentioned in the title of the post, but rather an adjustment in mindset. I am not using the term “changing down a gear” to imply that small shops or anything less than a maxed-out Nexus 7K is somehow less worthy. Having been ensconced in a larger organization, I have been designing and implementing networks knowing that I or someone with my skill set will be managing the finished product (as much as any networking project is ever truly finished). I have been free to look at all of the bells and whistles and complex configurations available at the CLI. To build in complexity that can be obtuse when looked at by the outsider; using some cool new feature not because it is strictly needed, but because it is there and it might be cool or useful. The kind of thing that the uncharitable may describe as the attempt to my justify ongoing existence by building an inscrutable and opaque solution.

Rather, the gear change is that I find that I have to think much more carefully about the choices I make, rather than rushing headlong into the shiny end of the Cisco product catalogue. Which switchport security options? How about error disable recovery? Are things like wired 802.1x, dynamic ARP inspection or DHCP snooping going to be more trouble than they are worth if something breaks six months down the track? What kind of inexpensive or free management and reporting tools can I deploy that will be helpful to non-networking professionals and which are also easy to maintain in and of themselves? I suppose I have really learned to appreciate that I have to figure out what are the true needs of the organization as distinct from my own indulgent technological onanism. With bigger margins in budgets and advanced technical solutions, sometimes “needs analysis” can be an empty phrase, subordinated to engineering self-indulgence.

I have another month before my proposed budget is looked at by the board, then three months to procure and implement. I really am hoping to do well and impress, and to learn skills which will help me in a future consulting or contracting role. In the meantime, funny as it sounds, I have come up with some positive lessons from the current implementation which I originally derided internally as “primitive” when I came on board. The Draytek routers, of which I have previously had next to no knowledge have turned out to be surprisingly good little boxes. I find the interface horrible and frustrating, and the lack of diagnostics was less than helpful, but I am quite impressed with a unit that can do what it does for around AU$300. Having to re-learn NAT after coming from a place with a whole public /16 was also a surprisingly positive experience.

I started the job with some trepidation for reasons I outlined in my previous post, and there was some initial regret on the downsizing of the lumps of metal and plastic I was used to; and although the song goes “if you can’t be with the one you love, love the one you’re with”, I am afraid I will never love unmanaged hubs and switches. I suppose I have had the opposite experience moving out of higher education than Mrs. Y had (listen to the start of Episode 95). I certainly was not overwhelmed by complexity – the compete reverse. Changing down a gear has had its own challenges; ones I have eagerly embraced so far. And on a side note, my work-life balance is in fact infinitely better so far.

As a newbie in the world of experiencing new sites and variable budgets, what would your advice be in changing down a gear, or for that matter, up a gear?

Matthew Mengel
Matthew was a Senior Network Engineer for a regional educational institution in Australia for over 15 years, working with Cisco equipment across many different product areas. However, in April 2011 he resigned, took seven months of long service leave to de-stress and re-boot before becoming a network engineer for a medium sized non-profit organisation. At the end of 2013, he left full-time networking behind after winning a scholarship to study for a PhD in astrophysics. He is on twitter infrequently as @mengelm.
Matthew Mengel

Latest posts by Matthew Mengel (see all)

  • Ccjaph

    With a small budget Linux can fill many rolls.

  • David James

    Try Quest’s Foglight NMS.  It’s free for 100 devices and unlimited interfaces.  Great for a small shop on a small budget.

  • Automatom

    I would stay with managed switches, but look at other vendors such as HP Procurve at a lower cost than Cisco while still maintaining full manageability. 

    Provider managed networks are usually significantly more expensive than IPSec or PPTP VPN over DSL links.

    I would possibly look at Cisco DSL routers to replace the Draytek ones as you can use tools such as rancid to manage configuration archives.

    Take a look at racktables for asset management of equipment in racks and switch port usage.

  • http://twitter.com/bobmccouch Bob McCouch

    A number of my smaller customers are closer to the environment you’re in now than a major enterprise network. I find one thing that needs constant consideration is whether a specific technology/product/feature provides enough benefit to be worth the effort of deployment and support and the risk of it breaking. Some features that are “must haves” in larger environments like storm controls, port-security, maybe DHCP snooping, I find are rarely needed in smaller networks and instead offer one more cog in the machine that may unexpectedly break. 

    That said, a key tenant for me is the idea of bringing enterprise-class capabilities to smaller businesses/networks even if they don’t use them all. With options like the Cisco 800-series routers, ASA5505 firewall, and 2960S, or even the SG300-series small business switches, almost every single business out there can afford enterprise-class gear to provide them many of the same technological advantages that larger businesses enjoy. In my opinion with product classes like that there’s really no room for unmanaged switches and no-name or consumer-grade routers in any business network, large or small. I cringe when I come across unmanaged switches in a business network with no diagnostic capability, no spanning-tree, no ability to be polled or monitored…. We *can* do better for these networks, and the customers/sites *can* afford it. If they truly can’t afford a basic 24- or 48-port low-end managed switch, they’ll be out of business before it would show up from the vendor anyway.

    Can a flower shop get by with a department-store broadband firewall/router/switch/WAP? Probably. But for a modest investment they can be using advanced remote-access VPN to enable the owner to be more productive, redundant Internet connections and secure, user-authenticated wireless. Alongside those features you also gain the ability to remotely manage, maintain, monitor, and troubleshoot that network. Such small investments can pay big dividends even to smaller businesses.

    The same goes for design… 6, 7, or 15 unmanaged switches daisy-chained in one terrible path is something no one deserves. The simplest network can benefit from having something acting as a topological core, having redundant links for bandwidth aggregation and resiliency, etc. Again, I try to drive these principles with my customers. You’re never too small for a decent network architecture.

    I think you’re on the right track. Consider what is really *needed*, but also consider what simple enterprise capabilities we may often take for granted that can, in fact, provide a perceptible technical advantage to those smaller companies or sites.

7ads6x98y