Change control isn’t so bad. With the underlying goal of risk mitigation, good change control can save a network engineer from the dreaded resume generating event we sweat over during cutovers.
Change management frameworks such as ITIL layer an element of bureaucracy over network operations to provide a basis for a quantitative approach to IT management and to ensure quality service delivery. The bane of many IT professionals’ lives and the fodder for some hilarious memes, practices such as ITIL change control are necessary for risk management in networks that are growing in size and complexity.
Large organizations have burdensome change control for a reason.
Change introduces the potential for an outage, and in networking, almost everything we do is impactful. A large network that’s legacy in its design likely has huge failure domains. Transactional networks depend on a completely reliable infrastructure to make money, so even the smallest amount of data loss is out of the question. Healthcare organizations have lives on the line, and law firms live and breathe by email and document scanning. Risk mitigation is important.
The bureaucracy part is what many network engineers and IT managers struggle with. It’s very frustrating to have to write up a change proposal, justification and rollback plan to assign a VLAN to a switchport for simple end-user access. Sometimes each individual part of the change process must be approved by a different person and be presented in a separate meeting. It slows the process down and creates a tremendous amount of paperwork.
However, consider how this type of cumbersome process is necessary for high impact changes. For those changes that make network professionals stay up at night, annoying change control becomes welcomed collaboration. Careful peer review makes sense to us when we swing BGP traffic among peers in a global network. A methodical rollback procedure makes sense to us when we move all routing for a college campus from one set of devices to another.
Much smaller changes can be deceptively impactful, though. In a data center, adding a single switchport to VLAN X may inadvertently cause an inordinate amount of DHCP discovers to flood a network. Advertising a network into EIGRP without remembering to disable auto-summary may break everything. Removing an ACL that doesn’t have an increasing hitcount can take down a site-to-site VPN the engineer isn’t aware of. These types of errors aren’t intentional, of course, and they happen to the very best engineers. Therefore going through the annoying process of submitting a change request, writing up the script and rollback plan and attending one or more CAB meetings is important for a high level of quality control.
Change control reduces risk by adding systematic processes to IT management.
It’s this formal and systematic process of request, approval, planning and peer review that ensures a successful change. The collaborative effort may uncover typos, design flaws, compliance issues, or simply that the rollback plan is missing a step. Though it may feel burdensome, this process mitigates risk.
Network engineers who balk at these types of formal processes are often the first to complain when there is no simple change log recording who broke the ASA that morning. Some change logging can be extremely tedious, but simple change logs can be very effective as part of the change control process.
But in instances when the change control process really does seem like a burdensome waste of time, consider that it may be the specific implementation of the process that’s flawed and not the actual concept of change management. Processes can be improved and streamlined without change control being thrown out altogether. This may be contrary to many network engineers’ opinion of change management in larger enterprises, and admittedly change control can be a real pain. But taking down a network is so much worse.
As network engineers, we’re anxious to see those echo-replies. It’s in our blood. The successful ping response is a special joy for a professional network engineer working at the command line. However, the blinking cursor that stops blinking during a remote cutover is the cause of cold sweats and panic attacks. So slow it down, think it through, and remember that risk mitigation is important. Embrace the red tape.