Change isn’t something to be taken lightly in the world of networking – every single change has the potential for great disaster as downstream effects can snowball into something that you didn’t expect would happen. However, change is an unavoidable reality that we as network engineers must face, and the challenge becomes how to properly implement a change while mitigating as much risk as possible and ensuring that downtime does not occur. Following some simple (yet often overlooked) suggestions can help you become a better network engineer and will help win you points with management.
Compartmentalize, compartmentalize, compartmentalize!
One of the biggest problems that occur during network change is the tendency of everyone (managers and technical engineers alike) to cram as many changes into a maintenance window as possible. Project managers will absolutely try to speed up implementation by scheduling multiple changes into one window, and technical engineers with no spine will let them do it. As a network engineer, you’re the expert, and if something goes wrong because of a rushed change, you’re the one that has to fix it. Remember this one simple truth: you can’t rush IT.
I often refer to the 12 Networking Truths, which in my opinion should be taught at technical schools and university programs to all aspiring network engineers. Refer to rule 2:
12 No matter how hard you push and no matter what the priority,you can't increase the speed of light.
Project managers that rush through steps of a project are usually just doing their job – they have a pretty daunting task of balancing project scope with time and money, and it’s rarely an easy thing to do. As a technical expert, it’s your job to speak up during project inception and set realistic timelines from the get-go so that things don’t have to get rushed. Break up your changes as small-scoped as they can get, minimizing the impact of each change, and improving your potential for success.
Use a Comprehensive Change Management Tool.
As mentioned before, every change has a chance to (and often does) impact something that you didn’t expect. Using a comprehensive change management tool can help counteract this risk by logging all changes to specific Configuration Items (CI) and forming a timeline history of your environment. When you want to make a change, you can inform the software of what CIs will be touched in any way during the change, and the software may inform you of other changes or configurations in place that may cause problems you didn’t expect. This is especially helpful in large environments where inter-departmental communication isn’t exactly stellar.
Another good feature to look for in change management software is the ability to store documentation on the change. It should require that you provide a detailed risk assessment, or at least generate it automatically based on some input. It should also require a complete back-out plan, such as what’s required to back out the change, and who should be on hand to assist should something go wrong, as well as the “drop dead time” for when a back-out plan should be put into effect.
Finally, and this should be a given, document EVERYTHING about your change. Not only is this useful for yourself, but it can be an easy and often reusable way to communicate the details of your changes. Such documentation should always be placed within change management so that everyone can get access to it. This is actually one of the key areas of change management, so you need to remember to make comprehensive diagrams and walkthroughs of each change, even though you think you don’t need it for your own benefit – because it’s not just for your own benefit.
Provide Relevant, Timely Communication.
When making changes, it’s important to communicate well to everyone involved. You should know WHO you should be communicating change details to, but it’s also important to focus on HOW the communication is being done. Focus on providing customized, relevant details in any updates (i.e. email) to anyone you’re sending it to. For instance, you may be patching some software on a server for your change. You know what the server name is, what the patch is for, the name of the service, etc. Typically, your customer does not. Make sure that you think with the mind of the reader in your updates and provide them with sufficient detail that’s relevant to them. This might require you to send different updates to different groups of people before/during/after your change, but it will save you time in the long run, since they won’t have to call your phone to tell you they didn’t understand your email while you’re performing configurations. It will also show that you’re being responsible with your change and taking the time to keep them in the loop. As a result, they’ll be far less inclined to keep bugging you, and you’ll be free to work.
Requiring approval for changes by upper management can help force proper channels of communication. Some change management software requires digital sign-off on certain types of changes. Used correctly, this can work to your advantage. Getting approval for your changes means that you’ve put effort into making sure management knows the details of your change and approves of it. This helps protect you in the event that something goes wrong.
There are likely to be anomalies to these suggestions. For instance, emergency changes that are put into place to fix something that went wrong should obviously be done with a sense of urgency – and realistically, there’s rarely time to do proper change management in such situations. When possible, though, these suggestions can save you time and effort over the long run.
Undoubtedly, many feel that all of this “red tape” is a hindrance to doing the work that needs to get done. Unfortunately, many organizations implement change management poorly and it becomes that kind of beast. However, as a network engineer, you can control many aspects of this process – being proactive with your communication, documenting everything well, and being responsible with the amount of work you’re trying to get done in a certain amount of time are all things you can do to make life easier for everyone, especially yourself.