I never enjoy the night before a big change. By “big change”, I mean where you know you’re headed into work at some unusual hour to execute a plan you’ve been obsessing about for a couple of weeks or even months. The sort of a change where there is most definitely going to be a service impact. The highway is under construction, and the users are going to have to find some other way to access Facebook.
Some changes are one-for-one. You’re doing a simple hardware replacement, and there’s no real design change. Those are troublesome, but minimally threatening. Usually all that’s happening is faster, better hardware – but at least it works the same. Probably. I mean, sure the software is a newer rev with new features that might have some bugs, and the hardware is unproven and might go all weird once you start putting a production load through it. But, hey…it’s usually no big deal.
Other changes aren’t hardware replacements. They are design changes. Implementing new features, segmenting the network, improving redundancy, etc. That kind of work is potentially scarier. There’s often a big risk associated with a design change, and the blamethrower will be aimed in your direction, ready to fire. Did you think the changes through effectively? Do you really understand what that command is actually going to do when you tap your magic fingers across the keyboard at oh-dark-thirty? During those kinds of changes, there’s a bit more anxiety. The more complex the change, the more devices you have to touch, the further away the equipment, the more people involved, the wider the potential impact to meatspace if things go badly, the greater the anxiety.
This night before, I’m set to go for a change tomorrow morning with a bunch of moving parts: changing firewall vendors, changing ISPs, and changing public facing IP block. I documented sixteen steps to go from start to finish. Not that sixteen steps is all that much…but it’s plenty enough to keep track of. All the code is written out already, so that all I’ll have to do is copy and paste for most things. The conference bridge is ready. The impacted users have had their expectations set quite low for the four hour change window.
I have medium anxiety. I was supposed to be on-site to do the change, but my passport was victimized by the woeful inadequacy of the US federal government. The passport didn’t make it in time. So, I’ll be doing the change remotely, as I can’t cross the border. Doing the change remotely doesn’t bother me, except that the only connection into the remote office is via Internet-based VPN. Since I’m messing with all the parts that that support the VPN tunnel I’ll be using, I’ve had to take careful thought to making sure I don’t cut myself off from the remote site while working through the change. I have that sorted…backups of backups if something unexpected happens.
I expect to sleep well, but I’ll probably wake up early. Once I’m awake and my brain starts visualizing all the steps and second-guessing itself, I won’t go back to sleep. It’s game day. It’s time to shine. High alert. Adrenalin pumping. Drive to the office on the nearly empty highway, and park the car in the nearly empty lot. Walk into the office with a bit of an attitude, like a cowboy stepping onto dusty Main Street at high noon for a showdown. Slap the laptop into the dock station, and fire it up. Angle the monitors. Check the clock. Bring up all the windows. Log into all the equipment. Review the plan one last time. Check the clock again. Dial into the conference bridge. Wait a bit. Swipe the mouse to prevent the monitors from sleeping.
Everyone’s on the bridge. Clock says it’s time to go. Step 1. Copy. Take a deep breath.