I spent 4 hours on the phone today with my new best friend in a data center far, far away, while I walked him through password recovery on several different devices.
First up were a 2 sets of redundant Cisco PIXen running 7.0x code. That went pretty much without a hitch, although it’s a little more annoying of a process than many other Cisco devices. With the PIX, you break the boot sequence using a console port like most other Cisco boxes, but then you have to load a bin file via TFTP that will let you reset the system passwords. Worked fine, no issues. Tip if you’re trying to recover on a redundant pair – shut them both down, recover the passwords on one of them, then power up the second one. The second box will get the recovered config mirrored to it from the first box. If you instead leave one of the PIXen up and running while recovering the password on the other, you’ll end up with the recovered PIX getting his config overwritten by the other PIX you left up and running, meaning you’ll just have to start all over again.
Next up was a box that my new best friend told me said “Altiga”. Ah, yes – Altiga. Cisco bought them and made the VPN3000 concentrator series once upon a time – and 5000s, although I never ran into anyone that had a 5000. The 3000 was one of my favorite VPN boxes. I could make it talk IPSEC to just about anything, do bizarre NATs, and so on. I just haven’t worked on one for about 5 years. I was a little nervous that the box might be an early Altiga example and that the Cisco password recovery docs wouldn’t apply, but it was all good. By the way, a DB9 – DB9 straight through cable is what you need for the Altiga console from what I can tell – the typical Cisco rollover cable is a non-starter here.
A Cisco Catalyst 4507R, which I expected to be the easiest password recovery, was a bit of a pain. First round, we discovered that one of the two SupIV engines was hosed. My new best friend reported that the sup was at rommon when he consoled into it. I had him reload it to see what was up, and it was stuck in a loop. I’m guessing it couldn’t find its IOS, but I didn’t get enough information to know for sure. They’ll ship it to me, I’ll throw it in a spare 4500 chassis, and we’ll see if I can heal it. The remaining SupIV was obviously functional, as the switch was in service. We did a reload, broke the boot process with ctrl-c, and then at the rommon prompt, set confreg to 0x2142 per Cisco’s instuctions. It seemed to boot right back up as normal instead of loading IOS without loading the config, which is what was supposed to happen. So we reloaded again, got back into rommon, ran confreg by itself with no value, and then answered the questions per Cisco’s doc. This time, the switch loaded as expected: config free. We proceeded as normal from there, only to discover a bit later that all of the layer 3 interfaces were shut. Weird. I walked my new best friend through turning two of the critical interfaces back up: the one facing the Internet, and the one facing his server network. When he started getting mail and pinging Internet hosts, he peeled himself off the ceiling and we were okay again. I wrote a little script to “no shut” the rest of the SVIs, and we were done.
What a mess, but a good Saturday’s work. Monday, we’ll work on locking things down, DOCUMENTING how they are locked down, and making the network devices manageable from afar. One of these days, I’ll meet my predecessor, and I hope to ask him what he found so frightening about documentation. Without it, sometimes a network engineer’s job comes down to password recovery.