More Cisco 3750 Fun: Password Recovery Without Reloading The Stack

First off, there are no miracles here, but in certain circumstances, this process could help you recover the credentials to a production Cisco 3750 stack without having to reload the entire stack and run the normal Cisco password recovery procedure. This will work for local credentials where the passwords are stored in the configuration with weak encryption (Cisco type 7).

Here’s the summary: take a spare 3750 and join it to the stack using normal, Cisco-approved procedures. It will get a copy of the config from the stack master. Power it down, and take it back out of the stack. Interrupt the boot sequence, initialize flash, and cat the config file. Use a Cisco-7 password decoder to decrypt local credentials.

Disclaimer: I and most other people have had good luck joining and removing stack members in live stacks, when you follow the all-important step of making sure the switch you’re going to add is POWERED OFF before plugging in the stackwise cables. If you power up the new switch, then plug in the stackwise cables to join it to the stack, you will probably blow up the stack. Don’t do it. That said, as far as I’m concerned, anytime you are plugging and unplugging stackwise cables, you risk upsetting the IOS deities. Therefore, if this stack is part of your super-critical “five nines is not enough” environment, don’t do this…unless you’re feeling lucky. You’ve been warned.

More details:

  • You need a spare 3750 running an IOS version compatible with the stack. If the IOS version is too far off, the switch won’t be able to join the stack due to stackwise version incompatibility. An exact IOS match is ideal, but you might be able to get away with minor differences. If you don’t know what IOS version the stack is running, you can find out via SNMP assuming you know the proper SNMP information and the switch has been configured for SNMP. As a last resort, you can check the version sticker on the back of the switch (what Cisco shipped it with), and try that one…if it’s never been upgraded, you’ve gotten lucky.
  • The stack must be smaller than nine high. Nine switches is as high as a 3750 stack goes last I knew, so you can’t join a tenth switch.
  • You can temporarily use the redundant stack cable to join your switch to the stack, but make sure that ALL of your stackports are functioning first…otherwise, you’ll partition the stack when you unplug the redundant stackwise cable.
  • Plug in the stackwise cable to the temporary switch, making sure the new switch is powered off.
  • Power up the new switch. Presumably, you’re watching it boot via a console cable and terminal session. The boot could take several minutes, but once the new switch has joined the stack, the stack master will send a copy of the configuration to the new stack member you’ve just added.
  • Power down the new switch, and put the stackwise cables back the way they were. As you were using the redundant stack cable, this should not pose any problem for the stack. Probably. (See the disclaimer above.)
  • Take the 3750 and start up the normal password recovery procedure, which is to power it and hold down the button on the front left of the switch. If you’re monitoring via console, you’ll see the switch come up to the “switch:” prompt, at which point you can let go of the button.
  • Do “flash_init” and “load_helper”. You should now be able to do a “dir flash:”. You should see “config.text” listed.
  • Do “cat flash:config.text”. This will list the contents of the switch configuration. Since this config is a copy of the switch stack config, you’re now seeing the config the stack is running.
  • Look for passwords in the configuration encrypted with Cisco-7 cipher. This is not a strong cipher, and can be decrypted with commonly available tools, such as the one available from SolarWinds (not free), or online. If you’re unlucky, there might not be anything you can work with in the config, but maybe you’ll be as fortunate as I was this morning when this little procedure bailed me out.
Ethan Banks
Ethan Banks, CCIE #20655, has been managing networks for higher ed, government, financials and high tech since 1995. Ethan co-hosts the Packet Pushers Podcast, which has seen over 2M downloads and reaches over 10K listeners. With whatever time is left, Ethan writes for fun & profit, studies for certifications, and enjoys science fiction. @ecbanks
Ethan Banks
Ethan Banks
  • Jack

    Clever. If you’re lucky enough to be cleaning up after someone that isn’t using ‘secret’ in the creds…

    Also, you don’t need a decryptor to break the weak passwords, IOS has one built in. Simply:
    !
    username foo password 7 04590A144F
    !
    key chain hax0r
    key 1
    key-string 7 04590A144F

    r1#sh key chain hax0r
    Key-chain hax0r:
    key 1 — text “bar”

    -John
    @StuckInActive

    • http://packetattack.wordpress.com Ethan Banks

      Awesome trick, did not know that.

    • http://www.TWNCommunications.Net/GC James Rogers

      That’s a nifty trick, thanks!

      • http://astorinonetworks.com Joe Astorino

        Really nifty trick with the key-chain, I was not aware of that feature. Nice work!

  • Steve

    Can you answer this one?
    I have two 3750G stacked and I am locked out of console access. The startup config Console settings are login and local. However, there is no local database, i.e, no username and password is specified. After attempting the normal password recovery (init_flash, load_helper, dir, rename, etc), and booting the console screen still shows the “username” prompt. These are not production switches so I can anything to fix this problem.

    • Yap Chin Hoong -

      Please post the recovered ‘flash:config.text” to us, most probably you are experiencing a AAA authentication or authorization problem. Thanks.

  • Erik Nagy

    Would this method be to recover the password of the whole stack or rather the individual “extra” switch. It is a bit unclear for me.

    I am looking for a solution to recover the password of a 3-device-stack. Tried getting rid of the config but due the the stack being interconnected the configuration is passed around and once new master is elected the device that I reset automatically joins in as a slave meaning it gets the config from the other devices on the stack. This puts me back to where I started.

7ads6x98y