Using SNMP Config Management to Fix Screw-Ups on a Remote Device

29 July 2012 by 6 Comments

There’s been plenty of talk around here about complex network control through SDN, Netconf/Yang network virtualisation, and other rainbow-producing, unicorn-style ideas. But in our look to the future, we shouldn’t forget that sometimes we’ve just got a simple problem to solve, and our existing tools may be all we need.

A customer had shipped a switch out to one of their remote sites. It had been pre-configured, so the local resources just needed to plug it in. This done, the engineers at HQ went to login to the device. Telnet works, except…

Switch>en
% No password set
Switch>

Okay, that’s not good. Obviously, something went wrong with pasting the config template into the device. We don’t have experienced network engineers on site. This thing is 1,500km away. How do we fix it? Well, the obvious option is to use someone on site, walk them through plugging a console cable in, and get them to set a simple enable password. No big deal, right? Except it just wasn’t working. Console connection problems, settings wrong…for whatever reason, it wasn’t happening. The person on site just couldn’t get a working console connection, so setting the enable password that way wouldn’t work.

What’s plan B then? Ship a new switch? Send a network engineer out to site? Yeah, except that’s time and money. What else? Well…we’ve got a working IP connection, right? Did the SNMP communities get pasted in properly? Let’s check that out – add the device to the NMS, yes, it shows up correctly. Right, well let’s try telling the NMS to push out “enable password cisco”. Set up the deployment job, wait for it to complete, and:

Switch>en Password
Switch#

Success! But how? How did the NMS change the config like that, if we didn’t have an enable password set? Not all engineers realise that if you have set an SNMP read-write string, you can use that to manage device configs on your Cisco gear, using the CISCO-CONFIG-COPY MIB. You can tell it to copy either the running or startup config to or from a network location. It’s more often used to tell a device to backup its running config to a TFTP server, but it can easily be used to tell a device to grab a config from a TFTP/FTP server. Cisco has some documentation on how to do this using freely available command line tools, although these instructions do seem out of date. This reference is probably a better one to follow if you’re using command line tools. Perl modules are also available if that’s your thing. It’s a lot faster using a tool designed for the job (CiscoWorks, HP iMC, SolarWinds NCM, HP NA, etc), and you don’t have to worry about the implementation details. But hey, you don’t always have the choice. Or maybe you’ve got some very specific thing you’re trying to do, and so custom tools are the only way.

This technique can also be used to save you if you’ve somehow hosed your AAA settings…not that any of us would ever admit to locking ourselves out of a switch when trying to get AAA configured. No sir, that must have been that other engineer over there…

Filed Under: Blogs, Network Management Tagged With: , , ,
About Lindsay Hill

Lindsay (@northlandboy) is a network management consultant, with experience across networks, servers, applications and security. He is CCIE #36708, RHCE, CISSP and HP MASE. More of his own content is at lkhill.com.

  • Rizky

    you can also try http to the device and set your enable password

    • http://twitter.com/northlandboy Lindsay Hill

      Yep – sure can. Only problem is that I don’t know anyone that ever (intentionally) enables the HTTP interface!

      • http://twitter.com/jullimaggi /Julli

        Except if they are using the WSMA features. BTW do you guys know if anyone is using that?

        • http://twitter.com/northlandboy Lindsay Hill

          I haven’t (yet) come across it in the wild, although I like the idea of it. Haven’t looked into it too deeply, but I think I’d prefer to use it over SSH, rather than HTTP.

      • MarvinRhoads

        Enabling the http(s) interface is useful for customers without in-house full-time network engineers wishing to use Cisco Configuration Professional and/or Cisco Network Assistant to manage and configure their devices.

        • http://twitter.com/northlandboy Lindsay Hill

          This is getting a bit off-topic, but sometimes I think that those customers might as well just buy a cheap switch, rather than paying the extra for Cisco kit. Too many times I’ve gone into places that have paid plenty for Cisco switches (because no-one ever got fired for buying Cisco), and all they’ve done is taken it out of the box and installed it in the rack. Default (ancient) IOS, everything on VLAN1, hostname Switch.
          If they don’t need inhouse network engineers – and to be clear, plenty of places don’t – then they probably don’t need the advanced features that Cisco makes possible.