HP IMC: Backup Cisco ASAs via SSH

Recently, I wrote a review of HP’s Intelligent Management Center. One of the things I mentioned having problems with was backing up some Cisco devices via SSH. Out of the box, IMC doesn’t like backing up IOS-based devices if you use SSH (telnet is fine), nor will it backup the configuration of a Cisco ASA.

The good thing about IMC is that it’s just a bunch of scripts underneath it all, and it also produces quite a good volume of logs, so you can work out where it’s failing and maybe fix it. First, we’ll take a look at the default setup, and what happens we try to run a backup. Then I’ll go through a possible fix – note that the method I’ve used here is very much of the “nasty hack” variety. I have heard that the next release of IMC will support this properly, and that’s what you should aim to use – this is just a short term measure, to get things working in the interim. I have implemented a better workaround at a customer site, using a different SSH client, but for some reason I could not reproduce this in my test lab. No doubt some readers will also have their own opinions on how it should be done.

The setup:

I’m going to assume you’ve added your ASA to IMC, SNMP is working, the “Login Type” is set to SSH, and you’ve given it valid credentials. I’ll also assume that you’ve tried manually connecting to the device via SSH from your IMC server. (Don’t laugh, sometimes when troubleshooting something more complex we forget the simple things like ACLs.) Next, make sure that the “File Transfer Mode” is set to SFTP. Go Service -> Configuration Center -> Options. Either set the default to SFTP, or if your default is something else, then add an exception for the ASA. Here I’ve set my default mode to SFTP:

File Transfer Options - Click for Larger Image

Now, let’s kick off a manual backup to see what happens. Go back to Device View, and find your ASA. Click on it. Scroll down, and on the right should be an option “Backup Configuration File.” Clicking this triggers an immediate backup attempt of both the startup and running config files. The next screen will show you the current status of the backup attempt. Stay on this screen for a while, waiting for it to complete. If everything is working properly, this should complete quickly. If there are problems, it may take 5-10 minutes to timeout. Be patient, get a cup of coffee. After a while, you’ll probably see something similar to this:

Backup Failure - Click for Larger Image

Failure. Not cool. Let’s click on the icon in the Details column, to get some more info:

Failure Details - Click for Larger Image

It looks like the error messages should be helpful, but to be honest, they’re not. Messages about TFTP servers not running aren’t much use if you’re using SFTP. We’ve already tested the SSH parameters; we know they’re right. “Please check if the script is correct,” is on the right track, but it doesn’t offer much detail. Where to from here? The key locations for us are C:\Program Files\iMC\server\tmp and C:\Program File\iMC\server\conf\log\

The log directory contains detailed logs on all aspects of iMC operation. The log file we’re interested in is imccfgbakdm.<current date>.log. It’s a pretty detailed log file, and it takes a while to learn what to look for here. But scrolling through it, we can see messages like these:
2012-04-14 21:01:12.861 [INFO (0)] [THREAD(4524)] [CComponentAdapter::isDevSupportServiceAction] dev_id: 3, adapter_name: CiscoASA
2012-04-14 21:01:12.861 [INFO (0)] [THREAD(4524)] [CScriptProcessor::exec()] Begin to execute by cli.
2012-04-14 21:01:12.861 [INFO (0)] [THREAD(4524)] [CScriptProcessor::exec()] Case_no: 3704_459038833, service_name: ConfigBackup, action_name: backup_running_config, input_vars: TFTPFile=C:\Program Files\iMC/server/tmp\running_459038832.cfg?_?TFTPServer=172.16.2.101?_?VpnName=
2012-04-14 21:01:12.924 [INFO (0)] [THREAD(4524)] [CScriptProcessor::exec()] Success to spawn process, pid: 5296
2012-04-14 21:06:12.925 [ERROR (-1)] [THREAD(4524)] [CScriptProcessor::exec()] Timeout to execute process, pid: 5296, ret_pid: 0
2012-04-14 21:06:12.943 [ERROR (10)] [THREAD(4524)] [CScriptProcessor::exec()] Failed to execute by cli method.
2012-04-14 21:06:12.943 [INFO (10)] [THREAD(4524)] [CScriptProcessor::exec()] Finished to execute, result:

Okay, so something was started, but it failed to execute? What happened? Here’s where the contents of C:\Program Files\iMC\server\tmp can help. IMC uses this for temporary storage of the output from the telnet/SSH session with the switch. By looking in this directory for files named “tcl_*”, we can see some of what happened. In this case we’ve got two files, with timestamps 5 minutes apart. If you read through all of the imccfgbakdm log file, you’ll see that they match the times IMC tried backing up the running config, then the startup config. Here’s the contents of one of the files:
The server's host key is not cached in the registry. You
have no guarantee that the server is the computer you
think it is.
The server's rsa2 key fingerprint is:
ssh-rsa 1024 f9:a1:bd:fe:23:89:61:b1:c6:0a:2b:e7:d5:14:c3:15
If you trust this host, enter "y" to add the key to
PuTTY's cache and carry on connecting.
If you want to carry on connecting just once, without
adding the key to the cache, enter "n".
If you do not trust this host, press Return to abandon the
connection.
Store key in cache? (y/n)

Hmmm. Looks like the standard warning you get when connecting to a new system via SSH. Why doesn’t it proceed? Here’s where we need to take note of the earlier line that mentioned “…adapter_name: CiscoASA”. This tells us which set of scripts IMC used when trying to connect to this device. These are located at C:\Program Files\iMC\server\conf\adapters\ICC\<vendor_name>\. I’m not going to go into the exact structure of this directory and the files in it – you can look at the Admin Guide for that. But basically, there’s a few TCL scripts to do the config backups, Perl scripts to clean up the output, and XML files to define what the different scripts do. After a while you’ll get the hang of what the different bits do. For now we’re going to look at enter_exec.tcl – this is the first script executed when logging into the device. Since it seems that it’s hanging at initial login, this is a good place to start. I’m no TCL/Expect programmer, but we can get the general idea of what it’s up to. It’s looking for login-related items, sending the password, and then looking for a successful login. But looking through the while loop, it doesn’t seem to have anything to cover the situation where the host key is unknown. So let’s try adding a bit of code to cover that, and see if it helps. Let’s add this stanza to the while loop:

} "Store key in cache? (y/n)" {
send "y\r"
}

Save that, and try the backup again. This time it fails again, but it’s much quicker to do so. So it’s not timing out, it’s getting some error. First we’ll verify that the SSH key has been saved, by checking the registry at Computer\HKEY_USERS\.DEFAULT\Software\SimonTatham\PuTTY\SshHostKeys\ – look for a key with the ASA’s IP address in the Name field. This is where iMC stores SSH keys, as iMC is just calling plink.exe (from the PuTTY developers) when making SSH connections. OK, the key is definitely there, what does the log file tell us then:

2012-04-14 22:01:43.118 [INFO (0)] [THREAD(8576)] [CTclExecutor::exec_impl()] Begin to exec: C:/Program Files/iMC/server/bin/../../server/conf/adapters/ICC/Cisco/CiscoASA/enter_enable.tcl
2012-04-14 22:01:44.117 [INFO (0)] [THREAD(8576)] [CTclExecutor::exec_impl()] Finished.
2012-04-14 22:01:44.117 [INFO (0)] [THREAD(8576)] [CTclExecutor::exec_impl()] Begin to exec: C:/Program Files/iMC/server/bin/../../server/conf/adapters/ICC/Cisco/CiscoASA/backup_startup_config_cli.tcl
2012-04-14 22:01:44.129 [ERROR (0)] [THREAD(8576)] [CTclExecutor::exec_impl()] Error occured to exec cmd: C:/Program Files/iMC/server/bin/../../server/conf/adapters/ICC/Cisco/CiscoASA/backup_startup_config_cli.tcl, error message: Could not show the startup-configuration (show startup-config).

2012-04-14 22:01:44.367 [INFO (0)] [THREAD(8576)] [ScriptTool::execScriptAction()] ResultNo: 11, CmdResp: Using username "lindsayh".
[email protected]'s password:
Type help or '?' for a list of available commands.
asa01pi>
asa01pi>
asa01pi> enable
Password:
Invalid password
Password: **********
asa01pi#
asa01pi# , ResultMap: ERROR_MESSAGE=Could not show the startup-configuration (show startup-config).

2012-04-14 22:01:44.427 [INFO (0)] [THREAD(4532)] [imcscriptttol] log: ===============================

Interesting. This time it manages to login, and gets to enable mode. Something’s not right though – it adds an extra carriage return after “enable” which results in the “Invalid password” error. It still gets to enable mode, so it should be OK, right? Unfortunately the backup_running_config_cli.tcl and backup_startup_confg_cli.tcl scripts both look for “nvalid”, and exit with an error message if they see it. Where does the extra carriage return come from though? Let’s try using plink.exe ourselves, and see what happens. Open a command window, cd to C:\Program Files\iMC\server\bin\ and run plink.exe <username>@<IP>. Enter a username and password to get to exec mode. Now enter enable, and hit enter – weird – someone seems to have slipped an extra carriage return in. Strange. I haven’t been able to work out exactly what causes this – it seems to be something to do with terminal settings, and interpreting escape characters. At a customer site, I switched to using a version of OpenSSH for Windows, which seemed to work better, and did not display this behavior with multiple carriage returns. That approach still requires some fixes to the TCL scripts, but is better overall. I couldn’t make it work in my test lab though, which is why I’m showing you a dirty hack here.

I couldn’t get the backup scripts to properly interpret any output they were getting from the ASA via plink.exe, so I’ve taken a more brutal approach: Disable some error checking, send “show run”, wait a little bit, grab the output and assume it’s correct. Here’s what I did:

  • Make the changes as above to enter_exec.tcl
  • Edit both backup_running_config_cli.tcl and backup_startup_config_cli.tclin <code<C:\Program Files\iMC\server\conf\adapters\ICC\Cisco\CiscoASA. Make these changes:
    • Comment out “set timeout $long_timeout” – not strictly necessary, but it makes it complete a little quicker
    • After that line, add “send "terminal page 0\r"” – this makes it display the config as one long file, with no page breaks
    • Within the while loop, comment out the stanzas matching on “nvalid” and “$enable_prompt” – this tells it to ignore the problems caused by the extra carriage returns
    • In the timeout stanza, comment out the “set ERROR_MESSAGE” and “set ERROR_RESULT” lines.

The effect of these changes is to make the scripts send “show running-config”, wait until the timeout is reached, and then assume all content received up to then matches the running config. I told you it wasn’t pretty, but it gets the job done – try running another backup now, and take a look at the result.

In normal circumstances, this works fine. But if the backup failed for some reason, you may not get any notification. You have been warned. As I said earlier, this will probably work out of the box with the next release. But it’s nice to know a bit about how some of the backup scripts work, so that you can go and make your own custom changes, should you need to support a random device that HP doesn’t support.

Lindsay Hill

Lindsay Hill

Network Management Consultant
Lindsay (@northlandboy) is a network management consultant, with experience across networks, servers, applications and security. He is CCIE #36708, RHCE, CISSP and HP MASE. More of his own content is at lkhill.com.
Lindsay Hill
  • http://www.ciscoacademy.org/ Rose Miller

    Sounds like a lot of work. I’ve been using RANCID (Really Awesome New Cisco confIg Differ) for years to back up configs from a variety of vendors’ gear

  • http://twitter.com/northlandboy Lindsay Hill

    You’re right – it is a lot of work to make this bit work. The thing is that this is a lot more than just configuration backup management – it’s just one bit of the overall monitoring/configuration/compliance picture. What they’re trying to do here is to have all the bits and pieces together in one place. You then have fewer tools to manage, and all your devices in one place. 

    But it does mean that sometimes it doesn’t do everything you need, for all your devices.  I tend to think that ultimately, a dedicated-purpose tool (e.g. RANCID) will always be better at its key purpose. But backup/diff is only one thing – I also want to monitor my devices, push configuration to them, check configs for compliance, push new IOS versions, etc. Do I want to have to have different tools for each, where I need to add my new devices in multiple places? Or maybe have semi-integrated suites (e.g. NNMi + NA) ? Or just have it all work in one place? Add my device once, be done with it. Well I can dream.

    In future, I expect this bug to be fixed, and you shouldn’t need to worry about this sort of mucking around!

  • http://www.myteneo.net Aaron Paxson

    Great post!

  • Steven Polley

    For a company as big as HP, how is it that they manage to have this many bugs in a production product like this? I’ve been using IMC for 6 months and I’m literally pulling my hair out. The sad thing is, all my switches are made by HP too (Procurve, not Comware, but still!).

    This is a pathetic excuse of a product for a really great idea. The idea behind IMC is awesome, but they way it’s implicated (java on the web front end, really?) is poor.

    The web interface is clunky, doesn’t use AJAX in any way (which has been around for YEARS, mind you) and is very unresponsive. The back end is full of problems as well. I’m not so pissed off that other vendors don’t function 100% as I am that HP’s own products don’t work with this software. SFTP filetransfers? No way, unless you spend hours working through the TCL scripts to make it work. Deploying scripts/segmetns to HP configs? Nope!

    The only thing this is 100% reliable for is monitoring my network and notifying me of a problem.

    1/10, should change product to Stupid Management Center.

    • http://northlandboy.com/ Lindsay Hill

      Can’t disagree too much, but just a few notes:
      * IMC comes from the 3Com acquisition – that’s why you tend to have more issues with Procurve than Comware devices. No excuse now, 4 years post-acquisition, but that’s the history.
      * Java is going away in the next version (later this year).
      * I’ve seen several SFTP problems with Procurves caused by firmware issues on the Procurves themselves. Seems to be fixed in later versions though. Seems to work fine on my test kit, but I don’t have a lot of Procurve platforms to test against.
      * We’ve got an IMC-specific forum running at http://www.netopscommunity.net – we’d love to have you add your experiences there. We might be able to help with debugging some of the problems you’ve got. Related to that, we’ve also got a GitHub repository of custom adapters, where we’ve either added new adapters, or published fixes to HP-written adapters. I’d love to be able to add any fixes you might have developed for Procurve support.

    • Mareg

      Over year it still 100% true :-(

7ads6x98y