I recently completed a challenging upgrade on a pair of production F5 3600s running 10.2.0, going to 11.2.1 running the LTM module. In hindsight, it shouldn’t have been a challenging upgrade, but that was due to the things I learned along the way.
License reactivation. The upgrade document doesn’t say much about this. “For each device, reactivate the license. On the Main menu, click System > License. Click Re-activate.” The rest of the story is that if you don’t reactivate the license, you’re not getting the upgrade done. In order to do the reactivation, you must have a current support contract on the F5 device, or the device is ineligible for the upgrade.
Now here’s the weird part. If you do a manual reactivation by going through the F5 activation site, the site will let you paste in a dossier and generate a license no matter what. By no matter what, I mean no matter whether you have a current support contract on the device or not, a license gets created. You can copy and paste the new license into the F5 device and think you’re all set. No red flags, no error messages, no warnings…even if you don’t have a current support contract. You have to read that license file to MAKE SURE that what got generated is actually a valid license. The generated file will tell you if the device is not covered by a support contract, buried down in text.
If you fail to notice that your license reactivation was in fact a failure, what happens is that once you boot into version 11, the F5 complains that you need to activate the license…which you’ll try to do…and fail (again). Then you’ll boot back into the v10 partition, curse the licensing deities, and schedule a new maintenance window to try the upgrade again.
Last point on this. YES. We had a support contract on the F5s we were upgrading. However, the reseller that sold us the contract renewal did it wrong. Therefore, F5’s database felt our 3600s were ineligible for the upgrade. Sigh. My F5 rep straightened out the mess.
Another way to verify that your support contract is current for a given F5 device serial number is via this link. And now on to the next lesson learned.
The SSL server profile defaults change from version 10 to version 11. Therefore, you’re going to get the new defaults if you want them or not. And chances are, you might not. In our specific situation, we were encrypting between the LTM and the pool member using an SSL server profile on several HTTPS virtual servers. Some of these virtual servers simply weren’t working after the upgrade. The site wouldn’t load into the browser window, and there was nothing obvious in the logs that clued us into the problem. After opening a case with F5 to help troubleshoot, a support engineer pointed to what he suspected was the problem, found in SOL11220. And he was right.
SOL11220 is an outstanding breakdown of the changes in the SSL profile across several versions of F5 software. Here’s the quote with the answer that resolved our issues.
Beginning in BIG-IP versions 10.2.3 and 11.0.0, the BIG-IP SSL profiles support the TLS Renegotiation Indication Extension, which allows the user to specify the method of secure renegotiation for SSL connections. In BIG-IP 10.2.3 and 10.2.4, the default value for the Server SSL profile is Request. In BIG-IP 11.x, the default value for the Server SSL profile is Require Strict.
The values for the Secure Renegotiation setting are as follows:
Specifies that the system requests secure renegotiation of SSL connections.
Specifies that the system requires secure renegotiation of SSL connections. In this mode, SSL connections initiated from the BIG-IP LTM system to an unpatched server fail when renegotiation is enabled.
- Require Strict
Specifies that the system requires strict secure renegotiation of SSL connections. In this mode, SSL connections initiated from the BIG-IP LTM system to an unpatched server fail when renegotiation is enabled.
In summary, the change to “require strict” as the default means that if you’ve got a server that doesn’t support secure SSL renegotiation, your app is broken. The solution is to update the SSL server profile to go back to “request”.
The security conscious are rightly skeptical that this is the proper thing to do from a security perspective. And if it’s that important to your environment, then most likely you’d have patched your servers already so that they can cope with secure SSL renegotiation. Clearly, F5 feels strongly enough about secure SSL renegotiation that they made it the default setting, knowing that it could break customer applications (how could they not know this?), so obviously there’s some sort of a risk they feel strongly about mitigating. That said, if you consider the SSL server profile, you’re talking about the session between the LTM and the pool member – a session that’s logically behind the F5, and very probably running over a trusted network infrastructure that you own. The chances of an SSL session being compromised between the F5 and the pool member should be small. Therefore, setting the default profile back to “request” will be a reasonable risk for most organizations, as it’s a small, manageable attack surface, IMHO.
Aside from those two issues which did cause us a lot of heartache, the upgrade actually went amazingly well. There’s a lot of changes in the system architecture when going from version 10 to version 11, especially with high availability. That all upgraded somewhat magically, as the F5 upgrade document said that it would. Having built several v11 HA pairs by hand in the past and always struggling to get the cluster working properly, I had little hope that an upgrade would pull it off successfully. But, that turned out to be the smoothest part of the upgrade process.
One last miscellaneous note is that if you’re still using the old fashioned blue heartbeat cable, don’t forget to unplug it when you are disconnecting the standby device from the network, getting ready to reboot it into the upgraded partition. If you don’t, you could end up with a failover happening when you don’t want it to, interrupting traffic to your user community.
So…is version 11 worth it? Get there if you can. There’s a lot of nice features, but more to the point is that F5 will leave you behind if you don’t keep up. Eventually, you’ll be forced to upgrade, which is never the desirable option. Buckle down and make it happen if your environment will allow it. And may $deity help you if you’re an ITIL shop.
F5 Product Information Request (check the support contract status)