You’ll like this, and you won’t; and that reflects on how I’ve felt variously about this task/burden.
So, I’ve spent three weeks, almost full-time, on the work necessary to use Puppet to configure F5 Networks LTM via SOAP. Not just a few Pools and Virtual Servers; the whole box, from scratch. I knew this would be hard work; I’ve dabbled with F5’s SOAP API: iControl here and there in the last year or so and I conceptually understand Puppet but that wasn’t going to help much, this was practical work in the real world. Yet, I was confident I could get the job done and I was sure the very mature iControl API and the recently updated (and widely written about) Puppet F5 module wouldn’t stand in my way, only my lack of experience and understanding. How naive… but its been a useful cough ‘learning experience’.
Update January 2015: Note that the Puppet REST module for F5 LTM (if you have TMOS v11.5 or above) is now available if you are running Puppet Enterprise. The ‘legacy’ SOAP related module this article describes is now no longer available (officially).
Where have you been? Automate all the things, DevOps, NetOps, virtualisation and on and on. In this specific case (which I’m sure is common) there’s a need to build a test or development environment, tear it down, rebuild it and so on, on a very regular basis, for a fair number of environments. Agile, continuous delivery and very rapid development and deployment all figure and time-lines get progressively more aggressive as the various teams and the unit as a whole improves and learns. The advantages where the particular product involved are concerned cannot be overstated. Speed (yet reliability) is the number one priority.
There are many, many tools involved in the overall process but from an automatic configuration perspective Puppet is the tool of choice. Integrating the build of network related components is obviously a good thing.
I’ll be blunt, out of a very sizeable team, I am the only network person. I’m not doing routing and switching. I’m part of this whole thing because they have F5s and not HA-Proxy, nginx, Squid or something of that ilk. I’m part of this thing because I talked the talk around automation and programming and I meant it. Lucky I did, else I wouldn’t still be a part of it.
Regarding the Puppet F5 SOAP module, it’s written by Puppet, not F5, just so we’re clear.
There’s F5’s iControl SOAP API, as documented here. iControl was introduced with v4 of the BIG-IP Controller way back in 2001. Aside from some random interactions here and there I first took a real good look at iControl when I contributed to Jason Edelman’s unfortunately now dormant CPAL earlier this year. As well as forcing me to learn some Python for real, I got a feel for the challenges most face around manipulating and formatting the returned data and the like. I also (as I said unfortunately) got a feel for what happens when a project gets abandoned. No hard feelings and I could always carry on with a fork but that wasn’t quite the point.
Puppet, well, we’ve all heard, read and probably watched so much about it. In my case its Puppet Enterprise so no need to worry about any functional restrictions, bang up to date with version 3.3.
This one is the biggie and where I’ve collected my scars and gained my battle experience. The Puppet F5 SOAP module (available on the Forge here) doesn’t have anywhere near what I needed in functionality. A Puppet module provides an abstraction; its the ‘interface’ between the declarative configuration you provide in a manifest file and defines how that configuration is actually implemented (regardless of transport, protocol etc.). I don’t think the documentation for this module is particularly useful when it comes to writing a manifest file . Things become more obvious as you look at the provider and type files (best viewed on GitHub here) and use Facter to pull (rather than push) configuration from a device.
Me, I’ve had to write my own providers and types (in Ruby I might add) as well as fix a few errors with the ones provided. I have to say few things have provided me (pardon the pun) with so much satisfaction (and genuine highs) in the last 10 years of my career but equally few things have been so frustrating (at times). The combination of a provider and type within a module define what you can declare (create, delete, modify) as a configuration object or parameter on a device, often in relation to a specific protocol (where a network device is concerned) or feature. So, a Puppet module for a router, for instance, would have separate providers and types for say, switch interfaces, layer three interfaces, every RP and so on.
I had to write new providers for things like ServerSSL profiles, HTTP profiles, WebAcceleration profiles, Administrative Partitions and a whole host of other stuff. I’ve probably written more than those available with the module on Forge. Unfortunately I can’t share the code, as much as I’d like to (at least not yet) as I’ve been paid to write it and actually some of it really sucks, despite the fact it works. I have however forked the GitHub repository, corrected some errors and put in a pull request.
Unfortunately Puppet seem set on not continuing development of the SOAP module, in favour of using iControl REST. Whilst I understand this means they are fully focussed on making the future a better place, they are also abandoning those who can’t get to a recent TMOS version in a hurry (I still know plenty of ‘shops’ running v10, let alone v11.5) and ensuring only those on the bleeding edge (or with large capex budgets) have the ability to experiment and move forward. That’s a shame.
So, yeah, I did it, whoop de do. What did I learn;
- I really love programming. Really, the power, once you get to grips with things, its just amazing. I don’t need Iron Man’s suit I want his [email protected]$£%
- Ruby is pretty sweet. Having gone in-depth with TCL and having played with Python quite a bit, my barely scratching the surface experience with Ruby has left me wanting more.
- Good code is hard. Really @£$%ing hard. What can I say, I’m well aware that some of what I’ve ‘created’ deserves to go to hell. Coding well and applying simple good practise (checking something exists before trying to create it for instance) is way harder than you think. Sure I’m a n00b where Ruby is concerned but still, I’ve a fairly good idea of what I should be doing, actually doing it is something else. Either way, some formal training in some form would be very useful. Online tutorials are really good at the specifics, not so good where approach and concepts are concerned. I say this in relation to programming as an overall discipline as well as where each language is concerned.
- Good documentation is incredibly valuable. If someone had spent 30 minutes writing an example manifest file it would have saved me days… whole days. Sure, now I know what I’m looking at and what tools to use I can work it out but should I have to? I don’t think so.
- Bad and/or incomplete documentation is worthless. Don’t tease me, please me.
- As Greg loves to say, “the only constant is change”.
- Puppet and vendors like them don’t care about you. They really, really don’t, open source or otherwise. You are not their customer or their audience, sysadmins are. Go make friends with the storage folks and rage against the injustice of it all. You are now as unimportant as the guy who does the structured cabling when you need a new rack in the DC. You got owned, you just don’t know it yet.
- No, don’t get defensive, you got owned, good and proper. Think about who let that happen.
- I knew this already but if you are in an enterprise, get away from the layer three and below work now (or get to a SP). Your time is up in the next few years.
- Network vendors are the bottom, not the top, in the relationship with VMware, Puppet and all the rest. OpenStack assumes a VM (network appliance or not) only needs one vNIC, go figure.
- You’re gonna run ODL? No, the server/sysadmin guys and girls will and it’ll be a slave to OpenStack anyway.
- Touchpads suck.
- Beginners guides are in short supply. I’ve lost count of the number of websites, web pages and PDFs I’ve read recently. Nothing really gave me more than a nugget or two. Many a man page has been useless. There’s an opportunity to be had here (and yes, I may take it.)
Enough of the negative, allow me to describe what helped me on my journey;
- A big, fat hi-res display. Invaluable for coding and running a few terminals with everything available and visible on a single screen at a good resolution. I’ve certainly underestimated just how useful this can be.
- A Linux host. I actually really like Windows 7 and 8.1, especially when using useful keyboard short-cuts to bypass the endless click, click but Linux does simply rock. I write in Windows and code, troubleshoot and configure on Linux at a command line. I’m grateful I can.
- Sublime Text 2/3 – not to everyone’s taste and perhaps language specific (or not) IDEs are better, who knows, Sublime does it for me, again, especially when you learn a few keyboard short-cuts.
- Vim – just an amazing text editor and with this one, its all keyboard shortcuts (or rather, commands). Incredibly efficient, mature and useful.
- I bought a couple of books along the way but I can’t say I read them in any real depth, regardless, Eloquent Ruby was a joy to read, when I bothered.
- tar, watch and a whole bunch of other Linux commands and tools, awesome!
I’m not a Puppet expert, nor a Linux, Ruby or SOAP one. Forgive me my errors and feel free to point them out in the comments.