In my continuing series on the decidedly boring, I would like to discuss the subtle, yet paralyzing, evil of stale documentation.
In my experience, stale documentation can be useful or it can be disastrous, depending on how much is wrong. Personally, when I see more than a couple of tiny mistakes in a diagram, spreadsheet, or other document, I mentally write it off as untrusted. Whether it is out of date or thoroughly incorrect to begin with is moot when the phones start ringing.
So how do we manage our documentation to avoid pulling up diagrams to troubleshoot and seeing they were last updated three years ago? I’m glad you asked ….
Document less stuff
A mistake I see in every direction I look is over documentation of minutiae. I believe this to be the result of well-intentioned people putting forth an honest effort to do a better job for the folks who come later. But, the result is a huge smattering of garbage documents with names that are just ambiguous enough that you need to open them to rule them out as outdated or not apropos. And if that time sink of a directory is shared amongst teams you will not be allowed to simply dump 90% of them into a sub-directory named “archive.”
To combat this, I try to get people to only document what they can commit to keeping up to date. Oddly enough, this level of information appears to be drastically lower than what someone is willing to record once and dump on a Sharepoint server. For example, a new data center should have things like the street address, remote hands phone numbers, and cage number recorded – things that change rarely to never. From there, work downward, recording circuit IDs, IP ranges – things that will change a reasonable amount of time but are important enough to have on hand. When you get to what is in Rack Unit (RU) 36 in cabinet 0012 on row 6 you might want to stop and consider whether you are going to change this wiki page every time that RU changes. Sometimes it really is critical knowledge, but most of the time it is not worth the effort. If you *do* document things to that level of detail and let the information get stale, eventually it will get pulled up during a planning session or outage and make someone pretty angry.
The next big cuplrit I see is location and redundancy. If you can commit to a high level of detail, put it in a database. Not databases – a database. Some of the best documentation I have seen was in the form of a database filled with everything from server operating systems to IP addresses of ISP links. Databases have a few key advantages:
- This source is now your authority, queried by any number of tools as well as humans trying to figure out why your San Diego site is down.
- They are just complicated enough to keep most fingers out of the cookie jar, with database admins being, on the whole, some of the most crotchety people I know. The result? People are not heaving new tables, categories or directories around randomly.
- Databases as repositories are arguably step one toward automating things like aggregate bandwidth graphing, configuration auditing, and cool new things as they become possible, like using LLDP to populate switchport-to-server-NIC tables.
Finally, perhaps the largest impediment I see to usable documentation is the lack of a technical stick to beat geeks with. Most mature IT shops have some level of “required” documentation, but that requirement is in quotes because it is entirely possible to bring up a new server or router while putting the documentation on layaway. But it’s okay because we know that geek will remember to write down what interfaces go where, right?
Effective documentation is forced on us as a part of the provisioning process. I know – I hated writing that as much as you hated reading it. But the honor system does not work. It did not work when I was a kid trick or treating on Halloween and ran into a basket of candy with a sign admonishing me to take just one piece, and it does not work now when a busy IT worker builds out six unique servers for three different teams in a week.
My experience with the “evil D” of documentation is that if you limit the scope of details to what you can reasonably maintain, put the information in a central database with controls around it (including backing it up!), and force the documentation to be done as part of provisioning, your work day is going to be a lot more sane. Operations folks will know what information is recorded, where it is, and how to get to it. Unfortunately, my experience also tells me that most companies’ “documentation” consists largely of a Sharepoint server that looks like an extremely disorganized person’s “My Documents” directory, with whatever someone thought seemed important that morning placed wherever they thought sounded right. That morning.