I have been working as a Network Administrator/Network Engineer/Senior Network Engineer at a small, regional educational institution for the past 15 years. Over that time, the reliance on the network has increased to the point where it is indispensible to the operation of the organization, both on the administrative side and the educational delivery side.
When back in 1996 the old HP routers locked up for reasons passing all understanding, the old hands would wait until after lunch, saunter over to the main comms room and power cycle the gear. When the old mainframe went down, teaching still went on, the students in the library used the old card file instead of the electronic catalogue, and admin staff got on with their filing. As we all know, this is no longer the case. Servers come and go, but if the network goes down, the alerts go out and the phone starts ringing.
But another thing has happened in the place where I work. The place has become “client-focused”. (Never mind the philosphical problem I have with calling students “clients” or education a “product”; I’m just old-school that way.) All well and good, but in a paradoxical way, the policies of becoming client focused – at least where I worked – reduced the communication between the IT staff and the clients. The network staff, always “back-room” types, became one more step removed from the clients.
Let me explain what I mean. In the interest of providing clients with a “clear channel of communication”, it was decreed that the IT staff were no longer individually allowed to send out certain types of information. Such as information about outages, new services and the like. All information needed to be channelled through an official arbiter of communication. Needless to say, this turned into the IT censor, seemingly in the interest of covering the butt of those in charge. I understand the need for coherent messaging, especially in large organizations, but in this case it didn’t seem to fit.
For example, some years ago, I fat-fingered a VLAN add command, and look a fairly large building off-line. We weren’t a bank, and as “critical” as the network is, these sort of things were not part of a change management procedure, dictating out-of-hours work. It was a simple change; I screwed it up, it was resolved fairly quickly. So, after fixing up my error, the first thing I did was to fire off an email, apologizing and explaining what had happened to my boss, the help desk, and to the clients in the building affected. This achieved a couple of things. Firstly, it put people’s mind at rest as to the nature of the problem. Secondly, and more importantly, the clients understood that the issue was resolved. Finally, I got lots of emails back from the people in the building thanking me for the information. While some are never happy and will gripe no matter what you say, people generally understand that things go wrong sometimes and, in my experience, are appreciative when kept in the loop.
Contrast this with a similar occurrence where a colleague did a similar thing recently. Perhaps having to go through channels was difficult and time consuming, so nobody did it. Or, having to go through channels, it may have been deemed by someone in the chain as not important enough to notify the clients that something had happened. After all, if they are interested, they’ll check the (deficient and almost useless) status page. This did not occur. For whatever reason, the clients were not informed. In the cafeteria, I had people come up to me asking, “What happened to the network this morning? Is it fixed?” People were left in the dark as to the problem and whether it was resolved.
So why was an announcement not made? I can only guess that it was an exercise in denial. If the clients don’t know what we’re doing, they won’t know what we’re doing wrong. And the attitude does not extend beyond screw-ups, but to equipment failure, power loss and other causes of sub-optimal packet delivery. Unfortunately, this attitude of denial from the top can quickly begin to permeate down through the organization. A culture of denial (or at least of not informing others of changes made), which leaves poor suckers like me searching for faults when someone “accidentally” reboots their server or something else which manifests as a so-called “network problem”.
Now let me be clear, in some organizations, a big enough screw-up will get you fired. But as anyone who works in the government sector in Australia knows, it can be difficult, though not impossible, to get fired. In my 15 years, I didn’t see it happen once, despite some huge screw-ups. So in whose interest is this culture of denial? In whose interest is the denial of information to clients? True, those clients don’t know what I’m talking about when I say I left a keyword out of an IOS command which caused their VLANs to disappear. But they appreciate the information. They appreciate an apology, sincerely felt, if you stuff up. After all, everyone does it at some point in their life. If information must be sourced through a central location, then it needs to be transparent and honest. Not the vague, “IT is aware of a problem. We are working on it.” And the information must be sent; clients generally won’t seek out information on the status page after the event, especially when things seem back to normal.
A culture of openness and honesty benefits everyone in the end. And it does bring the back-room staff a little bit closer to the clients they are supposed to be ultimately focused on.