I’m not your typical blogger, at least not compared to others on Packet Pushers. I’m not CCIE certified, I’m not (currently) a network engineer, and I can’t extoll all of the virtues of NHRP in NBMA environments. I am one of the often loathed and dreaded C-level executives who more often than not tie unrealistic budgets, timeframes, and expectations to projects and then request those pesky project status reports at the most inopportune times when you really just need to focus on doing your job.
Okay, now that the eye rolling is over, I’m really here because I have an agenda, but please hear me out. 1) I want to introduce you to the industry I work in (power and utilities). We can really use your help and expertise. More on this in subsequent posts (as I write this, Network World just published an article saying that Laura Ipsen left Cisco for Microsoft, so now the need is only amplified). 2) I feel guilty. Yup, I feel bad. How’s that for an agenda? Let me explain why I feel guilty and what should be considered as a part of the next evolution of our industry. Change is good folks.
For too many years, my organizations (….I) have reaped the rewards of having many skilled single, double, and triple CCIE’s (this also doesn’t include all of the Microsoft engineers, SAN engineers, VMware engineers, etc.) along with vendor support to help architect, design, and deploy a fantastic environment – something that I am very proud of, but I recognize that I can’t take the credit for it. Did we follow a design guide to do this? No. Did we have white papers showing us how to design and build an integrated, multi-vendor infrastructure that checked all of the business requirement, management, monitoring, cost, and administration check boxes? No. Did we leverage the expertise and experience from many engineers? Yes. Was there a bit of trial and error involved? Of course. Regardless if you find yourself building a new infrastructure from the ground up or inheriting someone else’s vision, there are a few key facts that always remain constant. 1) You’re here to support the needs of the business and thus, so is the infrastructure. 2) Technology always changes. 3) Whatever you think you know, you don’t know everything. 4) There’s always room for improvement.
So what am I doing here? Let me back up to give you a little context. I have spent a good portion of my technical career learning from those who are smarter, more experienced, and more knowledgeable than me. I consider this to have accounted for 25% of my training. More significantly (50% or so) I found that I learned a great deal more from managers who had had less than desirable traits (to be fair, they also were balanced with some positive traits as well), projects that failed, systems that blew up, users whose expectations I did not meet, etc. In other words, I found that I learned more from failures and the mistakes of others than projects that were successful. I think this makes sense after all. It’s how we naturally like to learn.
What about that final 25%? This is an interesting one. When I look back at all of the organizations I’ve worked for, I’ve lived through many different regimes and extremes. The first extreme was a very…frugal environment. We learned survival and how to make things work with very little to no budget. From taking three broken ThinkPad T20’s and disassembling them to make one working unit, to scouring eBay and Craigslist for spare server parts because we had no support agreements. Yep, been there done that. For the other extreme, I’ve also been in environments where, I wouldn’t say it’s raining money, but instead, we actually had a budget. We could spend money on equipment that the enterprise required. We could actually afford (and benefit) from being able to stay on the latest versions of software (what a novel concept!). We saw the value in hiring consultants who were the experts in their field and could bring their experience to our enterprise. Having a budget really does force you to think differently, perhaps say bring in that consultant to design and configure QoS on that new MPLS network of yours instead of trying to do it yourself…
Now, getting back to you and me. I’m not here to get on a soap box and spew useless jibber jabber about value creation, IRR of a project, or dissect why our multiple site DMVPN architecture does seem to suffer from sporadic loss of EIGRP peering due to spikes in multicast traffic (the TAC security team puts together another great podcast if you haven’t already listened). I’d like to offer you my perspective on this great industry of ours, how much it has changed and how much more I personally feel that it needs to change for the collective good of us all. Collectively improve our industry? What on earth am I talking about? You have to give a little to get a little; it’s karma, man.
I think the networking industry has changed. Routers, switches, firewalls; these are all commodity items. If you work for a Cisco partner, are your margins the same as they were 4 to 5 years ago? If your answer is yes, wow, you’ve got something special going because it’s still raining money. If it’s no, you’re not alone. Are you getting squeezed on your margins for hardware? Are you trying to make up for it in professional services? Are you struggling to convince that school district that they really, REALLY need to replace that 10 year old network and PBX? Is that school district putting the project out to bid and making the partners fight over the last half point in margin? Seriously? Is this any way to live?
I don’t work for a partner. I don’t work for Cisco. I have worked with enough partners and with Cisco long enough to know how the game is played. If your business plan is to make all of your revenue on hardware, then you had better be willing to compete w/ the box pushers out there and push some major quantity of boxes. This sure doesn’t sound like much fun for an engineer who toiled away getting their CCIE. You want to be able to design a solution. You want to get your hands dirty and deploy the latest and greatest technology. Hence the issue. That school district doesn’t care. Some might, but I would venture to guess that most are again, looking for the most “cost effective” solution they can find. If they are severely budget constrained, they’ll probably try and tap some internal resource to finish the project. It’s certainly not their fault, but at the end of the day, budgets are budgets and if there isn’t money to spend but there is a business need, someone will find a way to make it work.
Now we’re really compounding the issue. You have hardware, who is going to configure it? Configurations shouldn’t be a tightly held secret. Pay to play? Sure, there’s some of that, I get that. We all have to eat. But really, are you going to make your quota this month by selling that PS on a single firewall configuration?
I believe great, best of breed, well integrated, well thought out configurations and designs should be free. (Cisco tries to do this with their white papers, but to varying degrees of success and these are focused on Cisco products and technologies with little to no applicable comprehensive business or multi-vendor strategy.) I don’t think any one of us has the best answer, but collectively we do.
Given the extreme flexibility and the vast array of components to consider when managing a complex “converged” IP network, it really DOES take a village to do it right, not a one man/woman wrecking crew. I don’t think consultants should feel threatened. My friends who are consultants – my wife as well who is also a consultant – don’t necessarily agree with me but that makes for a stronger argument. If a consultant’s only job security was to configure an ASA 5505 to gain that 4 hours of billable work, then I don’t think of them much as consultants. There is no secret sauce to a configuration.
Customers will pay for a consultant regardless if you can “download” a best practice configuration template because they know the benefit of what they are getting with a consultant (things break and don’t always go as planned). The individual who does not have budget and has never configured a firewall before is of course going to use the old “let me Google” for my configuration trial and error method. Which do you think is better? What happens when that network administrator at that school district tries to configure their firewall and does it incorrectly, thus allowing a hacker to break into their network and steal the SSN’s of all of the students and teachers? Is that really benefiting anyone? None of us want to see that happen. That just gives our industry another black eye. Look at all of the great technical content that is available to you on this site alone. While the nuggets of content are great and meaningful, is it enough to put together that firewall configuration so it not only secures the network but takes into consideration things such as an Information Security policy that should define important parameters such as encryption standards, password standards, and acceptable use policies? What about setting up syslog, SNMP, NTP, or AAA? Is there a policy to command line level log all commands? What about change control or periodic reviews of firewall configurations? Did Google tell the administrator to do all of these things?
For quite some time now, I have felt that as a collective industry we need to all work together to build better, constantly improving best practice integrated designs that encompass all major aspects of our profession. I know it sounds like a lot and it’s a huge topic, but I think it’s possible. If you’re familiar with ITIL or COBIT, it’s a similar concept but in my opinion much more “applicable” to actual hands on technology. Instead of talking about the best way to manage IT as a service, let’s find the best way to configure that firewall/switch/router; create a plan to manage information security; plan and deploy an IPv4 (or even IPv6) network; perform comprehensive monitoring of all of your devices; capture, analyze, and trend log files; manage passwords; trend for performance; design an infrastructure capable of meeting expectations of a disaster recovery plan – etc.
As an example, Podcast #87 was a very interesting discussion about SIP. I believe the question was raised, when is it appropriate to use a centralized, distributed, or hybrid model for your “UC” environment? Is there one right answer? No, but given the various business requirements you can probably design a best practice, fully integrated solution that meets the majority of needs. Everyone on the show had great experiences from the various customers they support and thus were able to start forming conclusions and what I interpreted as preliminary design recommendations. Let’s take this example a bit further and look at a high level what else might be considered from a Cisco Unified Communications perspective:
Identify business requirements:
- Do you need to support a customer service center or even a call center?
- Do the majority of the users use the phone to conduct business?
- Do they call other parties that are in different time zones or countries?
- Do you have multiple offices to support?
- Do the employees travel?
- What is the pain threshold for system availability?
- Are you also trying to manage your mobile phone costs?
- What about video conferencing?
- Do you have a defined and approved capex budget?
- What is your opex budget?
- Etc.
Identify engineering requirements, product restrictions and other usage and integration requirements:
- Is there a legacy PBX it integrate with?
- Can your network infrastructure support VoIP, PoE, etc.?
- Are there voicemail integration requirements?
- What about conference calling integration requirements?
- Is there an existing dial plan requirement?
- Are there other applications that will be integrated into this system?
- Do you have resources who can manage this new system?
- Etc.
And the list goes on and on…
Let’s think of this in technical terms. Even a simple decision such as an MGCP versus H.323 controlled voice gateway should be evaluated. Certainly there is the configuration aspects of MGCP compared to H.323 as an example. But what if one of the requirements was high system availability? With H.323 and call preservation, you might be able to lessen the impact for calls to the PSTN with any WAN outages for that site.
What about SRST versus CME? I remember the debates from 5 years ago. Do you use CME instead of SRST for your spoke sites? Certainly there are more features (or at least there were more features 5 years ago), but what about the administrative costs trying to manage call manager in a centralized model but then having to update CME sites for every add move and change?
Certainly the SIP discussion concerning potential opex cost savings is huge. How do these translate to the business requirements? Are you replacing TDM or 1MB circuits? Do you have E911 restrictions? Does the state you operate in (Maine comes to mind) have specific PUC restrictions in terms of what you can do with the E911 database? Does that pose an issue if you have a campus environment?
Again, the majority of these questions can be answered based upon the requirements of a business and the available technology at any given point in time. A living, breathing design document that incorporates these business requirements that translate into technical specifications and configurations could be hugely important to our industry to promote best practices.
Many of you might feel that what I propose is impossible or impractical and you might be right, but again, as I said in the beginning, I do feel bad and you never really know if something is possible until you try, right?
Cheers,
Keske
About Keske
