An Open Letter to the Packet Pushers Community

I’m not your typical blogger, at least not compared to others on Packet Pushers. I’m not CCIE certified, I’m not (currently) a network engineer, and I can’t extoll all of the virtues of NHRP in NBMA environments. I am one of the often loathed and dreaded C-level executives who more often than not tie unrealistic budgets, timeframes, and expectations to projects and then request those pesky project status reports at the most inopportune times when you really just need to focus on doing your job.

Okay, now that the eye rolling is over, I’m really here because I have an agenda, but please hear me out. 1) I want to introduce you to the industry I work in (power and utilities). We can really use your help and expertise. More on this in subsequent posts (as I write this, Network World just published an article saying that Laura Ipsen left Cisco for Microsoft, so now the need is only amplified). 2) I feel guilty. Yup, I feel bad. How’s that for an agenda? Let me explain why I feel guilty and what should be considered as a part of the next evolution of our industry. Change is good folks.

For too many years, my organizations (….I) have reaped the rewards of having many skilled single, double, and triple CCIE’s (this also doesn’t include all of the Microsoft engineers, SAN engineers, VMware engineers, etc.) along with vendor support to help architect, design, and deploy a fantastic environment – something that I am very proud of, but I recognize that I can’t take the credit for it. Did we follow a design guide to do this? No. Did we have white papers showing us how to design and build an integrated, multi-vendor infrastructure that checked all of the business requirement, management, monitoring, cost, and administration check boxes? No. Did we leverage the expertise and experience from many engineers? Yes. Was there a bit of trial and error involved? Of course. Regardless if you find yourself building a new infrastructure from the ground up or inheriting someone else’s vision, there are a few key facts that always remain constant. 1) You’re here to support the needs of the business and thus, so is the infrastructure. 2) Technology always changes. 3) Whatever you think you know, you don’t know everything. 4) There’s always room for improvement.

So what am I doing here? Let me back up to give you a little context. I have spent a good portion of my technical career learning from those who are smarter, more experienced, and more knowledgeable than me. I consider this to have accounted for 25% of my training. More significantly (50% or so) I found that I learned a great deal more from managers who had had less than desirable traits (to be fair, they also were balanced with some positive traits as well), projects that failed, systems that blew up, users whose expectations I did not meet, etc. In other words, I found that I learned more from failures and the mistakes of others than projects that were successful. I think this makes sense after all. It’s how we naturally like to learn.

What about that final 25%? This is an interesting one. When I look back at all of the organizations I’ve worked for, I’ve lived through many different regimes and extremes. The first extreme was a very…frugal environment. We learned survival and how to make things work with very little to no budget. From taking three broken ThinkPad T20’s and disassembling them to make one working unit, to scouring eBay and Craigslist for spare server parts because we had no support agreements. Yep, been there done that. For the other extreme, I’ve also been in environments where, I wouldn’t say it’s raining money, but instead, we actually had a budget. We could spend money on equipment that the enterprise required. We could actually afford (and benefit) from being able to stay on the latest versions of software (what a novel concept!). We saw the value in hiring consultants who were the experts in their field and could bring their experience to our enterprise. Having a budget really does force you to think differently, perhaps say bring in that consultant to design and configure QoS on that new MPLS network of yours instead of trying to do it yourself…

Now, getting back to you and me. I’m not here to get on a soap box and spew useless jibber jabber about value creation, IRR of a project, or dissect why our multiple site DMVPN architecture does seem to suffer from sporadic loss of EIGRP peering due to spikes in multicast traffic (the TAC security team puts together another great podcast if you haven’t already listened). I’d like to offer you my perspective on this great industry of ours, how much it has changed and how much more I personally feel that it needs to change for the collective good of us all. Collectively improve our industry? What on earth am I talking about? You have to give a little to get a little; it’s karma, man.

I think the networking industry has changed. Routers, switches, firewalls; these are all commodity items. If you work for a Cisco partner, are your margins the same as they were 4 to 5 years ago? If your answer is yes, wow, you’ve got something special going because it’s still raining money. If it’s no, you’re not alone. Are you getting squeezed on your margins for hardware? Are you trying to make up for it in professional services? Are you struggling to convince that school district that they really, REALLY need to replace that 10 year old network and PBX? Is that school district putting the project out to bid and making the partners fight over the last half point in margin? Seriously? Is this any way to live?

I don’t work for a partner. I don’t work for Cisco. I have worked with enough partners and with Cisco long enough to know how the game is played. If your business plan is to make all of your revenue on hardware, then you had better be willing to compete w/ the box pushers out there and push some major quantity of boxes. This sure doesn’t sound like much fun for an engineer who toiled away getting their CCIE. You want to be able to design a solution. You want to get your hands dirty and deploy the latest and greatest technology. Hence the issue. That school district doesn’t care. Some might, but I would venture to guess that most are again, looking for the most “cost effective” solution they can find. If they are severely budget constrained, they’ll probably try and tap some internal resource to finish the project. It’s certainly not their fault, but at the end of the day, budgets are budgets and if there isn’t money to spend but there is a business need, someone will find a way to make it work.

Now we’re really compounding the issue. You have hardware, who is going to configure it? Configurations shouldn’t be a tightly held secret. Pay to play? Sure, there’s some of that, I get that. We all have to eat. But really, are you going to make your quota this month by selling that PS on a single firewall configuration?

I believe great, best of breed, well integrated, well thought out configurations and designs should be free. (Cisco tries to do this with their white papers, but to varying degrees of success and these are focused on Cisco products and technologies with little to no applicable comprehensive business or multi-vendor strategy.) I don’t think any one of us has the best answer, but collectively we do.

Given the extreme flexibility and the vast array of components to consider when managing a complex “converged” IP network, it really DOES take a village to do it right, not a one man/woman wrecking crew. I don’t think consultants should feel threatened. My friends who are consultants – my wife as well who is also a consultant – don’t necessarily agree with me but that makes for a stronger argument. If a consultant’s only job security was to configure an ASA 5505 to gain that 4 hours of billable work, then I don’t think of them much as consultants. There is no secret sauce to a configuration.

Customers will pay for a consultant regardless if you can “download” a best practice configuration template because they know the benefit of what they are getting with a consultant (things break and don’t always go as planned). The individual who does not have budget and has never configured a firewall before is of course going to use the old “let me Google” for my configuration trial and error method. Which do you think is better? What happens when that network administrator at that school district tries to configure their firewall and does it incorrectly, thus allowing a hacker to break into their network and steal the SSN’s of all of the students and teachers? Is that really benefiting anyone? None of us want to see that happen. That just gives our industry another black eye. Look at all of the great technical content that is available to you on this site alone. While the nuggets of content are great and meaningful, is it enough to put together that firewall configuration so it not only secures the network but takes into consideration things such as an Information Security policy that should define important parameters such as encryption standards, password standards, and acceptable use policies? What about setting up syslog, SNMP, NTP, or AAA? Is there a policy to command line level log all commands? What about change control or periodic reviews of firewall configurations? Did Google tell the administrator to do all of these things?

For quite some time now, I have felt that as a collective industry we need to all work together to build better, constantly improving best practice integrated designs that encompass all major aspects of our profession. I know it sounds like a lot and it’s a huge topic, but I think it’s possible. If you’re familiar with ITIL or COBIT, it’s a similar concept but in my opinion much more “applicable” to actual hands on technology. Instead of talking about the best way to manage IT as a service, let’s find the best way to configure that firewall/switch/router; create a plan to manage information security; plan and deploy an IPv4 (or even IPv6) network; perform comprehensive monitoring of all of your devices; capture, analyze, and trend log files; manage passwords; trend for performance; design an infrastructure capable of meeting expectations of a disaster recovery plan – etc.

As an example, Podcast #87 was a very interesting discussion about SIP. I believe the question was raised, when is it appropriate to use a centralized, distributed, or hybrid model for your “UC” environment? Is there one right answer? No, but given the various business requirements you can probably design a best practice, fully integrated solution that meets the majority of needs. Everyone on the show had great experiences from the various customers they support and thus were able to start forming conclusions and what I interpreted as preliminary design recommendations. Let’s take this example a bit further and look at a high level what else might be considered from a Cisco Unified Communications perspective:

Identify business requirements:

  • Do you need to support a customer service center or even a call center?
  • Do the majority of the users use the phone to conduct business?
  • Do they call other parties that are in different time zones or countries?
  • Do you have multiple offices to support?
  • Do the employees travel?
  • What is the pain threshold for system availability?
  • Are you also trying to manage your mobile phone costs?
  • What about video conferencing?
  • Do you have a defined and approved capex budget?
  • What is your opex budget?
  • Etc.

Identify engineering requirements, product restrictions and other usage and integration requirements:

  • Is there a legacy PBX it integrate with?
  • Can your network infrastructure support VoIP, PoE, etc.?
  • Are there voicemail integration requirements?
  • What about conference calling integration requirements?
  • Is there an existing dial plan requirement?
  • Are there other applications that will be integrated into this system?
  • Do you have resources who can manage this new system?
  • Etc.

And the list goes on and on…

Let’s think of this in technical terms. Even a simple decision such as an MGCP versus H.323 controlled voice gateway should be evaluated. Certainly there is the configuration aspects of MGCP compared to H.323 as an example. But what if one of the requirements was high system availability? With H.323 and call preservation, you might be able to lessen the impact for calls to the PSTN with any WAN outages for that site.

What about SRST versus CME? I remember the debates from 5 years ago. Do you use CME instead of SRST for your spoke sites? Certainly there are more features (or at least there were more features 5 years ago), but what about the administrative costs trying to manage call manager in a centralized model but then having to update CME sites for every add move and change?

Certainly the SIP discussion concerning potential opex cost savings is huge. How do these translate to the business requirements? Are you replacing TDM or 1MB circuits? Do you have E911 restrictions? Does the state you operate in (Maine comes to mind) have specific PUC restrictions in terms of what you can do with the E911 database? Does that pose an issue if you have a campus environment?

Again, the majority of these questions can be answered based upon the requirements of a business and the available technology at any given point in time. A living, breathing design document that incorporates these business requirements that translate into technical specifications and configurations could be hugely important to our industry to promote best practices.

Many of you might feel that what I propose is impossible or impractical and you might be right, but again, as I said in the beginning, I do feel bad and you never really know if something is possible until you try, right?

Cheers,

Keske

Keske

Keske

Keske is an IT professional who works in the renewable energy space. With more than a decade of experience in the IT industry, Keske has worked in various roles, disciplines, and management; with networking always being a focused passion. Keske now leads an IT, SCADA, and Operations center that tries to marry best practice design and implementation to business needs and requirements.
Keske

Latest posts by Keske (see all)

  • http://twitter.com/brandonrbennett Brandon Bennett

    Although a very good idea I think the majority of the problem is one that you hit on: Everyone’s business needs/drivers are going to be different.  We can get in the weeds about CME vs SRST or e911 considerations but doing a “choose your own adventure” type design document utilizing nothing but community support is going to come up a bit short.

    I roll my eye’s when anyone says “Best Practices”.  Best Practices usually just means common denominator.  All of us need NTP, all of us need syslog (although some smaller shops may not know it yet), but none of it is tailored to a business need.  There is some common ground between the needs of Company A and Company B but the real design importance is not the similarities but the differences.  Designing to a unique business need means that a human, and not a all-in-wonder document is needed.

    But it does bring up a good point.  You could drive the design documentation more on the issues and and the fixes around common issues, but directly tying business goals to technology SHOULD be unique and is why you have hired your wonderful team to perform.

    • Keske

      Brandon,

      I felt the same way trying to adopt ITIL.  I remember saying, wait, this isn’t the right fit for us.  This doesn’t work with how we operate..  Of course best practices should just be taken as guidelines and it still takes an engineer to interpret these guidelines and put them into service.  I suppose I have met too many junior/budding engineers who did not even have this to help them get started.  I’m sure you’ve all seen the horror stories, things as simple as selecting an IPv4 addressing scheme.. I recall one example where they knew enough to pick something from the RFC1918 space… But ended up w/ 192.168.0.0, using the third octet as the “physical” address of the building.. The first office with an address of 29335 13th street sort of broke that model.  Speaking of NTP, that same engineer didn’t understand why it was important to configure NTP or syslog for that matter.  As you can imagine, there was a issue, they went to check the logs and the fun started.  

      I will say that I’ve read Cisco’s white paper on NTP and while it’s a bit dry and doesn’t actually go into great detail why you should be a stickler for NTP, it is helpful and gets the job done.

      On that last point, absolutely guilty as charged.  I have ventured off and used Juniper, Enterasys, Ruggedcom, Garrettcom, GE, Moxa, Sonicwall, Avaya, Nortel, Adtran, etc. – none as extensively as Cisco.  

      Shoot, all cards on the table here.  I’ve also given a Cisco webinar on substation automation and physical security.. I’ve also spoken on behalf of Cisco at Distributech (utility trade) talking about substation security.  Cisco and Microsoft do play large rolls in my career and I’m not really ashamed of it.  I have my reasons (established account team, VAR’s, smartnet contract consolidation, platform uniformity (I could debate against this one as well), etc.) for using their products. Actually, it was a talking point in one of the previous presentations I mention above, why I chose Cisco over brand X.  Perhaps I’ll write a post.  

      So yeah, I’m a fan boy, but if you ask my Cisco account team and BU’s that I work with, I don’t pull any punches – they get my honest feedback.  I have certainly worked with my fair share of rotten, stinking Cisco products (AGM blades, Cisco Works, Cisco Conference Connection,etc) and been on the bleeding edge of their product roll-outs (CCM 3.1 that lead into a CAP case, version 1.0 software installations where the Cisco BU guys did the actual server installation, etc..)..  They’re not the best, but they’re the common platform and language.  

      I definitely agree, if you have a business need to fill, don’t start out thinking Cisco or Brand X is the solution, but do consider all of the factors including administration, features, integration, support, financing, etc.  

      I never liked the saying, “you can never be fired for buying Cisco” but there is an amount of truth to it.

  • Fernando Montenegro

    Keske,

    I would venture that the knowledge to extract business & technical requirements from a situation and then apply considerable domain and product knowledge – often based on experience – to create a design that fits those requirements is a key part of the competitive advantage of any service business. I am extremely grateful for all the knowledge shared by the community on Packet Pushers and elsewhere, but it is clear to me that if I wanted something more detailed, I should pay up for it.

    Coming from the security side of the things, I see a lot of very worthwhile content already available for free, including vendor hardening guidelines, SANS, OWASP and others, to name a few. Understanding HOW to apply that content to a real-life business scenario is where a paid professional comes in.

    Even that school district you mentioned in your examples, if it is not able to do things ‘right’, it us up to the district to recognize that and maybe pool their resources with other districts to have a common shared architecture that can then leverage a more senior design resource. And you’re right, it DOES take a village: server folks, network people, DBAs, developers, business analysts, security officers, storage experts, etc… But the end result from bringing all those folks togethers will be the best one can afford for *their* situation.

    As always, there’s differences of opinion, but if I want to respect karma, I’ll share what content I can, be grateful for the generosity of others and, when necessary, open my wallet or cut that purchase order…

    • Keske

      Fernando,

      You’re absolutely right, especially when it comes to security.  There is a lot of great content out there.  In my opinion, the best security tool available is user awareness training, yet most organizations that I see (again I’m not a consultant, so I don’t necessarily have the breadth of others – but I will say that my wife is a security consultant for fortune 500 companies and her experience mimics mine) do not follow this simple practice.  Knowledge is everything… SANS has great guidelines but I don’t know that it goes far enough to help a budding security administrator or officer put together a comprehensive policy that would include adequate training material, training schedules and policies, etc.

      The next post I’m putting together covers security for the industry I’m in.  I’m afraid it’s even more backwards for a lot of utilities but there is a lot of positive movement forward.

  • Romans

    Keske, 
    the idea you are talking about sounds like utopia…. sorry if I’m breaking your heart, but there is no “one-size-fits-all” solution.Google (oh my, I say Google, mean Internet!) is full of solutions and code snippets for most of standard solutions. Even more, nowadays most boxes comes with “safe to run” configuration, like deny all as the last rule in the firewall, as an example. Microsoft for example provides best practices analyzers for their products.If the IT administrator is incompetent however, then these googled solutions/best practices are useless for him, because you have to learn first how to use the tool/technology you have.It says: configure NTP, SNMP, etc. How is he supposed to do so if these abbreviations  are only random letters to him?The best approach here, to my mind – is outsourcing. Outsource IT in that school (yes it costs less or the same than having own incompetent IT admin) and let them do their business – teach our kids.And in case of outsourcing you are not paying for lines of code, but for the service and responsibility so now it is up to school’s lawyer to create a good responsible contract with IT company. The problem, however I see is different – incompetent management, which does not know how to run company, that is why it has cost ineffective IT solutions, outdated (if any) DRP (disaster recovery plan), inexperienced IT etc, etc… 
    Best Regards,
    Romans