Science is the belief in the ignorance of experts.
Recently, I was criticized by a vendor’s sale rep after I posted a review from Network World on the performance of various next-generation firewalls on Twitter and Linkedin. When I posted it, I made the comment, “Hmmm, would love to see the testing parameters.” I was trying to make a point that the article announced some results, but without making any supporting data available. When confronted by him, I was told that if I had emailed him and asked for the data, he would have sent it to me. He also mentioned that NSS labs had tested his product as well. Unfortunately, that report is only available with a subscription (the dreaded paywall), so I can’t confirm how detailed it is with regards to the various test scenarios utilized. The whole experience made me feel as if I had asked McDonald’s for the recipe of their Secret Sauce. I immediately contacted an engineer I trust immensely, because he’s the ultimate skeptic. If you tell this guy it’s raining, he won’t check the news, he’ll send up his own weather balloon. Whenever I get starry-eyed about the latest technology, he’s usually the one to bring me down to earth. He was flabbergasted by the situation I related to him and I asked, “Isn’t it part of an engineer’s job to question and judge results according to a scientific method?”
In my role as an engineer, I’m paid to evaluate a problem and propose various solutions after thoroughly considering the cause. I inquire, design, test, validate, then redesign and test as needed. This process is loosely based upon the Scientific Method, a set of thousand year-old techniques used to investigate and understand our world. It separates us from the general population who believes in Boogie Men, magicians and miracles. I admit that I don’t always get it right and sometimes I have to scrap my hypothesis and start over again. But in using the framework of the Scientific Method, I can be held accountable.
I would call what often passes for analysis and research in IT to be little more than Cargo Cult Science. Techniques that seem to use the Scientific Method, but actually don’t. The concept was introduced by theoretical physicist and academic, Richard Feynman, in a famous commencement address at Caltech from 1974.
There is one feature I notice that is generally missing in “cargo cult science.” It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it; other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked — to make sure the other fellow can tell they have been eliminated. Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can — if you know anything at all wrong, or possibly wrong — to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. … And it’s this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.’
Does any of this sound familiar? Look, I’m not trying to say the work I do requires the same intellectual rigor of a theoretical physicist in the academic realm, (even though trying to find a spanning tree loop in your network can sometimes feel like attempting to prove the existence of the Higgs-Boson particle), but I do think engineers should subscribe to a methodology based in critical-thinking. If you’re a vendor, a journalist or an analyst and you publish results, be prepared for an engineer to ask how you got there. We need to see it for ourselves, by recreating the scenario in a lab and adding the variables applicable to our real-world networks. And if (and when) we challenge you, don’t tell us that our “mileage may vary,” be prepared to prove it.
Before you get the idea that I’m anti-Gartner or opposed to any other company that performs analyses, I’d like to emphasize that I’m not. However, I think people have misunderstood and misapplied this research. So I did some fact-finding on the Garner site and found the following documentation, Magic Quadrants and MarketScopes: How Gartner Evaluates Vendors Within a Market. On the first page it states the following:
Magic Quadrants and MarketScopes offer visual snapshots of a market’s direction, maturity and participants. Understanding our research methodology will help you use these models effectively when choosing a product or service, or managing a vendor relationship.
I don’t interpret anything in that statement to indicate that the Magic Quadrant is anything more than a market analysis.
3.5 How to Use a Magic Quadrant
Your needs and circumstances should determine how you use the Magic Quadrant, not the other way around. To evaluate vendors in the Leaders quadrant only and ignore those in other quadrants is risky and thus discouraged. For example, a vendor in the Niche Players quadrant could offer functions that are ideally suited to your needs. Similarly, a leader may not offer functions that meet your requirements — for example, its offerings may cost more than competitors’, or it may not support your region or industry. Use a Magic Quadrant to narrow your list of choices, but don’t base your decision only on the model. Talk to the Gartner analyst who created the research for more details and insight.
To me, this says “here’s our opinion, but take it with a grain of salt.” Then why does the industry quote these reports like Holy Scripture and reference them as if they were the result of meticulous scientific research? Pseudo-science is successful for one reason: intellectual laziness. Genuine analysis is undermined by attempts to shortcut the necessary process of investigation and I find many engineers to be the worst offenders. Sometimes we need to stop and evaluate our motives, before rushing to implementation. We need to be diligent in the application of scientific methods or we’ll lose our credibility and integrity.
“Science is a way of trying not to fool yourself. The first principle is that you must not fool yourself, and you are the easiest person to fool. Science alone of all the subjects contains within itself the lesson of the danger of belief in the infallibility of the greatest teachers in the preceding generation…” Richard Feynman
About Mrs. Y
