Lies, Damned Lies, and Statistics
With all due apologies to Benjamin Disraeli, who uttered that phrase before Mark Twain popularized it, I have spent a good part of the last two days working on the analysis of traffic statistics from a new site one of our clients launched last month. In the course of trying to come up with meaningful comparisons and analysis, I used one of the words in the headline more than once.
I love statistics. My old team at Aurora used to call me the “stats hound.” But you have to be careful. The humorist Des McHale once wrote that the average human has one breast and one testicle. The author W.I.E. Gates similarly noted the man who drowned crossing a stream with an average depth of six inches.
It gets doubly confusing when you switch analytics packages at the same time you launch a new site. I’ve switched between WebTrends and NetTracker and saw a nearly 20 percent difference in such basics as site visits. Explaining that to leadership was an interesting challenge.
Switch from one of those mainstream packages to Google Analytics, however, and you might hear Disraeli’s quote tossed back into your face. Why? Google claims to screen out all bot traffic. Talking to one CMS vendor, they tell me when they switch clients, they typically see only 20-30 percent of the traffic the client was used to reporting. I’m seeing the same thing with our client.
That makes apples-to-apples comparisons about impossible. It also puts all kind of egg on the face of marketers or IS folks who had been bragging about apparently inflated traffic numbers.
There’s more power in analysis like conversion rate, percent of visits to the jobs section that result in an application, percent of visits to the site that include a physician search, etc. And even more power still when you engage in meaningful benchmarking so you have something to compare to. But even those numbers get skewed when the base figure of visits drops by 75 percent!
I believe the goal is to report accurate results, and those don’t include spiders and bots. Nevertheless, like a famous redhead was once told, “somebody’s got some ‘splaining to do!”