Should it annoy me that I find pertinent articles (at least if I twist them a bit) on the Harvard Business blog site? In this case, an article about how a human phenomenon can wreck decisions made from web analytics, weather forecasts or predicting the next American Idol.
Freek Vermeulen, Associate Professor of Strategic & International Management at the London Business School tells in a recent blog a wonderful and very useful story about selective metrics.
“During World War II, American military personnel noticed that some parts of planes were hit by enemy fire more often than other parts. They analyzed the bullet holes in the returning planes and launched a program to have these areas reinforced so that they could withstand enemy fire better.”
They saw a pattern of bullet holes on the returning planes and decided to strengthen the planes at those points. What they did not see is the pattern on the planes that did not return. They did not have all the data before making a decision. “This is why we call it “selection bias”; we only see a selection of the outcomes, and therefore draw false conclusions. And the world of business is full of it,” claims Vermeulen, and states several cases.
Thank you, I’ll use my own examples from here. And they’re not all business.
When a business looks at a successful sale on a web site, they only see the sales, which comprise, for many sites, about 2 percent to 5 percent of the visitor sessions. They will then try and replicate this pattern, attempting to force other visitors to behave like those who have purchased. What the business ignores is why the other 95 percent to 98 percent of potential customers did not purchase. Through selection bias, they are systematically ignoring potential growth by focusing on the smaller but positive group, not addressing the opportunity, and drawing false conclusions about all their users.
Businesses must realize that people interact with content differently, with different goals (or sometimes no goals), and with different buying criteria. It would be helpful to analyze what content was needed, what was confusing or useful, as well as what price point was desired. The process to quantify this can take some time and may not be conclusive. Managers must accept these facts when seeking accurate guidance.
I have yet to see an analytics package that will quickly break down the majority of the users who did not achieve the goal of the site (for example, a sale) to an actionable result.
As the abilities of analytics continue to evolve, we will be able to identify individual personae (or behavior sets) and measure the success of treating groups of similar people in the same way. But that will still need to be done with a test group and a control group of similar users. It must be planned and documented to make sure selection bias doesn’t enter the picture.
Next generation analytics will also measure their behaviors on other sites, and perhaps even social media activity to deliver a more robust image of the buyer. This will determine the order and depth of content. For example, if you’ve been to several web sites looking at television technical specs, the third site, recognizing you’re technical, should bring the specs to you quickly.
The results: Content sites will keep users longer, social sites will keep visitors more engaged, and sales sites will appear more interesting. And yes, they will also watch for margin of errors and statistical bias, and deliver reports on emerging trends in the user community.
So what does this have to do with the weather?
When I worked morning drive at a private weather service we were prohibited from listening to any radio station within 500 miles. (Thank you AM radio skipping of the atmosphere with CKLW in Windsor, Ont. and WWWE in Cleveland!) Why so strict? We may hear a forecast that may result in a bias leading to an incorrect forecast. Today, there are a good dozen forecast models available to meteorologists, but each still have their own selective bias based on their programming. Morning runs of a model may be “wetter,” exaggerating the moisture a storm may get. Over the year, you learn the biases of each model and how to forecast around them.
How does this play with reality television? Glad you asked.
Google Trends, the measure of what people are searching on, can provide amazing insight. Whoever had the most buzz won American Idol.
Let’s see what people are searching on today (March 23, 2009). Pending any horrific performances on American Idol, Danny Gokey, Anoop Desai, Adam Lambert, and Kris Allen will be safe this week. The reason? They are burning up Google searches, as compared to the other competitors. Buzz is safety. Lil Rounds is falling from the top the quickest. However, I am only addressing who was on top – selective bias.
Michael Sarver is actually falling almost off the Google Trends table. Now let’s add another bias – reasons for voting for different people can change from week to week.
From bullets and buying to weather and fans, the idea of selective bias can create losers from companies who should be winning, and vice versa.