The Myth of Data-Driven Decisions

Long ago, when I worked as a webmaster for CDC, I tired of noisy arguments about the website. So, I turned to data. If I could support a decision with data, I thought, then I could avoid most arguments. I dreamed of a quieter world where data would tell me how to design and what content to post. I wouldn’t have to think so hard. And, if anyone asked why I made a decision, I could point to the data. Brilliant.

Then, a funny thing happened. I had to think even harder to apply the data to decisions. And, the arguments did not end. If anything, they got deeper and louder. And, I thought “Oh sh#$!”

I look back on that experience as a cold splash of water that woke me into putting data into a different perspective. Today, interactive professionals and executives face more design and content decisions. How does data fit into the equation now?

Myth: Data = Good Decision

I think there’s a myth that data alone will save us from making bad decisions. This myth has had at least two incarnations. In 1999, academia and the U.S. federal government started to compile data-based best practices. As more data became available, the best practices were updated, and it’s still around today as www.usability.gov.* That effort grew out of a desire to make designing federal websites more efficient. Although never formally stated in print, what everyone involved meant is we could make decisions quicker by ending the arguments with data. But, as my experience shows, the arguments did not end. They just grew deeper and louder. Ultimately, someone still had to decide and be accountable for it.

This view of data as our savior returned in around 2006 as the answer to handling HiPPOs, or the highest paid person’s opinion.** I agree that a decision made because an uninformed executive said so is bad. But the opposite extreme—because the data said so—is just as bad. Why? Many people misunderstand this sentiment as the data can make decisions for us. A mindless decision is often a bad decision. As Jared Cole of Adaptive Path notes,

The problems of business can rarely be solved with an algorithm or simple equation, but more often require the consideration of many probable outcomes and a number of moving variables, all viewed through the lens of multiple human perspectives. (Adaptive Path, June 2009 Newsletter)

The bottom line is we humans still have to and always will have to think about how design and content decisions solve business problems. The ideal experience for an end user might be “don’t make me think,” but a decision-maker in this industry absolutely must think.

Reality: The Right Data + The Right People = Good Decision

The question, then, is how do we best arrive at these decisions? I see no other route than a combination of the right data and people.

The right data is high quality, which requires a data expert understanding where the data comes from and its limitations. For example, if you have Omniture analytics implemented on your main site but not your applications, you have a major limitation that requires careful interpretation.  The right data also is relevant to the decision. For instance, if you’re deciding how to increase shopping cart conversions, looking at metrics about when and how people abandon the cart is relevant.

The right people bring a combination of expertise–business, data, conte. Is expert opinion valid? Yes—if the people truly are experts. True experts have at least 10,000 hours of experience and deeply understand the context of the decision. They can synthesize the data and the context into the right decision. They quickly know when a decision can be made immediately and when it requires more investigation. And, sometimes, experts have to work through how their areas of expertise affect each other. (For more about expert decisions, check out Outliers and How We Decide.)

We often confuse our data experts with other experts. A few months ago, I attended a session at Web 2.0 Expo about analytics for social media. The presenter, a data expert, offered a very useful report of convincing metrics for social media for a software company, an electronics retailer, and other businesses. During the Q&A, however, attendees asked the presenter how they should implement social media and what techniques work best. Why ask the data expert? The data expert can’t recommend a social media, content, or any strategy other than a data strategy.

Let’s say you go to the doctor, and she orders you a blood test. Your blood sample is sent to a lab, where a blood expert analyzes your sample and reports your cholesterol levels. Your doctor tells you that you have high cholesterol. To lower your cholesterol, do you ask the doctor or the blood expert who reported the results for a strategy? I’m betting on the doctor.

Toward Data-Informed Decisions

The future of digital business is bright, and it’s growing more complex. We face hard design and content decisions each day. We need to know how to best make them. Data is an asset that the interactive industry offers to help. But, as tempting as it sounds, data doesn’t automate those decisions. Instead, data needs to inform true experts—our industry’s other asset—in making those decisions efficiently. Data will never replace human judgment.

To that end, I think the interactive industry should act less like data-driven production teams and more like data-informed experts who partner with clients to make the best decisions. We can better identify the right experts for the right decisions, too. If we don’t, our assets turn into liabilities. We risk turning our work into a commodity, making promises we can’t fulfill, and getting poor results for our clients or businesses.

Then, we’ll all be saying “Oh sh#$!”

* Usability.gov is a handy resource. My point is just that its original, informal goal was not possible to reach.

** Kaushik’s explanation is not this simplistic, but, in my experience, many people’s interpretation is.