Why Are We Afraid of Data?

My friend, Gbenga Ajilore, is an economics professor. Last month, he gave a great talk at AlterConf in Chicago entitled, “How can open data help facilitate police reform?” It concisely explains how data helps us overcome anecdotal bias.

I was particularly struck by his point about how we need police buy-in for this data to be truly useful, and I was left with a bit of despair. Why is buy-in about the importance of data so hard? This should be common sense, right?

Clearly, it’s not. Earlier this year, I expressed some disbelief about how, in professional sports, where there are hundreds of millions of dollars riding on outcomes, there is still strong resistance to data and analytics.

On the one hand, it’s incredible that this is still an issue in professional sports, 14 years after Moneyball was first published and several championships were won by analytics-driven franchises (including two “cursed” franchises, the Boston Red Sox and the Chicago Cubs, both led by data nerd Theo Epstein).

On the other hand, it’s a vivid reminder of how hard habits and groupthink are to break, even in a field where the incentives to be smarter than everyone else come in the form of hundreds of millions of dollars. If it’s this hard to shift mindsets in professional sports, I don’t even want to imagine how long it might take in journalism. It’s definitely helping me recalibrate my perspective about the mindsets I’m trying to shift in my own field.

The first time I started to understand the many social forces that cause us to resist data was right after college, when I worked as an editor at a technology magazine. One of my most memorable meetings was with a vendor that made a tool that analyzed source code to surface bugs. All software developers know that debugging is incredibly hard and time-consuming. Their tool easily and automatically identified tons and tons of bugs, just by feeding it your source code.

“This is one of the best demos I’ve ever seen!” I exclaimed to the vendor reps. “Why isn’t everyone knocking on your door to buy this?”

The two glanced at each other, then shrugged their shoulders. “Actually,” one explained, “we are having a lot of trouble selling this. When people see this demo, they are horrified, because they realize how buggy their code is, and they don’t have the time or resources to fix it. They would rather that nobody know.”