That's an Empirical Question!

That's an empirical question! I was walking down Church Street toward Coffman Union on the U of Minnesota campus, shooting the breeze and arguing with a fellow grad student when I first heard those words from Bill. Bill [William D. Wells] had recently been lured out of semi-retirement from the advertising agency DDB Needham and back into academia. I had the distinct pleasure of being assigned to him as a research assistant along with a couple other doctoral students. Bill was one of the smartest yet humble and professionally generous people I have ever worked with. Bill was practicing some of that professional generosity by taking us to lunch.

As we walked I and the other student were arguing/bantering about something in a friendly manner. Bill had been quiet for a few moments when he got this little smile and emphatically interrupted with "That is an empirical question!"

That one phrase became an anchor for three inter-related lessons that I learned from Bill over the next couple of years. These lessons are not those of standard deviations or regression equations, though Bill also taught me the nuts and bolts of quantitative analysis, rather they speak to mindset and attitude. Twenty years later these three lessons still ring true in my professional practice.

Lesson 1: Arguments Give Birth to Research Questions

Arguments can be a rich source of research questions and specific hypotheses. Arguments have the benefit of a built-in audience of stakeholders who seem to care. Bill was always asking the “So what?” and “Who cares?” questions about our joint research. Now, admittedly some arguments aren't worth the trouble and expense of research but others, particularly those of business line executives, often are.

It can be difficult to step back, pull yourself out of contestant mode, and really listen. Active, thoughtful listening is powerful and doubly so when it is paired with the training and mindset of a scientist. The data scientist with a background in research hears important questions and testable business hypotheses when others only hear passion, anecdote and opinion.

Learn to listen to arguments like a research scientist.

Lesson 2: The World Is Measurable

When Bill says "That is an empirical question" he firmly believed that nearly everything in life is measurable. Physical objects and processes are obviously measurable. Individual human actions, intentions and attitudes more difficult but still surprisingly amenable to measurement. Economic and society level phenomena are harder to measure accurately in many ways and yet still quite useful in their high level abstractions and aggregations.

Easy or hard, Bill believed that humans in particular are worth measuring. Bill practiced what he preached, creating a project to turn qualitative assessment of the meaning of everyday objects in popular television shows into a repeatable measurement process with inter-rater reliability and more.

I believe Bill would have delighted in one of the key side effects of the rise of the internet in so many areas of life and commerce. Humans leave behind digital trails of behavior and words. Those trails become data and data can become measurement.

Data scientists and organizations which employ them, need a firm commitment to empirical measurement, and a belief that measurement is both possible and worthwhile. Additionally those organizations need to commit to robust data quality practices so that automated systems of analysis are built on a sound foundation.

Lesson 3: Have Humility toward Evidence

The final lesson that I learned from Bill was the mix of skepticism and open-mindedness of a good scientist. The scientific mindset shows a willingness to reconsider and modify underlying theory and models when faced with contradictory evidence. Or, conversely that healthy scientific skepticism says something like the following “That sounds plausible but why don’t we gather some data to see if that is how X and Y are actually related?”

Over the years of doing research-based consulting for many different businesses it is still remarkable to me that some organizations will pay very good money to do customer, market or operations research and when the results don't come back with the answer that some key executive wanted the study gets buried. The company goes on to do exactly what that executive wanted to do in the first place.

I didn’t realize how jaded I had become until I had a scientific software company as a client. I had done a major study of their customer base and knew that the outcome of the research should lead them to a product decision that would increase costs and reduce margins for the next few years but was clearly necessary if they didn't want to lose this particular customer segment entirely over the next seven to ten years. Knowing how painful this news was from a business stand-point, I went into the board-room presentation with some trepidation. I was thinking this might be my last project for this company but as a professional researcher I believe it is my duty to be a truth teller.

After a 30 minute presentation of the research and my unwelcome business recommendation, I experienced one of the most refreshing discussions of my career. These executives, almost entirely former scientists, began asking questions about the sample, my analytical methods and reviewed various charts to clarify my findings. Fully expecting them to nitpick the study apart in order to avoid the hard business decision, I was surprised as various executives seemed satisfied with the answers and consensus started settling in around the table. And, then I realized they were still scientists at heart and now that very solid customer data was in front of them, they were prepared to make the tough business decision and do what was necessary to reposition that business to survive and thrive for years to come.

It was their humility in the face of sound empirical evidence that made me realize how cynical I had become about the ability of business executives to really hear and accept evidence when it contradicted what they wanted to do.

"That is an empirical question!" implies a willingness to hear and accept sound measurement and analysis and then grapple honestly and humbly with the implications.


Implications for Data Science

Data Science is a fairly new label, but I still find myself repeating this phrase that I first heard from Bill. “That's an empirical question!” Maybe we can help build organizations that truly respect the practices of measurement, data management, sound analysis and empirically grounded decision-making.