Measuring Humans: The Great Reversal

Astronomers, physicists and botanists might vehemently disagree but I believe that human behavior, words, beliefs, emotions and attitudes are the most compelling subjects of research. When I was studying a form of social science in grad school back in the early 1990s it was common to bemoan the lack of unobtrusive measurement of naturally occurring human behavior.

We set up experiments, but almost by definition these were highly contrived settings and stimuli with college students as the test subjects. It was extraordinarily expensive and often simply impossible to measure human behavior in a direct manner. So we asked people to reflect and report on their opinions, attitudes, emotions and intentions or to recall and estimate the occurrence and frequency of past behaviors. We could afford a more reasonable and random sample of the population of true interest if we shifted our focus to survey research. While still expensive, it was within reach. The survey polling approach to understanding people is constrained to those who will cooperate and we worry about response artifacts like social desirability or leading questions.

It was extraordinarily expensive and often simply impossible to measure human behavior in a direct manner. So we asked people to reflect and report on their opinions, attitudes, emotions and intentions or to recall and estimate the occurrence and frequency of past behaviors.

Ethnographic researchers would get around some of the problems endemic to self-reporting or experiment by spending extended periods of time with subjects observing them in their ordinary work or home environment. Some ethnographic studies combined interviews with observation in an effort to weave together external observables and internal narrative. Ethnographic studies, richly detailed as they are, don’t typically scale because the cost-per-observation is too high.

As I transitioned from academia to industry I adopted a much more pragmatic view of these measurement issues. Unless you were working in direct marketing and had a lot of money for message and price testing, or you worked for a supermarket with the rise of UPC labeling and loyalty programs you were not going to get access to behavioral data at any scale that enabled quantitative analysis.

Therefore most of the research I conducted was either qualitative conversations, think focus groups and in-depth interviews, or some form of survey research. If we were careful to avoid leading questions and did a good job with sampling and analysis we could learn a great deal from customers using these methods. We conducted significant valid research that enabled businesses to make better products, services and strategic decisions.

Often it didn’t take much to improve on an organization's current methods for accumulating evidence. Executives will often talk to a handful of customers and develop a working hypothesis about what direction would be fruitful for the business to take. If I could expand their sample from 5 customers that they personally talked with to a broader range of 30 customers spread across 3 focus groups and then extend that through survey research to 300 customers, that executive will almost certainly be in a better position to make an informed business decision than if he or she had proceeded ahead based on their personal conversations with 5 customers. As good as those methods were at providing actionable intelligence, there was still that nagging sense that we would really like some good naturalistic capture of consumer behavior, not just what they remembered or were willing to share.

Fast-Forward 20 Years

Over the past 20 years we have witnessed the rise of the Internet. It has transformed from a curiosity serving government research labs and universities to an essential platform for life and commerce. A side effect of moving so much communication and commerce online is that getting data about natural (self-chosen, functional) human behavior has become much cheaper. In fact the economics of research on humans are almost exactly opposite of where they were when I started. Now humans leave digital trails and record of their activities all over the internet. Previously we often captured opinion, attitude, narrative and intent but couldn’t afford to connect those “mental states” to actual behavior. Today we sometimes find ourselves easily able to capture large swaths of behavior without a good way to connect it directly to meaningful semantics. We know what an individual in our system did, but have no information on what they were thinking or feeling. The narrative and meaning are missing! We used to bemoan the lack of real behavior data and now we sorely feel the lack of attitude, emotion and intention in our data.

Previously we often captured opinion, attitude, narrative and intent but couldn’t afford to connect those “mental states” to actual behavior. Today we sometimes find ourselves easily able to capture large swaths of behavior without a good way to connect it directly to meaningful semantics.

As a researcher I would love to have both the narrative and the natural behavior data connected at the level of the individual. As a citizen and consumer I am uncomfortable with those streams being closely tied together. Our privacy is only tenuously maintained by separate silos. My cell phone provider knows my location and much of my communication network. Facebook, LinkedIn and Google individually know a very great deal about my interests, my network, impending decisions and the storyline of my life. My credit-card provider and bank understand where I am shopping and how I allocate my spending.

While the researcher in me wants to connect all this in order to understand how interests arise, decisions get made, where and when they get translated into behavior and what narrative and meaning we ascribe, as an individual I am glad it isn’t quite that easy yet.

For now I recommend that researchers connect the two domains in an iterative investigative process and respect the privacy of individuals. If you are an operations researcher or data scientist dealing with highly instrumented platform make sure you get out of the office. Visit the retail setting, or work the customer service desk for a couple days. Find customers and listen to them. Let the behavior data you collect generate new questions. Spend time with people capturing stories, intentions, emotions and meaning. Go back, analyze your behavioral data again with these stories in mind. Ask yourself and your team questions like: “If the story I heard from person X is a common customer experience how would I expect to see that reflected in our data?”

If you are an operations researcher or data scientist dealing with highly instrumented platform make sure you get out of the office. Visit the retail setting, or work the customer service desk for a couple days. Find customers and listen to them.

The grand reversal in human measurement is that collecting data on human behavior in a real-life situation has gone from exceedingly expensive to close to zero on a cost-per-observation basis, while the cost to get narrative and meaning has decreased by a much smaller margin. Today behavior data is often cheaper to acquire than opinion, attitude and intent.