Clustering Illusion

We tend to see patterns in random data sequences or events, even when no correlation or causal relationship is present. This bias reflects the human brain's tendency to seek order in randomness.

But first, let's hear from this week's sponsor...

Now, back to the Clustering Illusion ⏬

What’s the opportunity cost of seeing patterns in random data?

​

Would you rather listen? Follow the Cognition Catalog on Spotify!

Cognition Catalog: Clusterin... Feb 19 · Beyond UX Design 9:49   ​

Some time ago, I worked on a team with a stakeholder who tended to be somewhat stubborn. He was a nice enough guy, but when he got something in his head, good luck convincing him of something else.

Our team was building software with a pretty diverse user base. There were large customers with thousands of users, small customers with a few hundred, and everything in between.

Now, a large chunk of our user base consisted of really small customers with only a handful of users. But as is usually the case, the big-name, larger customers got direct access to high-level executives to voice their concerns. The little customers? Not so much.

And unfortunately, as we all know, the squeaky wheel tends to get the grease.

The trouble was these large, big-name customers, were actually a smaller overall fraction of the total user base.

But, they tended to be the loudest. And they tended to be the ones that most of the executives tended to listen to.

You can probably guess what happened next.

Our stubborn stakeholder had a nice long chat with someone from one of these large, well-known customers. They had a feature they wanted built and were insistent that it was business-critical.

Armed with this information and a bit of confirmation bias, this stakeholder wanted to know if the other larger, well-known customers also needed this feature.

Turns out they did!

... Or at least they said they did.

Fast forward a year and god knows how much money spent, and guess what?

Only a handful of users were using the feature.

Doing some interviews we found most customers had no use for this feature.

---

In hindsight, this stakeholder talked to only a handful of specific customers and heard what he wanted to hear.

In general, it wasn’t a bad thing that we gave this larger customer what they wanted. It didn’t kill the business, but there was certainly an opportunity cost here.

What else could we have been working on that would have brought more value to the software overall?

We’ll never know, because we didn’t do the research to find out. 🤷
​

---

Two important psychologists who investigated the clustering illusion were the late Daniel Kahneman and Amos Tversky. They found that the representativeness heuristic causes the illusion. This heuristic is a cognitive shortcut in which a small sample of data is assumed to be representative of the entire population from which it is drawn.

The human brain’s ability to recognize patterns and draw conclusions is really incredible. However, remember to be cautious when using small data sets to make assumptions about larger populations.

That’s because the representativeness heuristic can lead to the clustering illusion, where we mistakenly believe that a non-random pattern in a small subset of data is present throughout the entire sample.

It’s important to remember that random samples often contain more variability than we initially understand. By acknowledging this human tendency, we can approach data analysis with a more open mind and avoid drawing incorrect conclusions.

---

A classic example of this bias is the belief that if a coin toss results in “heads” several times in a row, “tails” is surely next. This is a fallacy because each coin toss is an independent event, and the odds remain 50/50 each time that it will be “heads.”

But how might we apply this to UX?

Suppose you’re analyzing user data in an e-commerce platform. After observing a few customers making purchases on consecutive days, you might perceive a clustering illusion, assuming there is a pattern or preference for online shopping on certain days of the week. However, this apparent clustering might be purely coincidental, and there may be no actual preference for those specific days.

This bias leads us to overestimate the importance of these perceived clusters and overlook the overall randomness of the data set.

This can happen to us when we analyze large data sets. But it’s also essential to think about how to represent the data more objectively when we share this information with the rest of the team. If we’re not careful, key team members can misconstrue the data to see what they want to see, as in the example I shared a few minutes ago.

---

🎯 Here are some key takeaways

1️⃣ Be cautious of pattern-seeking. Recognize that our brains naturally seek patterns and structure, even in random data. Always validate observations with rigorous analysis before drawing conclusions, especially for important decisions where being right the first time is critical.

2️⃣ Rely on statistical significance: Use appropriate statistical methods to determine whether observed patterns are statistically significant or simply the result of chance. If you have a data scientist on your team, for the love of god, get their help. They’re on the team for a reason!

3️⃣ Gather more data: It’s not always the case, but the larger the dataset, the more reliable the analysis. Especially if you have a diverse set of data, such as clients of different sizes or different types of users, collecting more data may help reduce the influence of random variations and provide a more accurate picture of what is going on. But be careful because sometimes gathering TOO much data won’t really improve your analysis.

4️⃣ Pair Quant with Qual data if possible: You might not always be able to do it, but when you can, complement quantitative data with qualitative insights through interviews, ethnography, or usability studies, etc. Find the “why” behind the “what.” This will help you gain deeper insights into the problem.

5️⃣ Communicate uncertainty: When we’re presenting data or insights to the team, communicate the possibility of random variations and the potential for the clustering illusion to avoid having your audience draw misleading conclusions.

Learn more about the the Clustering Illusion!

---

---

---