In their 1974 paper, Judgment under Uncertainty: Heuristics and Biases, Amos Tversky and Daniel Kahneman identified a series of cognitive biases and heuristics that humans often use when making judgments under uncertainty. These shortcuts can lead to systematic errors in decision-making.
The ‘Insensitivity to Sample Size’ bias was one of these cognitive biases they discovered. Tversky and Kahneman showed that people often ignore the size of the sample when making judgments based on statistical data. This can lead to an overestimation of the reliability of the results from small samples and can impact various areas such as business decisions, policy-making, and scientific research.
For UX professionals, this bias can come into play during the research and synthesis phases. Imagine the following scenario:
You’re conducting usability studies for a new feature. After gathering feedback from three participants, you decide to implement significant changes based on their responses.
Insensitivity to sample size could lead you to draw premature conclusions about the effectiveness of the feature, as the feedback from such a small sample may not accurately represent the views and preferences of your entire user base.
Obviously, the size of the sample totally depends on your user base. If you’re building an internal enterprise tool with only ten key users, the sample size will be very different from a commercial application used by thousands of different people.
🎯 Here are some key takeaways:
Prioritize larger samples for critical decisions
When making important design choices or assessing user satisfaction, prioritize data from larger, more diverse samples to gain a more reliable understanding. Always ask “What if we’re wrong?”
Emphasize statistical significance
Be cautious when interpreting data from small sample sizes, and avoid making sweeping decisions based on limited data without considering statistical significance. When presenting findings, communicate the sample size and potential limitations, ensuring everyone understands the level of confidence in the results.
Conduct robust research
Aim for larger sample sizes whenever possible to ensure more reliable and representative results. Adequate sample sizes help reduce the impact of random variations on data interpretation.
Use confidence intervals
When presenting research findings, include confidence intervals to indicate the range of uncertainty around the estimated values, giving everyone a more nuanced understanding of the data's reliability.
Combine qualitative and quantitative data
Use qualitative insights to complement quantitative data, providing a deeper understanding of user behavior and motivations, especially in cases where large sample sizes may not be feasible.
📚 Keep exploring
To dive deeper into the topic of Insensitivity to Sample Size and its implications for decision-making, check out these resources:
