Why Three Good Reviews Don’t Mean Much: Understanding Sample Size Blindness
Sixteen-year-old Arjun and his friends were planning a special dinner to celebrate their board exam results. Searching online for restaurants, Arjun found a new place with only three reviews—all five stars with glowing comments. “This is perfect!” he declared. “Every single review is five stars. It must be amazing!”
His older sister Maya, studying statistics in college, looked over his shoulder. “Wait, it only has three reviews? That doesn’t tell you much at all.” Arjun was confused. “But they’re all perfect ratings! If it was bad, wouldn’t at least one person have given it a lower rating?”
Maya explained: “Imagine flipping a coin three times. You might get three heads in a row just by chance. That doesn’t mean the coin will always land on heads. With only three reviews, you could easily get all five stars by random luck, even if the restaurant is just average. Now look at this other restaurant with 300 reviews and a 4.7-star average. That tells you much more because it’s based on many more experiences.”
Arjun decided to trust the restaurant with hundreds of reviews. When they went to the three-review restaurant months later out of curiosity, it turned out to be mediocre—nothing like those three glowing reviews suggested. Arjun had learned about insensitivity to sample size—our tendency to ignore how much information we actually have, treating three data points as if they’re just as reliable as three hundred.
What Is Insensitivity to Sample Size?
Insensitivity to sample size is our tendency to ignore how much data we’re basing conclusions on, treating small samples as if they’re just as reliable as large ones. We look at three coin flips, see three heads, and conclude the coin is probably biased—when actually three heads happens about one in eight times with a fair coin. We see five students from a school score well and conclude the school is excellent—when actually with such a small sample, this could easily be random variation even if the school is average.
The phenomenon was identified in groundbreaking research by psychologists Daniel Kahneman and Amos Tversky. In studies at Hebrew University, they presented people with scenarios involving sampling from populations. For example: “A certain town has two hospitals. In the larger hospital, about 45 babies are born each day. In the smaller hospital, about 15 babies are born each day. About 50% of all babies are boys, but the exact percentage varies from day to day. Over one year, each hospital recorded days when more than 60% of babies born were boys. Which hospital recorded more such days?”
Most people answered that both hospitals would have similar numbers of such days, showing insensitivity to sample size. The correct answer is that the smaller hospital would have many more days with extreme percentages because small samples show much more variation. With 15 births per day, getting 10 boys (67%) is relatively common by chance. With 45 births per day, getting 30 boys (67%) is much rarer. But people intuitively ignore this fundamental statistical principle.
Research from Stanford University shows that insensitivity to sample size affects judgments across all domains—from sports performance to academic achievement to business success. We see a player make three consecutive shots and declare they’re “hot” or “in the zone,” ignoring that with small samples, three successes proves nothing about underlying ability or state. We see two students from a school succeed and recommend that school highly, ignoring that two successes from a large graduating class tells us little about the school’s overall quality.
According to studies from Yale University, the bias occurs because our intuitive statistical reasoning doesn’t naturally account for sample size. We focus on proportions or patterns—”all three reviews were five stars!”—while ignoring the critical question of how many observations support that pattern. Our brains evolved to detect patterns quickly, not to carefully weight evidence by sample size, leading to overconfidence in conclusions based on insufficient data.
The King’s Misleading Test
An ancient Chinese folktale tells of a king who wanted to select the best archer for his army. Three candidates came for testing. The king’s advisor suggested they each shoot 100 arrows at a target to fairly assess their skill. But the king was impatient. “That will take too long. Have each shoot three arrows. The one who hits the target most is clearly the best.”
The first archer, nervous about the high-stakes test, missed all three shots despite being quite skilled. The second archer, of average ability, happened to hit two of three by luck. The third archer, also average, hit all three arrows through fortunate chance.
The king immediately declared the third archer the champion and appointed him to lead the archery corps. The advisor protested: “Your Majesty, three arrows cannot reliably distinguish skill from luck. The first archer might have just had an unlucky start. The third might have had a lucky streak.” But the king dismissed this, confident that three arrows were sufficient evidence.
In the following battle, the “champion” archer performed dismally, revealing his average ability. The king had fallen victim to insensitivity to sample size—treating three arrows as sufficient evidence when they could easily produce misleading results through random variation.
Buddhist philosophy addresses this cognitive error in teachings about right understanding and evidence. The Buddha taught that conclusions should be proportional to evidence—weak evidence merits weak conclusions, strong evidence merits strong conclusions. Jumping to firm conclusions based on small samples represents a failure of this proportionality. In the Kalama Sutta, the Buddha explicitly warns against drawing strong conclusions from limited observation, teaching that understanding should be based on sufficient, repeated experience rather than isolated instances.
The Bhagavad Gita touches on this through Krishna’s teachings about discrimination (viveka)—the ability to distinguish true knowledge from appearance. Small samples create appearance of patterns that may not reflect underlying truth. Krishna teaches that wisdom requires looking beyond surface appearances to understand deeper realities, which in statistical terms means gathering enough data to see through random variation to actual patterns.
How Small Samples Mislead Us Daily
In education and school choice, insensitivity to sample size leads to poor decisions based on insufficient evidence. Parents hear about two students from a particular school who got into IIT and conclude the school must be excellent. They don’t consider that the school might graduate 500 students annually, meaning two IIT admissions actually represents a very low success rate. Or they hear about one student who struggled at a different school and write it off entirely, not considering that one struggling student out of hundreds proves nothing about the school’s overall quality.
Research from Harvard University shows that school rankings based on small graduating classes—especially private schools with 20-50 graduates—show enormous year-to-year variation simply due to sample size. A school might rank in the top ten one year and drop to mediocre the next, not because anything changed about the school but simply because small sample performance is inherently unstable. Parents treating these volatile rankings as meaningful signals make decisions based on noise rather than information.
In health and medical decisions, small sample insensitivity causes people to trust miracle cures based on testimonials from a handful of people. “Five people tried this supplement and all got better!” sounds convincing until you realize that with many illnesses, people naturally recover or experience placebo effects, so five successes proves nothing. You’d need to compare hundreds of people taking the supplement versus hundreds taking placebo to determine if it actually works better than doing nothing.
Traditional medicine and alternative treatments often rely on this bias. Practitioners share several dramatic success stories—small samples that could easily occur by chance, placebo effect, or natural recovery—and people conclude the treatment works. Meanwhile, rigorous studies with large samples frequently show no effect, but these lack the emotional impact of individual success stories and get ignored.
In consumer decisions and product reviews, small sample insensitivity makes new products with few reviews seem better than established products with many mixed reviews. A product with three five-star reviews seems perfect. A product with 300 reviews averaging 4.5 stars seems flawed because you see complaints among the reviews. Actually, the second product is far more reliably good—the 4.5 average based on 300 reviews is much more informative than 5.0 based on three reviews.
Online marketplaces struggle with this because new sellers can easily get a few friends to leave glowing reviews, making them seem more trustworthy than established sellers with thousands of reviews and a realistic mix of positive and negative feedback. Smart platforms weight reviews by quantity, but users often don’t intuitively understand why the newer seller with perfect reviews from three people might be riskier than the established seller with imperfect reviews from thousands.
In social judgments and stereotypes, small sample insensitivity reinforces prejudice. Someone meets two rude members of a particular group and concludes the entire group is rude, not recognizing that two encounters provide almost no information about millions of people. They meet one helpful member of another group and conclude that group is friendly. Both conclusions overweight tiny, unrepresentative samples.
This creates confirmation bias feedback loops. Once you’ve concluded from a small sample that a group has certain characteristics, you notice and remember group members who fit that expectation while discounting those who don’t. The initial conclusion based on insufficient data becomes self-reinforcing despite remaining based on small, unrepresentative samples.
Thinking Properly About Sample Size
The most important principle is simple: strong conclusions require strong evidence, which means large samples. If you’ve seen three examples, you have weak evidence suitable for tentative hypotheses, not confident conclusions. If you’ve seen 300 examples, you have much stronger evidence. Always ask “How many observations is this based on?” before deciding how much to trust a pattern.
Develop intuition for the law of large numbers: small samples show lots of random variation, while large samples reveal true underlying patterns. Flip a coin three times, and getting three heads wouldn’t surprise you—it happens often. Flip a coin 300 times and getting 300 heads would be astronomically unlikely, clear evidence of a biased coin. The smaller the sample, the more you should expect extreme, unrepresentative results by pure chance.
Practice calculating what random variation looks like for different sample sizes. If a school has 30 students taking an exam where average is 70%, how many might score above 85% by luck? Probably 3-5 students. If you hear about three students who scored high, that’s expected random variation, not evidence the school is special. If the school has 300 students and 50 scored above 85%, that’s stronger evidence of something real—either better teaching or different student population—because it exceeds what you’d expect from random variation.
Look for systematic comparisons using large samples rather than trusting individual testimonials or small sample successes. Medical treatments should be tested on hundreds or thousands of people in controlled trials, not judged by a handful of success stories. School quality should be assessed by years of outcomes from full graduating classes, not by a few successful students. Product quality should be judged by patterns across hundreds of reviews, not the first three experiences.
Embrace uncertainty when data is limited. Saying “I don’t know enough yet” or “This is interesting but inconclusive” reflects statistical wisdom when your sample is small. Premature confidence based on insufficient data leads to poor decisions. The wisdom is recognizing that three data points tell you very little, even when those three points show a perfect pattern.
Remember Arjun and the three perfect restaurant reviews that meant almost nothing, and the king’s three-arrow archery test that completely failed to identify the best archer. Both ignored sample size, treating tiny samples as if they were large ones. The restaurant with three five-star reviews could have been amazing or terrible—three reviews simply don’t tell you. The archer who hit all three arrows could have been brilliant or lucky—three shots simply don’t tell you. In both cases, the additional data from larger samples revealed truths invisible in the small samples. When someone shows you three examples of anything, your response should be “That’s interesting—tell me about the other 297.” Without knowing about those additional observations, you simply don’t have enough information to draw reliable conclusions, no matter how perfect those three examples appear.
Frequently Asked Questions
How many data points do I need before I can trust a pattern?
It depends on the context, but as a rule of thumb: single digits (1-9 observations) tell you almost nothing reliably, tens (10-99) give you hints worth considering tentatively, hundreds (100-999) provide reasonably reliable patterns, and thousands give you strong confidence. For medical conclusions, you need controlled studies with hundreds or thousands. For restaurant quality, 100+ reviews are far more reliable than 10. The key is that you need enough observations for random variation to average out and true patterns to emerge clearly.
If small samples are unreliable, why do we naturally trust them?
Because in ancestral environments with limited information, acting quickly on small samples was often better than waiting for more data. If you saw two people get sick after eating berries, better to avoid those berries immediately than wait to gather data from 30 more people. This made evolutionary sense when information was scarce and slow to gather. In modern environments with easy access to large datasets, the same instinct misleads us by making us overconfident in patterns based on tiny samples.
Does this mean personal anecdotes and testimonials are worthless?
Not worthless, but they should be treated as weak evidence—data points worth noting but insufficient for strong conclusions. One person’s testimonial that a treatment worked means that treatment worked for at least one person, which is mildly interesting. But it doesn’t tell you whether the treatment works better than placebo, works for most people, or just happened to coincide with natural recovery. Personal stories are emotionally compelling but statistically weak evidence that should never override large-sample systematic studies.
How can marketers use small samples to make products seem better than they are?
By highlighting the few successes while hiding the many failures. A supplement that helped 3 out of 100 people can be marketed with “Three people saw amazing results!” featuring dramatic testimonials, while ignoring the 97 people it didn’t help. Or a new product can solicit reviews from a few friendly early adopters to get perfect initial ratings based on 3-5 reviews, creating false confidence before real customers reveal the product’s actual mixed quality through larger sample reviews. Smart consumers look for large sample data and ignore impressive small sample claims.
Can I overcome insensitivity to sample size just by learning about it?
Learning helps but doesn’t eliminate the intuitive error. Even statisticians must consciously remind themselves to check sample sizes because the bias operates at an intuitive level that knowledge doesn’t fully override. The solution is developing the habit of always asking “How many observations?” before trusting a pattern. Make sample size assessment automatic—when you see “100% success rate,” immediately ask “100% of how many?” If the answer is three people, you know to be skeptical. If it’s 300 people, that’s much more meaningful. The habit takes practice but becomes natural with use.
Observer Voice is the one stop site for National, International news, Sports, Editor’s Choice, Art/culture contents, Quotes and much more. We also cover historical contents. Historical contents includes World History, Indian History, and what happened today. The website also covers Entertainment across the India and World.
