Statistical Significance Pitfalls
Part of the Data Analyst Roadmap
In short
'Statistically significant' sounds like 'important and true'. It's neither, automatically. These traps catch smart people every week, so learn to spot them in your own work first.
Common ways statistics gets misused in practice — p-hacking, multiple comparisons without correction, and treating a significant result as automatically important or actionable.
Think of it like this
Painting the target after shooting
A cheeky archer shoots arrows at a fence, then paints a bullseye around wherever one landed and claims perfect aim. Testing lots of things and only reporting the one that 'hit' is the same trick, and it's called p-hacking.
Watch it happen
Ten tests, zero real effects
1/5You test a change with no real effect on ten different metrics. Each gives a p-value.
The big ideas
What you need to know
Multiple comparisons
More tests mean more lucky false positives. Adjust thresholds or pick one primary metric upfront.
p-hacking
Trying filters, segments or dates until something hits p < 0.05. Results won't hold up.
Significant isn't important
With millions of users, a 0.01% lift can be significant but worthless. Always check effect size.
Replicate surprises
If a result is surprising, run it again before betting the roadmap on it.
Save yourself the pain
Common mistakes
Oops
Slicing results by city, device and age until one segment wins.
Do this instead
Decide segments before the test. Treat anything found afterwards as a hypothesis for the next test.
Oops
Presenting 'p < 0.05' without the size of the effect.
Do this instead
Show the estimate and interval, then ask: is this big enough to matter for the business?
Your mission
Catch yourself p-hacking
Quick check
An analyst tests 20 user segments and finds one with p = 0.04. What's the fairest conclusion?
Go deeper
- 1Learn what 'p-hacking' means and why running many tests inflates false positives.
- 2Understand the difference between statistical significance and practical significance.
- 3Practice questioning a 'statistically significant' claim you see in the wild.
More topics at this stop
Descriptive Statistics (Mean, Median, Mode, Std Dev)
Probability Basics
Distributions (Normal, Binomial)
Sampling & Sampling Bias
Correlation vs Causation
Hypothesis Testing
p-values & Confidence Intervals
A/B Testing Statistics
Regression Basics
Outlier Detection
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
