Stop 04 · Statistics for AnalystsTopics · 7/11

p-values & Confidence Intervals

In short

p-values are probably the most misunderstood number in analytics. Learn what they actually say, pair them with confidence intervals, and you'll be ahead of plenty of people.

A p-value estimates how surprising your result would be if there were truly no effect; a confidence interval gives a plausible range for the true value — both are ways of quantifying uncertainty.

The jaadu in a small boat casts a wide fishing net into a lake.

Think of it like this

Fishing with a net, not a spear

A spear aims at one exact point and usually misses. A net covers a range and probably catches the fish. A single estimate is the spear. A confidence interval is the net: a range likely to contain the true value, with the net's width showing your uncertainty.

Watch it happen

Same estimate, different certainty

Lift: +2.0%

CI: -1.5% to +5.5%

CI: +1.2% to +2.8%

1/4Your test shows a new checkout lifts conversions by 2.0 percentage points. That's the spear.

The big ideas

What you need to know

What a p-value is

If there were truly no effect, the chance of seeing a result at least this extreme.

What it is not

It's not the chance the null is true, and it doesn't measure how big or important the effect is.

Confidence interval

A range of plausible values for the true effect. Wide means uncertain, narrow means precise.

They agree

If a 95% CI excludes zero, the matching two-sided test gives p < 0.05.

See it in code

Walk through the code

python

1from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep2 3conversions = [460, 400]      # treatment, control4visitors    = [10000, 10000]5 6z, p = proportions_ztest(conversions, visitors)7low, high = confint_proportions_2indep(460, 10000, 400, 10000)8 9print(f"p = {p:.4f}")10print(f"95% CI for the difference: {low:.4f} to {high:.4f}")

1/4Conversions and visitors for each group.

Comparing two conversion rates (4.6% vs 4.0%) with statsmodels: a p-value and a confidence interval for the gap.

Save yourself the pain

Common mistakes

Oops

Saying 'p = 0.03 means a 97% chance our feature works'.

Do this instead

Say 'if there were no effect, a result this large would be unusual'. That's all a p-value claims.

Oops

Reporting just 'significant' with no effect size.

Do this instead

Give the estimate and CI: '+0.6 points, 95% CI roughly 0.04 to 1.2'. People need the size to decide.

Your mission

Read real intervals

~20 min

Quick check

A 95% confidence interval for a lift is -0.4% to +3.1%. What can you say?

Go deeper

1Learn the common (and wrong) misinterpretations of a p-value, and the correct one.
2Practice reading a confidence interval and stating what it does and doesn't mean.
3Understand why p < 0.05 is a convention, not a law of nature.