Common Experimentation Pitfalls

Summary

Novelty effects (a change performs well just because it's new), seasonality, sample ratio mismatch, and running too many tests at once without correction — the recurring ways experiments mislead teams.

• 1Learn the term 'novelty effect' and how a longer test duration helps detect it.
• 2Read about sample ratio mismatch (SRM) and why it can silently invalidate a test.
• 3List the pitfalls most likely to show up in your own team's testing process.