Stop 12 · AI & Modern Product ManagementTopics · 4/7

Evaluating AI Feature Quality

In short

"I tried it five times and it looked fine" isn't a quality check. AI features need an exam paper, a marking scheme, and lots of real answers to mark.

Measuring an AI feature isn't just 'does it work' — it requires defining what a good vs bad output looks like, at what error rate, and testing against real (not cherry-picked) inputs.

The purple blob marks answer sheets with a red pen beside a checklist.

Think of it like this

The examiner's marking scheme

Before checking papers, examiners agree on what a full-marks answer looks like. Then they mark a big pile, not just the toppers' sheets. AI evaluation is the same: define good first, then grade many real outputs against it.

Watch it happen

Building an evaluation for an AI summary feature

Define good

Test set

Run

Grade

Track

1/5Write a rubric: summary is accurate, under 80 words, mentions the customer's main issue, no invented details.

The big ideas

What you need to know

Rubric first

Decide what good and bad look like before testing, or you'll grade by mood.

Real, messy inputs

Test on real examples, including edge cases. Cherry-picked demos hide failures.

Not all errors are equal

A typo is minor; a made-up refund promise is serious. Track severe failures separately.

Re-test every change

A new prompt can fix one case and break three others. Keep the test set and rerun it.

See it in code

A tiny example

excel

1=COUNTIF(D2:D101, "pass") / COUNTA(D2:D101)2=COUNTIFS(D2:D101, "fail", E2:E101, "severe")

In a sheet where column D holds pass/fail grades and column E holds severity, the first formula gives your pass rate and the second counts severe failures.

Save yourself the pain

Common mistakes

Oops

Testing with ten examples you wrote yourself.

Do this instead

Use a large sample of real user inputs, including the strange ones.

Oops

Reporting one overall score only.

Do this instead

Report the pass rate plus a count of severe failures, since one bad answer can hurt trust badly.

Your mission

Build a tiny eval

~45 min

Quick check

Your new prompt improves the demo examples. What should you check before shipping?

Go deeper

1Define 3 criteria for 'good output' for a hypothetical AI feature.
2Learn the basic idea of an eval set: a fixed batch of test inputs you score consistently.
3Understand why demo-day performance often overstates real-world reliability.