Stop 13 · AI & Modern QA PracticesTopics · 6/7

Evaluating AI-Generated Tests

In short

AI can write a test that runs, passes and looks professional, yet checks nothing important. Reviewing generated tests is now a core QA skill, as important as writing them.

AI-generated tests can look reasonable while missing critical edge cases or asserting the wrong thing entirely — critically reviewing AI output is now as important as generating it quickly.

The purple blob as a teacher marks answer sheets with a red pen while a small robot student waits.

Think of it like this

The teacher with the red pen

A student hands in neat, confident answers. The teacher still checks each one, because neat handwriting isn't the same as a correct answer. You're the teacher; the AI is the student who writes beautifully and sometimes bluffs.

Watch it happen

Five checks before you accept a generated test

Runs?

Asserts?

Right rule?

Edges?

Can fail?

1/5Does it run cleanly on your setup, with real locators and real test data?

The big ideas

What you need to know

Green isn't proof

A passing test only matters if it would fail when the feature breaks.

Weak assertions

Watch for toBeVisible or status 200 used where the real check is a value, like a total or a message.

Hallucinated details

Generated tests sometimes use URLs, fields or rules that don't exist. Check each one against the real app.

Duplicates and bloat

Ten tests that check the same thing slow the suite and add no safety. Merge or delete them.

See it in code

Walk through the code

1// Looks fine, passes, and proves almost nothing2test("discount works", async ({ page }) => {3  await page.goto("/cart");4  await page.getByRole("button", { name: "Apply coupon" }).click();5  await expect(page.locator(".total")).toBeVisible();6});7 8// Fix: enter the coupon and assert the actual business result9await page.getByLabel("Coupon code").fill("SAVE10");10await expect(page.locator(".total")).toHaveText("900 rupees");

1/3It clicks apply without even entering a coupon code.

The first test passes even if the discount is completely broken. The fixed lines check what the feature promises.

Save yourself the pain

Common mistakes

Oops

Judging generated tests by how many there are and whether they pass.

Do this instead

Judge by whether each one would catch a real bug. Try breaking the feature and see.

Oops

Assuming the AI read your requirements correctly.

Do this instead

Compare every expected result with the spec yourself, especially numbers and error messages.

Your mission

Break it to test it

~30 min

Quick check

A generated test checks only that the order confirmation page loads. What's the risk?

Go deeper

1Take a set of AI-generated test cases and audit them for missing edge cases or weak assertions.
2Practice defining what 'good coverage' looks like for a feature before evaluating AI's attempt at it.
3Learn to treat AI-generated tests as a first draft, not a finished, trustworthy artifact.