Skip to content

07 · AI-Assisted Testing

"AI-assisted testing" covers two genuinely different things: tools that generate test cases automatically by analyzing your code (search-based, no LLM involved — like the real tool run below), and using an LLM as a drafting assistant for test code, reviewed by a human before it's trusted. This module demonstrates the first with real, executed output, and covers the second with the review discipline it actually requires.

1. Search-based automatic test generation with Pynguin

pip install pynguin

Pynguin analyzes a module's code paths and evolves a test suite — generating input combinations, running them, and mutating the ones that hit new branches — without any LLM involved at all. This actually ran:

# mathutils.py
def clamp(value, low, high):
    if value < low:
        return low
    if value > high:
        return high
    return value

PYNGUIN_DANGER_AWARE=1 pynguin \
    --project-path src --output-path out \
    --module-name mathutils --maximum-search-time 15

# out/test_mathutils.py  — genuinely auto-generated, not hand-written
# Test cases automatically generated by Pynguin (https://www.pynguin.eu).
# Please check them before you use them.
import pytest
import mathutils as module_0


@pytest.mark.xfail(strict=True)
def test_case_0():
    int_0 = 5347
    none_type_0 = None
    module_0.clamp(int_0, int_0, none_type_0)


def test_case_1():
    bool_0 = False
    list_0 = [bool_0, bool_0]
    list_1 = [bool_0, list_0]
    list_2 = [bool_0, bool_0, list_1]
    module_0.clamp(list_2, list_2, list_0)


def test_case_2():
    set_0 = set()
    module_0.clamp(set_0, set_0, set_0)


def test_case_3():
    int_0 = 5347
    bool_0 = False
    var_0 = module_0.clamp(bool_0, int_0, int_0)
    assert var_0 == 5347

$ pytest test_mathutils.py -v
test_mathutils.py::test_case_0 XFAIL
test_mathutils.py::test_case_1 PASSED
test_mathutils.py::test_case_2 PASSED
test_mathutils.py::test_case_3 PASSED
3 passed, 1 xfailed in 0.55s

Every one of these ran for real. test_case_0 is genuinely interesting: Pynguin discovered that calling clamp(5347, 5347, None) raises a TypeError (comparing int > None is illegal in Python 3) and marked it xfail(strict=True) — meaning it expects that call to keep raising, and will itself fail if a future code change makes it stop raising. That's Pynguin finding an untyped-input edge case a human would need to think of deliberately: nothing in clamp's signature stops someone from passing None, and the tool surfaced exactly what happens.

2. Reading Pynguin's own warning literally

The generated file's own comment — "Please check them before you use them" — is not boilerplate to skip past. test_case_1 through test_case_3 pass, but their inputs (nested lists, an empty set) are not realistic values for a function meant to clamp numbers. They're valid from a pure code-path perspective (Python's comparison operators happen to work on them) but useless as documentation of intended behavior. The correct workflow is: generate, then curate — keep the genuinely revealing cases (like test_case_0's None discovery), rewrite the rest with meaningful values, and delete anything redundant.

3. Using an LLM to draft tests — with the same discipline

The second common meaning of "AI-assisted testing" is prompting an LLM ("write pytest tests for this function") and reviewing the result. The failure modes are different from Pynguin's but the discipline is identical:

# A plausible LLM-drafted test for the same clamp() function
def test_clamp_returns_input_when_in_range():
    assert clamp(5, 0, 10) == 5

def test_clamp_low_boundary():
    assert clamp(-1, 0, 10) == 0

def test_clamp_high_boundary():
    assert clamp(11, 0, 10) == 10

These read naturally and are individually correct — but an LLM drafting tests from the function's implementation (rather than its specification) can reproduce the same bug in both the code and the test: if clamp had an off-by-one error using >= instead of >, an LLM asked to "write tests that pass" might generate an assertion that encodes the bug rather than catching it. This is the single most important thing to check when reviewing AI-drafted tests: was the expected value derived from what the function should do, or copied from what it currently does?

4. A concrete review checklist for AI-generated tests

  1. Does each assertion encode the spec, or the implementation? Trace the expected value back to a requirement or a hand-computed answer, not to running the code and copying its output.
  2. Are the test names and structure honest? An LLM will sometimes generate a test named test_handles_edge_cases that doesn't actually cover an edge case — read the body, not just the name.
  3. Is there real edge-case coverage, or just happy-path repetition? Pynguin's search-based approach (section 1) tends to find genuine edge cases because it explores code paths mechanically; an LLM prompted loosely tends to generate variations on the same input shape.
  4. Would this test survive a legitimate refactor? A test asserting internal implementation details (mock call counts, private attribute values) rather than observable behavior will break on a refactor that changes nothing externally — the same anti-pattern from Level 2 Module 4's mocking discussion, independent of who or what wrote the test.

5. Where AI assistance genuinely helps in this course's context

  • Drafting the tedious first pass of parametrize tables (Level 1 Module 8) for a function with many input/output pairs — fast to draft, fast to review against a spec.
  • Suggesting property candidates for Hypothesis (Level 3 Module 5) — an LLM can suggest "this should be commutative" or "this should round-trip," which a human then verifies is actually true before encoding it.
  • Explaining an unfamiliar failure — pasting a stack trace and asking "why would this fail" is a legitimate, low-risk use, since the human still verifies the explanation against the actual code.
  • Generating exploratory candidates (Pynguin's actual mechanism) to surface inputs a human wouldn't have thought to try, as section 1 showed concretely with the None case.

6. Testing-specific traps

Trap 1 — trusting generated assertions without deriving the expected value independently. Section 3 is the core risk: any tool (search-based or LLM) that derives its expected output by running the code under test can encode that code's bugs as "passing" tests. Always independently verify at least the non-trivial assertions.

Trap 2 — committing auto-generated tests unreviewed. Pynguin's own output explicitly warns against this. A CI pipeline that runs generated tests without a human review step can accumulate tests-for-tests'-sake (like test_case_1's nested-list input above) that add runtime and noise without adding real confidence.

Trap 3 — over-indexing on AI-found edge cases while missing domain-specific ones. Pynguin found the None-comparison TypeError mechanically, but it has no idea that, say, a clamp used for a percentage should probably reject values above 100 as a business rule, not just a type error. Mechanical tools find code-path edge cases; domain edge cases still need a human who understands the requirements.

Trap 4 — using AI-assisted tests as a substitute for understanding the code. A team that generates tests without reading and understanding what they assert loses the main non-bug-catching benefit of a test suite: serving as living documentation of intended behavior for the next person who reads it.

How It Actually Works

AI-assisted test generation tools (Copilot-style code completion, or LLM-based test- case generators) work by treating your source code as context fed into a language model that predicts likely-plausible test code token by token, conditioned on patterns learned from a huge corpus of existing test suites — it is pattern completion over code structure and naming conventions, not program analysis or execution. This is precisely why AI-generated tests reliably produce syntactically correct, idiomatically-styled pytest that can still assert something subtly wrong (mirroring a common but incorrect pattern from training data) or entirely miss a domain-specific edge case that never appeared in similar-looking training examples — the model has no access to your program's actual runtime behavior or business requirements, only to surface-level code shape.

This is also why AI-assisted testing tools are safest used as a first draft that a human then runs and checks against real behavior (does the assertion's expected value actually match what the code produces, verified by executing it) rather than as an oracle — you already have every tool needed to verify an AI-generated test rigorously: run it, check the assertion is checking something meaningful (not assert result is not None when a real assertion on value is possible), and confirm it fails when the implementation is deliberately broken (a mutation-testing-style sanity check, since a generated test that can't fail isn't testing anything).

Cheat sheet

Tool/technique What it's good at What still needs a human
Pynguin (search-based) mechanically finding code-path edge cases curating which generated cases are meaningful
LLM-drafted tests fast first drafts, boilerplate parametrize tables verifying expected values against spec, not implementation
LLM-explained failures fast hypothesis generation for a stack trace confirming the explanation against the actual code
Either, for property suggestions (Level 3 Module 5) surfacing candidate invariants verifying the invariant is actually true
Neither encoding domain/business rules as edge cases requires understanding the requirements, not just the code

Exercise

  1. Install pynguin, run it against a small function of your own (something with at least one conditional branch), and inspect the generated test file — identify which generated case, if any, reveals a genuine edge case you hadn't considered.
  2. Run the generated suite with pytest -v and report the pass/fail/xfail breakdown, same as shown in section 1.
  3. Pick two of the generated test cases and rewrite them with realistic, documented input values instead of the raw generated literals — explain in a comment what each now demonstrates.
  4. Draft (by hand, imagining you're an LLM, or by actually using one) three tests for a function with a known subtle bug (write one deliberately), and show how an assertion derived from running the buggy code would "pass" while a spec-derived assertion would correctly fail.
  5. Write a two-paragraph team guideline for reviewing AI-generated test PRs, using the checklist in section 4 as your starting point.