Investigate tests that pass while the code is broken

Mutate the source and check whether the suite would actually notice, instead of trusting a green run.

Also usesclaude-plugins-official

A green suite proves the tests ran, not that they would notice. The specific trap worth naming: an assertion whose expected value brackets a hardcoded default, so the test passes whether or not the logic under it ever executes.

Start here

One prompt to start

This suite is green but I don't trust it -- mutate the source (flip a boundary, negate a condition, shift an index) and tell me which mutations it wouldn't catch, review the tests for real behavioral coverage rather than line coverage, and verify the verifier itself since it's the checker that's actually in question here.

Needszeugnis, pr-review-toolkit, andon

Beats

Run these in order. Each prompt is copy-pasteable straight into Claude Code.

1. zeugnis:zeugnis-assertion-auditalso in 3 other recipes

   • Investigate a job that reports success but changed nothing
   • Investigate a fix that did not stick
   • Do oracle engineering and numerical V&V

   Proposes off-by-one, boundary-flip, and condition-negation mutations and judges whether any existing test catches them.

   Run this beat on its own

   mutate this module — flip a boundary, negate a condition, shift an index — and tell me which mutations the tests would not catch

2. zeugnis:zeugnis-mutation-harden

   Runs a real mutation tool on the module the audit flagged and writes the tests that kill what survives, each kill proven by applying its mutant.

   Run this beat on its own

   run mutmut on this module and write the tests that kill the mutants that survive

3. pr-review-toolkit:pr-test-analyzeralso in 1 other recipe

   • Whole-branch review without re-trusting the branch's own self-assessment

   Coverage percentages are compatible with assertions that assert nothing.

   Run this beat on its own

   review the tests on this branch for behavioral coverage, not line coverage — where are the real gaps?

4. andon:andon-verifyalso in 15 other recipes

   • Audit supply-chain pinning and dependencies
   • Finish and integrate a development branch
   • Investigate a job that reports success but changed nothing
   • Investigate a fix that did not stick
   • Root-cause a bugfix from a tracked issue
   • Build a new feature in an existing codebase
   • Collapse duplication hand-synced across N sites
   • Execute a written plan across parallel workstreams
   • Fix an existing dashboard without breaking it
   • Migrate a return shape or type representation
   • Propagate a vendored artifact to N copies
   • Audit your own agent and skill definitions for loop-reliability defects
   • Rework a plugin to the official Anthropic standard
   • Do oracle engineering and numerical V&V
   • Whole-branch review without re-trusting the branch's own self-assessment

   One of its seven strategies is verify-the-verifier — the right shape when the checker is what is suspect.

   Run this beat on its own

   I don't trust this test suite. Verify the verifier before we trust anything it says.

Worked example

Grounded in — why this beat order is trustworthy

auditing `plugins/zeugnis/scripts/test_cycle_engine.py` and `tools/enforcement-audit/test_audit_enforcement.py` for assertions whose expected value is the same hardcoded default the code falls back to when the real path never runs.

Do / Don't

• ✓Mutate the source with off-by-one, boundary-flip, and condition-negation changes, and check which ones the suite would miss.
• ✓Review for behavioral coverage, not line coverage -- a high percentage is compatible with assertions that assert nothing.
• ✓Verify the verifier itself when the suite, not the code, is what's actually suspect.
• ✓When an estimate is not enough, run a real mutation tool on a fresh copy and report the untested mutants apart from the score.

• ×Don't trust a green suite as proof the tests would notice a regression -- it only proves the tests ran.
• ×Don't treat a high coverage percentage as evidence of real assertions -- it's compatible with tests that assert nothing meaningful.
• ×Don't write an assertion whose expected value is the same hardcoded default the code falls back to -- it passes whether or not the real path ever executes.
• ×Don't run a mutation tool in the working tree -- mutmut 3 writes mutants/ there and its apply rewrites the source file.