Investigate tests that pass while the code is broken

Mutate the source and check whether the suite would actually notice, instead of trusting a green run.

Also usesclaude-plugins-official

A green suite proves the tests ran, not that they would notice. The specific trap worth naming: an assertion whose expected value brackets a hardcoded default, so the test passes whether or not the logic under it ever executes.

Start here

One prompt to start

This suite is green but I don't trust it -- mutate the source (flip a boundary, negate a condition, shift an index) and tell me which mutations it wouldn't catch, review the tests for real behavioral coverage rather than line coverage, and verify the verifier itself since it's the checker that's actually in question here.

Needsconfab, pr-review-toolkit, andon

Beats

Run these in order. Each prompt is copy-pasteable straight into Claude Code.

1. confab:confab-assertion-auditalso in 3 other recipes

   • Investigate a job that reports success but changed nothing
   • Investigate a fix that did not stick
   • Do oracle engineering and numerical V&V

   Proposes off-by-one, boundary-flip, and condition-negation mutations and judges whether any existing test catches them.

   Run this beat on its own

   mutate this module — flip a boundary, negate a condition, shift an index — and tell me which mutations the tests would not catch

2. pr-review-toolkit:pr-test-analyzeralso in 1 other recipe

   • Whole-branch review without re-trusting the branch's own self-assessment

   Coverage percentages are compatible with assertions that assert nothing.

   Run this beat on its own

   review the tests on this branch for behavioral coverage, not line coverage — where are the real gaps?

3. andon:andon-verifyalso in 13 other recipes

   • Audit supply-chain pinning and dependencies
   • Finish and integrate a development branch
   • Investigate a job that reports success but changed nothing
   • Investigate a fix that did not stick
   • Root-cause a bugfix from a tracked issue
   • Build a new feature in an existing codebase
   • Collapse duplication hand-synced across N sites
   • Execute a written plan across parallel workstreams
   • Migrate a return shape or type representation
   • Propagate a vendored artifact to N copies
   • Audit your own agent and skill definitions for loop-reliability defects
   • Do oracle engineering and numerical V&V
   • Whole-branch review without re-trusting the branch's own self-assessment

   One of its seven strategies is verify-the-verifier — the right shape when the checker is what is suspect.

   Run this beat on its own

   I don't trust this test suite. Verify the verifier before we trust anything it says.

Worked example

Grounded in — why this beat order is trustworthy

auditing `plugins/confab/scripts/test_cycle_engine.py` and `tools/enforcement-audit/test_audit_enforcement.py` for assertions whose expected value is the same hardcoded default the code falls back to when the real path never runs.

Do / Don't

• ✓Mutate the source with off-by-one, boundary-flip, and condition-negation changes, and check which ones the suite would miss.
• ✓Review for behavioral coverage, not line coverage -- a high percentage is compatible with assertions that assert nothing.
• ✓Verify the verifier itself when the suite, not the code, is what's actually suspect.

• ×Don't trust a green suite as proof the tests would notice a regression -- it only proves the tests ran.
• ×Don't treat a high coverage percentage as evidence of real assertions -- it's compatible with tests that assert nothing meaningful.
• ×Don't write an assertion whose expected value is the same hardcoded default the code falls back to -- it passes whether or not the real path ever executes.