Do oracle engineering and numerical V&V

Ground expected numeric values in an external source, since the code and the test can agree from the same misunderstanding.

Numerical correctness has a distinct failure mode: the code and the test agree because both were written from the same misunderstanding. The fix is an oracle the implementation did not author.

Start here

One prompt to start

Check where every expected numeric value in these tests actually comes from -- cite an independent source for each one, not a value derived from the code under test -- then prove any numeric claim we have no independent oracle for, and check whether the tolerances are actually tight enough to fail if the computation were subtly wrong.

Needscompass, andon, confab

Beats

Run these in order. Each prompt is copy-pasteable straight into Claude Code.

1. compass:compass-ground-evidencealso in 2 other recipes

   • Run a read-only design study with an evidence legend
   • Verify a brief's premises before acting

   An expected value with no citation is a second implementation, not an oracle.

   Run this beat on its own

   where does each expected number in these tests come from? Cite a source for every one — no values derived from the code under test.

2. andon:andon-verifyalso in 13 other recipes

   • Audit supply-chain pinning and dependencies
   • Finish and integrate a development branch
   • Investigate a job that reports success but changed nothing
   • Investigate a fix that did not stick
   • Investigate tests that pass while the code is broken
   • Root-cause a bugfix from a tracked issue
   • Build a new feature in an existing codebase
   • Collapse duplication hand-synced across N sites
   • Execute a written plan across parallel workstreams
   • Migrate a return shape or type representation
   • Propagate a vendored artifact to N copies
   • Audit your own agent and skill definitions for loop-reliability defects
   • Whole-branch review without re-trusting the branch's own self-assessment

   Strategy b is "oracle-gap numerical V&V" — named for exactly the gap between a numeric claim and an independent source of truth.

   Run this beat on its own

   check whether this numeric claim is actually right — we have no independent oracle for it

3. confab:confab-assertion-auditalso in 3 other recipes

   • Investigate a job that reports success but changed nothing
   • Investigate a fix that did not stick
   • Investigate tests that pass while the code is broken

   A tolerance wide enough to pass is a tolerance wide enough to hide the defect.

   Run this beat on its own

   are these numerical assertions tight enough to fail if the computation were subtly wrong?

Worked example

Grounded in — why this beat order is trustworthy

`tools/symbol-indexer/test_build_symbol_index.py` is the suite CI runs for the indexer; the oracle question is whether its expected index is derived independently or regenerated from the same `build_symbol_index.py` it is meant to check.

Do / Don't

• ✓Cite an independent source for every expected numeric value -- a value derived from the code under test is a second implementation, not an oracle.
• ✓Prove a numeric claim explicitly when no independent oracle exists for it yet.
• ✓Check that tolerances are tight enough to actually fail on a subtly wrong computation.

• ×Don't derive an expected test value from the same code the test is meant to check -- that's a second implementation of the bug, not an oracle.
• ×Don't leave a numeric claim unproven just because the code and the test already agree -- they can agree from the same misunderstanding.
• ×Don't set a tolerance wide enough to pass regardless -- that's wide enough to hide the defect too.