review: measure what adjudication kills, then raise detection power #120

New issue

Closed

opened 2026-08-13 15:44:54 +00:00 by john · 1 comment

john commented

2026-08-13 15:44:54 +00:00

Owner

Four changes to the review subsystem. The first costs nothing and is what makes the rest measurable, so it lands first.

1. Record kills, not just survivors (do this first, zero tokens)

references/review-triggers.md:238 records the adjudication as:

"<trigger>: <n> survivors; voices <...>"

lib/trace-suggest.mjs:118 (rule R1) then proposes turning a gate OFF when survivors === 0 across two or more fires. But 0-of-0 and 0-of-9 are opposite facts:

0 raised, 0 survived: the gate found nothing, and may genuinely be unnecessary
9 raised, 0 survived: the reviewer is miscalibrated, and the gate is doing real work catching it

Both produce the identical suggestion today. The re-arm veto softens this but does not separate the two.

Change the detail to <n> survivors of <m> raised. That is the whole change, and every item below becomes measurable rather than asserted once it lands.

2. Send the contract to the cross-model arm (highest value per byte)

review-provider.mjs:830:

system: payload.instruction, user: payload.artifact

instruction is one model-authored line. So an external reviewer's ENTIRE system prompt is that line. It never sees <stance>, never sees <what_to_look_for>, never sees the severity definitions, never sees "Approach differences are NOT findings", and never sees that an empty findings: [] is a valid result. It also has no repo access by construction.

Every cross-model finding is therefore uncalibrated against our own severity bar. With review.mode at panel or single and a blocking gate, that output can FAIL a gate on a bar the model invented.

Fix: compose instruction in fire() step 2 as stance + what_to_look_for + severity definitions + the trigger-specific ask. Roughly +600 tokens against a 120,000 token cap, which is half a percent. Prose change in references/review-triggers.md, no code, no schema.

3. Hand the reviewer the criteria it is told to check

The contract at skills/cad-reviewer-contract/SKILL.md:44-47 demands the reviewer find "a requirement with no task, a task that does not deliver its requirement, a 'done' truth no task makes true, a contradicted locked decision".

The plan trigger's payload is the PLAN path alone.

The sibling gate does it right: workflows/plan.md hands cad-plan-checker PLAN*.md, ROADMAP.md, REQUIREMENTS.md, CONTEXT.md and names all four in the prompt. The reviewer is asked the same questions with a quarter of the evidence.

Fix: add ROADMAP, REQUIREMENTS and CONTEXT to the plan payload, and the phase CONTEXT plus PLAN as a second reference on diff, phase_diff and pre_ship. These are references, not inlined text, so the orchestrator cost is about zero.

4. Give the reviewer a process with a self-refutation step

cad-reviewer is the ONLY role contract in the subsystem with no <process> section. plan-checker, verifier and executor all have one.

Worse, <guardrails> reads "One pass. Report everything you find now; there is no second look." Read literally that discourages self-refutation, because refuting your own candidate findings IS a second look at them.

The one line that asserted falsification was deleted in 49ba72e:

- [ ] A genuine falsification attempt was made, grounded against the real files

design-notes/sweep-2026-08-10-context-weight.md:46 framed that cut as "easiest to reverse if compliance drops". This is the reversal condition. It was about 200 bytes and it was the subsystem's only self-check assertion.

Add: resolve the reference, generate candidates, open the cited file at the cited line and try to KILL each candidate, return only survivors carrying the check that failed to kill them. About +250 tokens per dispatch. It finds nothing new directly; it removes the false positives that bury the real findings, and adjudication only runs at the adjudicated gate, so at advisory and blocking nothing filters them today.

Scope the "one pass" guardrail to "no second DISPATCH" so it stops contradicting the new step.

Deliberately not in scope

R-05, requiring the reviewer to RUN the failure. It has Bash and is told to ground findings to a line, never to reproduce them. Making it prove the failure would convert plausible findings into confirmed ones, and the verifier has precedent for bounded execution (cad-verifier-contract:118-131, under 10s, no servers, no state mutation, no network). But the reviewer's read-only guarantee is load-bearing and running anything is a side effect. That is a genuine design call, not an obvious win, and it should not ride along in a hardening release.
Collapsing risk_surface and diff at critical. Looked like a duplicate fire over an identical range. It is not: risk_surface fires only on a detector match, and execute.md:262-270 already names the overlap as the reason diff is off at lower levels.
Differentiating the reviewer rungs. They are byte-identical apart from the effort frontmatter, but lib/rung-agent.mjs:150-158 enforces that deliberately. Whether a higher rung actually finds more is unmeasurable until item 1 lands.

Four changes to the review subsystem. The first costs nothing and is what makes the rest measurable, so it lands first. ## 1. Record kills, not just survivors (do this first, zero tokens) `references/review-triggers.md:238` records the adjudication as: ``` "<trigger>: <n> survivors; voices <...>" ``` `lib/trace-suggest.mjs:118` (rule R1) then proposes turning a gate OFF when `survivors === 0` across two or more fires. But 0-of-0 and 0-of-9 are opposite facts: - 0 raised, 0 survived: the gate found nothing, and may genuinely be unnecessary - 9 raised, 0 survived: the reviewer is miscalibrated, and the gate is doing real work catching it Both produce the identical suggestion today. The re-arm veto softens this but does not separate the two. Change the detail to `<n> survivors of <m> raised`. That is the whole change, and every item below becomes measurable rather than asserted once it lands. ## 2. Send the contract to the cross-model arm (highest value per byte) `review-provider.mjs:830`: ```js system: payload.instruction, user: payload.artifact ``` `instruction` is one model-authored line. So an external reviewer's ENTIRE system prompt is that line. It never sees `<stance>`, never sees `<what_to_look_for>`, never sees the severity definitions, never sees "Approach differences are NOT findings", and never sees that an empty `findings: []` is a valid result. It also has no repo access by construction. Every cross-model finding is therefore uncalibrated against our own severity bar. With `review.mode` at `panel` or `single` and a `blocking` gate, that output can FAIL a gate on a bar the model invented. Fix: compose `instruction` in `fire()` step 2 as stance + what_to_look_for + severity definitions + the trigger-specific ask. Roughly +600 tokens against a 120,000 token cap, which is half a percent. Prose change in `references/review-triggers.md`, no code, no schema. ## 3. Hand the reviewer the criteria it is told to check The contract at `skills/cad-reviewer-contract/SKILL.md:44-47` demands the reviewer find "a requirement with no task, a task that does not deliver its requirement, a 'done' truth no task makes true, a contradicted locked decision". The `plan` trigger's payload is the PLAN path alone. The sibling gate does it right: `workflows/plan.md` hands cad-plan-checker `PLAN*.md, ROADMAP.md, REQUIREMENTS.md, CONTEXT.md` and names all four in the prompt. The reviewer is asked the same questions with a quarter of the evidence. Fix: add ROADMAP, REQUIREMENTS and CONTEXT to the `plan` payload, and the phase CONTEXT plus PLAN as a second reference on `diff`, `phase_diff` and `pre_ship`. These are references, not inlined text, so the orchestrator cost is about zero. ## 4. Give the reviewer a process with a self-refutation step cad-reviewer is the ONLY role contract in the subsystem with no `<process>` section. plan-checker, verifier and executor all have one. Worse, `<guardrails>` reads "One pass. Report everything you find now; there is no second look." Read literally that discourages self-refutation, because refuting your own candidate findings IS a second look at them. The one line that asserted falsification was deleted in `49ba72e`: ``` - [ ] A genuine falsification attempt was made, grounded against the real files ``` `design-notes/sweep-2026-08-10-context-weight.md:46` framed that cut as "easiest to reverse if compliance drops". This is the reversal condition. It was about 200 bytes and it was the subsystem's only self-check assertion. Add: resolve the reference, generate candidates, open the cited file at the cited line and try to KILL each candidate, return only survivors carrying the check that failed to kill them. About +250 tokens per dispatch. It finds nothing new directly; it removes the false positives that bury the real findings, and adjudication only runs at the `adjudicated` gate, so at `advisory` and `blocking` nothing filters them today. Scope the "one pass" guardrail to "no second DISPATCH" so it stops contradicting the new step. ## Deliberately not in scope - **R-05, requiring the reviewer to RUN the failure.** It has Bash and is told to ground findings to a line, never to reproduce them. Making it prove the failure would convert plausible findings into confirmed ones, and the verifier has precedent for bounded execution (`cad-verifier-contract:118-131`, under 10s, no servers, no state mutation, no network). But the reviewer's read-only guarantee is load-bearing and running anything is a side effect. That is a genuine design call, not an obvious win, and it should not ride along in a hardening release. - **Collapsing risk_surface and diff at critical.** Looked like a duplicate fire over an identical range. It is not: `risk_surface` fires only on a detector match, and `execute.md:262-270` already names the overlap as the reason `diff` is off at lower levels. - **Differentiating the reviewer rungs.** They are byte-identical apart from the `effort` frontmatter, but `lib/rung-agent.mjs:150-158` enforces that deliberately. Whether a higher rung actually finds more is unmeasurable until item 1 lands.

john added this to the v3.2.0 milestone

2026-08-13 15:44:54 +00:00

john added the

enhancement

label

2026-08-13 15:44:54 +00:00

john closed this issue

2026-08-14 17:01:43 +00:00

john commented

2026-08-14 17:02:01 +00:00

Author

Owner

Fixed in v3.2.0 (PR #124). RVW-01, phase 3. Adjudication records kills as well as survivors: of raised replaces survivors, so 0-of-0 and 0-of-9 stop collapsing to the same suggestion. The cross-model reviewer gets references/reviewer-brief.md composed into its payload, and the reviewer contract gained a process requiring the cited line be opened and each candidate finding killed before it is reported.

Verified through the phase UAT walk. The milestone audit traced 12/12 requirements with 0 broken and 36/36 acceptance criteria covered.

Fixed in v3.2.0 (PR #124). RVW-01, phase 3. Adjudication records kills as well as survivors: <n> of <m> raised replaces <n> survivors, so 0-of-0 and 0-of-9 stop collapsing to the same suggestion. The cross-model reviewer gets references/reviewer-brief.md composed into its payload, and the reviewer contract gained a process requiring the cited line be opened and each candidate finding killed before it is reported. Verified through the phase UAT walk. The milestone audit traced 12/12 requirements with 0 broken and 36/36 acceptance criteria covered.