verify: persist the verifier's findings block so its behaviour is measurable #77

New issue

Closed

opened 2026-07-28 19:12:40 +00:00 by crenshawdev · 1 comment

crenshawdev commented

2026-07-28 19:12:40 +00:00

(Migrated from github.com)

Split out of #70. This is a measurement-infrastructure gap, not a feature.

What happened

agents/cad-verifier.md requires the verifier to return its findings in its final message and to write them to no file. A history-wide search confirms no such artifact has ever existed.

The consequence surfaced while running #70's kill-measurement: the comparison #70 asked for could not be performed directly for any phase. The measurement wanted verifier-derived truth names against UAT item names. The verifier's original wording is gone the moment its dispatch ends. What survives in UAT.md is residue: source: verifier, the assigned k, and the expected == reported signature that marks an appended gap. That residue recovers whether a match succeeded, but not what the verifier actually called the thing when it did.

So the question "does wording drift break merges" was answerable only by inference, and the same will be true of every future question about verifier behaviour.

What you want Cadence to do

Persist the verifier's findings block. verify-deep.md already instructs the model to report an auto_passed / gaps / added / skipped / rejected envelope, so the payload exists and is already structured. Write it to disk next to the phase's other artifacts instead of discarding it after the merge.

Why it is worth doing

It makes an entire class of question answerable. Not just #70's. Verifier UNCERTAIN rate by effort rung is a measurement #63 names and cannot currently run. Any future claim about how the verifier behaves is currently unfalsifiable for the same reason.

It is nearly free. The data is already produced and already structured. This is a write, not a new computation.

It is the precondition for reopening #70 honestly. #70 was rejected on a zero count that had to be inferred from residue, with the caveat that thin-CONTEXT projects are the untested regime. Without persistence, re-testing that next cycle means inferring again.

What it costs

Artifact growth in .planning/. Every deep verify pass adds a file. /cad-milestone prunes completed phases, so these would be pruned with them and live in git like everything else, but the size question should be sized rather than assumed.

A schema to keep stable. A persisted artifact is one more thing that can drift from what writes it. If it is written, self-verify should assert its shape.

The counter-argument. Cadence deliberately does not persist everything, and "we might want to measure it later" justifies almost any file. What makes this one different is that a measurement was actually attempted, actually blocked, and the blocked question was gating a milestone decision.

#70, rejected, where the gap surfaced.
#63, whose verifier-rung measurement has the same blocker.

Split out of #70. This is a measurement-infrastructure gap, not a feature. ## What happened `agents/cad-verifier.md` requires the verifier to return its findings in its final message and to write them to no file. A history-wide search confirms no such artifact has ever existed. The consequence surfaced while running #70's kill-measurement: **the comparison #70 asked for could not be performed directly for any phase.** The measurement wanted verifier-derived truth names against UAT item names. The verifier's original wording is gone the moment its dispatch ends. What survives in `UAT.md` is residue: `source: verifier`, the assigned `k`, and the `expected == reported` signature that marks an appended gap. That residue recovers *whether a match succeeded*, but not what the verifier actually called the thing when it did. So the question "does wording drift break merges" was answerable only by inference, and the same will be true of every future question about verifier behaviour. ## What you want Cadence to do Persist the verifier's findings block. `verify-deep.md` already instructs the model to report an `auto_passed` / `gaps` / `added` / `skipped` / `rejected` envelope, so the payload exists and is already structured. Write it to disk next to the phase's other artifacts instead of discarding it after the merge. ## Why it is worth doing **It makes an entire class of question answerable.** Not just #70's. Verifier UNCERTAIN rate by effort rung is a measurement #63 names and cannot currently run. Any future claim about how the verifier behaves is currently unfalsifiable for the same reason. **It is nearly free.** The data is already produced and already structured. This is a write, not a new computation. **It is the precondition for reopening #70 honestly.** #70 was rejected on a zero count that had to be inferred from residue, with the caveat that thin-CONTEXT projects are the untested regime. Without persistence, re-testing that next cycle means inferring again. ## What it costs **Artifact growth in `.planning/`.** Every deep verify pass adds a file. `/cad-milestone` prunes completed phases, so these would be pruned with them and live in git like everything else, but the size question should be sized rather than assumed. **A schema to keep stable.** A persisted artifact is one more thing that can drift from what writes it. If it is written, self-verify should assert its shape. **The counter-argument.** Cadence deliberately does not persist everything, and "we might want to measure it later" justifies almost any file. What makes this one different is that a measurement was actually attempted, actually blocked, and the blocked question was gating a milestone decision. ## Related - #70, rejected, where the gap surfaced. - #63, whose verifier-rung measurement has the same blocker.

john added this to the (deleted) milestone

2026-07-30 05:13:38 +00:00

john modified the milestone from (deleted) to v2.1.0

2026-07-30 05:15:35 +00:00

john commented

2026-08-05 17:08:45 +00:00

Owner

Closed by v2.1.0 (COV-01), extended by v2.3.0 (RES-02). The findings are now persisted twice over.

cad-verifier writes .planning/phases/<N>/verifier-findings.json in the uat merge payload shape and returns a digest plus that path. Separately, the seam persists its own counters — plus rejected_entries and skipped_entries verbatim — to .planning/phases/<N>/FINDINGS.json on every merge, so a finding counted and then dropped survives the dispatch that produced it.

The two filenames are deliberately distinct: the seam owns FINDINGS.json and overwrites it on every successful merge, so a verifier writing that name would destroy its own input.

Verified against the tree at the v2.3.0 close: cadence-core/workflows/verify-deep.md:9,23,39.

Closed by v2.1.0 (COV-01), extended by v2.3.0 (RES-02). The findings are now persisted twice over. `cad-verifier` writes `.planning/phases/<N>/verifier-findings.json` in the `uat merge` payload shape and returns a digest plus that path. Separately, the seam persists its own counters — plus `rejected_entries` and `skipped_entries` verbatim — to `.planning/phases/<N>/FINDINGS.json` on every merge, so a finding counted and then dropped survives the dispatch that produced it. The two filenames are deliberately distinct: the seam owns `FINDINGS.json` and overwrites it on every successful merge, so a verifier writing that name would destroy its own input. Verified against the tree at the v2.3.0 close: `cadence-core/workflows/verify-deep.md:9,23,39`.

john closed this issue

2026-08-05 17:08:58 +00:00