crenshawdev/cadence

Fork 0

traceability: give truths and UAT items IDs so cross-artifact drift cannot silently break a merge #70

New issue

Closed

opened 2026-07-28 17:41:31 +00:00 by crenshawdev · 2 comments

crenshawdev commented

2026-07-28 17:41:31 +00:00

(Migrated from github.com)

What you want Cadence to do

Give truths and UAT items stable IDs, so a merge or an audit stops depending on two agents choosing the same words.

The same truth currently passes through six artifacts written at different times by different agents: the ROADMAP goal, CONTEXT acceptance criteria, PLAN's "Must be true when done", the SUMMARY goal check, UAT items, and the verifier's derived truths. If the planner writes "users can authenticate" and the verifier derives "login works", a human reads those as the same thing and the merge logic does not.

Replace the prose rule with a schema property wherever drift breaks a merge or an audit. Keep the prose rule as the fallback for cases where drift is only cosmetic.

Cadence already does exactly this for requirements, so the pattern is proven in-repo rather than imported: REQUIREMENTS.md assigns IDs, plan frontmatter carries requirements, and /cad-audit traces by ID. The verifier's gap payload already carries k, the UAT item number, alongside name.

Cadence also already states the underlying rule twice, locally and from first principles:

agents/cad-verifier.md:67 says to "dedupe toward the ROADMAP wording", designating a canonical source.
agents/cad-verifier.md:155 says the output block's field names are the merge payload's names on purpose, "never invent synonyms it would have to translate."

That second line is one-word-one-meaning, arrived at independently. This proposal generalizes it and makes it structural.

The prose generalization, which stays as the fallback: one artifact owns the canonical phrasing of each thing, and downstream artifacts quote it and add detail after, never rephrasing the thing itself.

Why the default no is wrong here

Because terminology discipline is a compliance problem and IDs are not. A prose rule telling agents to reuse wording fails silently the moment a model paraphrases, and paraphrase is the single most reliable thing a language model does. Nothing in the current design detects the failure. Two artifacts describing the same truth in different words look exactly like two artifacts describing two truths, and the merge treats them as two.

Because this removes a failure class rather than adding a rule that must be obeyed, which is the manifesto-friendlier shape of the two available fixes. The MANIFESTO's objection is to surface that can fail. An ID is surface that cannot drift.

Because the partial version already shipped and works. Requirements have IDs and /cad-audit traces them both directions, catching orphans on either side. Truths and UAT items are the same kind of object crossing the same kind of boundary, with no IDs and therefore no audit. The inconsistency is the argument: the mechanism is already here, already understood, and already load-bearing for one artifact type.

What the current surface cannot earn: an audit that can tell a dropped truth from a renamed one. Today those are indistinguishable, and the failure looks like success in both directions, which is the specific failure shape v1.4.0's stated-grammar work was aimed at everywhere else.

What it costs

A schema change across artifacts that already exist, which is the real cost and it is not small. UAT items already carry a number (k); truths do not. Making truth IDs stable across a re-verify, a --sweep, and a /cad-plan --gaps cycle is where the design work is, and getting it wrong produces IDs that renumber under the user and are worse than none.

What can now fail that could not before. An ID that collides. An ID that renumbers on a phase insert, which /cad-phase already has to handle for phases and would now have to handle for truths. A merge that matches on a stale ID and silently pairs the wrong two items, which is a quieter failure than the drift it replaces.

Scope discipline. This should extend IDs only where drift actually breaks a merge or an audit. Every artifact that gets an ID it does not need is bookkeeping the user pays for. Name the boundaries in the design, and be willing to leave the SUMMARY goal check as prose if nothing consumes it structurally.

The measurement. Count merge mismatches attributable to wording drift, from existing UAT.md files and verifier outputs across completed phases. If that count is near zero across the shipped history, this proposal is solving a hypothetical and should be rejected. That data exists in .planning/ and has not been counted, and it should be counted before any of this is scheduled.

Why this belongs in `main` and not a fork

The drift happens between Cadence's own agents, in Cadence's own artifacts, on the default path. A user never writes the truths or the UAT items; the planner, the verifier, and /cad-verify do. There is no workflow preference here to fork over.

The merge logic is in main, the audit is in main, and the half-implementation is in main: requirements have IDs and nothing else does. Anyone who hits the failure hits the same one, and no user-side configuration reaches it.

The writing-discipline proposal is the other half of the same analysis. It stays a prose rule because the drift it addresses is cosmetic and degrades gracefully; this half is structural because it does not.
The verifier's L4 hollow-seam data-flow trace (#14) touches the same truth objects. If both land in one cycle, sequence this first so the trace has stable IDs to attach evidence to.

## What you want Cadence to do Give truths and UAT items stable IDs, so a merge or an audit stops depending on two agents choosing the same words. The same truth currently passes through six artifacts written at different times by different agents: the ROADMAP goal, CONTEXT acceptance criteria, PLAN's "Must be true when done", the SUMMARY goal check, UAT items, and the verifier's derived truths. If the planner writes "users can authenticate" and the verifier derives "login works", a human reads those as the same thing and the merge logic does not. Replace the prose rule with a schema property wherever drift breaks a merge or an audit. Keep the prose rule as the fallback for cases where drift is only cosmetic. Cadence already does exactly this for requirements, so the pattern is proven in-repo rather than imported: `REQUIREMENTS.md` assigns IDs, plan frontmatter carries `requirements`, and `/cad-audit` traces by ID. The verifier's gap payload already carries `k`, the UAT item number, alongside `name`. Cadence also already states the underlying rule twice, locally and from first principles: - `agents/cad-verifier.md:67` says to "dedupe toward the ROADMAP wording", designating a canonical source. - `agents/cad-verifier.md:155` says the output block's field names are the merge payload's names on purpose, "never invent synonyms it would have to translate." That second line is one-word-one-meaning, arrived at independently. This proposal generalizes it and makes it structural. The prose generalization, which stays as the fallback: one artifact owns the canonical phrasing of each thing, and downstream artifacts quote it and add detail after, never rephrasing the thing itself. ## Why the default no is wrong here **Because terminology discipline is a compliance problem and IDs are not.** A prose rule telling agents to reuse wording fails silently the moment a model paraphrases, and paraphrase is the single most reliable thing a language model does. Nothing in the current design detects the failure. Two artifacts describing the same truth in different words look exactly like two artifacts describing two truths, and the merge treats them as two. **Because this removes a failure class rather than adding a rule that must be obeyed**, which is the manifesto-friendlier shape of the two available fixes. The MANIFESTO's objection is to surface that can fail. An ID is surface that cannot drift. **Because the partial version already shipped and works.** Requirements have IDs and `/cad-audit` traces them both directions, catching orphans on either side. Truths and UAT items are the same kind of object crossing the same kind of boundary, with no IDs and therefore no audit. The inconsistency is the argument: the mechanism is already here, already understood, and already load-bearing for one artifact type. What the current surface cannot earn: an audit that can tell a dropped truth from a renamed one. Today those are indistinguishable, and the failure looks like success in both directions, which is the specific failure shape v1.4.0's stated-grammar work was aimed at everywhere else. ## What it costs **A schema change across artifacts that already exist**, which is the real cost and it is not small. UAT items already carry a number (`k`); truths do not. Making truth IDs stable across a re-verify, a `--sweep`, and a `/cad-plan --gaps` cycle is where the design work is, and getting it wrong produces IDs that renumber under the user and are worse than none. **What can now fail that could not before.** An ID that collides. An ID that renumbers on a phase insert, which `/cad-phase` already has to handle for phases and would now have to handle for truths. A merge that matches on a stale ID and silently pairs the wrong two items, which is a quieter failure than the drift it replaces. **Scope discipline.** This should extend IDs only where drift actually breaks a merge or an audit. Every artifact that gets an ID it does not need is bookkeeping the user pays for. Name the boundaries in the design, and be willing to leave the SUMMARY goal check as prose if nothing consumes it structurally. **The measurement.** Count merge mismatches attributable to wording drift, from existing UAT.md files and verifier outputs across completed phases. If that count is near zero across the shipped history, this proposal is solving a hypothetical and should be rejected. That data exists in `.planning/` and has not been counted, and it should be counted before any of this is scheduled. ## Why this belongs in `main` and not a fork The drift happens between Cadence's own agents, in Cadence's own artifacts, on the default path. A user never writes the truths or the UAT items; the planner, the verifier, and `/cad-verify` do. There is no workflow preference here to fork over. The merge logic is in `main`, the audit is in `main`, and the half-implementation is in `main`: requirements have IDs and nothing else does. Anyone who hits the failure hits the same one, and no user-side configuration reaches it. ## Related - The writing-discipline proposal is the other half of the same analysis. It stays a prose rule because the drift it addresses is cosmetic and degrades gracefully; this half is structural because it does not. - The verifier's L4 hollow-seam data-flow trace (#14) touches the same truth objects. If both land in one cycle, sequence this first so the trace has stable IDs to attach evidence to.

crenshawdev commented

2026-07-28 17:41:50 +00:00

(Migrated from github.com)

Cross-links:

#69 is the other half of the same analysis, kept as a prose rule because its drift is cosmetic.
#14 (verifier L4 hollow-seam data-flow trace) touches the same truth objects. If both land in one cycle, sequence this first so the trace has stable IDs to attach evidence to.

Cross-links: - #69 is the other half of the same analysis, kept as a prose rule because its drift is cosmetic. - #14 (verifier L4 hollow-seam data-flow trace) touches the same truth objects. If both land in one cycle, sequence this first so the trace has stable IDs to attach evidence to.

crenshawdev commented

2026-07-28 19:12:37 +00:00

(Migrated from github.com)

Ran the kill-measurement. Closing on it.

This issue named its own rejection condition: "Count merge mismatches attributable to wording drift, from existing UAT.md files and verifier outputs across completed phases. If that count is near zero across the shipped history, this proposal is solving a hypothetical and should be rejected."

The count is zero.

Corpus

Every completed phase was recovered from git, since /cad-milestone prunes them and .planning/phases is empty. Complete history: 5 milestone cycles (v1.1.0-rc.1, v1.1.0-rc.2, v1.2.0, v1.3.1, v1.4.0), 20 completed phases, 181 UAT items, 122 CONTEXT acceptance criteria, 139 verifier findings merged across the 18 phases where a deep pass ran.

0 of 139 merged findings (0%). With zero events in 139 trials the honest bound is below roughly 2% (rule of three), not exactly zero. Of the 139, 123 matched an existing UAT item in place and 16 were appended; all 16 appends were inspected individually and every one is a genuinely new gap rather than a renaming.

Why it is zero, which matters more than the count

The premise is that six artifacts describe the same truth and a matcher treats rewordings as different truths. Five of the six boundaries have no matcher at all. ROADMAP to CONTEXT to PLAN to SUMMARY to UAT are LLM authoring steps. Nothing in cadence-core/bin/ ever compares an acceptance-criterion string to a UAT item string.

Exactly one automated matcher exists, at cadence-core/bin/planning.mjs:485-488:

const find = (ref) => uat.items.find((i) => {
  if (ref.k !== undefined && Number(i.k) === Number(ref.k)) return true;
  const name = usableName(i);
  return name !== null && name === usableName(ref);
});

It keys on k and only falls back to exact name. And k is not independently invented by two agents: verify-deep.md hands the verifier the existing UAT items and cad-verifier.md requires it to echo k back, so the verifier is annotating a list it was given rather than naming truths freely. The stable ID this proposal asks for already exists on the only boundary where a merge can break. That is a structural reason, and it does not change with more data.

Prose paraphrase, meanwhile, is close to universal: only 3 of 122 UAT item names appear verbatim inside their acceptance criterion. None of it was counted, because no code reads it. A rewording no matcher ever sees cannot mismatch.

For completeness, the one real merge mismatch in the repo's history was 5d4b07f, where find compared names untrimmed while the append path trimmed. Real instance of this failure shape, but the cause was a normalizer asymmetry rather than drift, and it was closed structurally.

What the measurement did find, which is a different issue

Two acceptance criteria in v1.4.0 phase 1 got no UAT item at all on the first checklist build, appearing only in a second /cad-verify pass. That is 2 of 122 (1.6%) silently dropped. It is the "a dropped truth is indistinguishable from success" failure this issue uses in its rationale, but the cause is omission during extraction, not drift during matching, and IDs on truths and UAT items would not have caught it. IDs on CONTEXT acceptance criteria would. Filed as its own issue.

Limits, stated plainly

Verifier outputs are never persisted. cad-verifier.md requires findings in the final message and written to no file, so the exact comparison this issue asks for is not directly available for any phase. The above is measured from its residue in UAT.md. Filed separately, because it is the precondition for ever re-testing this honestly.
Small n, one project. 139 merges over 13 days, one author, one model family, on Cadence's own repo. Distinguishing a 0% rate from a 3% rate needs roughly 300-500 merges.
Dogfood bias runs toward this proposal, not away from it. This repo's phases are unusually well specified. A user with thin CONTEXT files hits the verifier's derive-from-goal branch, where it invents truth names with no list to anchor to. That is the regime where drift would appear and it is untested here.

Verdict

Rejected as scoped, on this issue's own terms. Reopen if the thin-CONTEXT regime produces evidence, which is the one place the conclusion is only moderately confident.

## Ran the kill-measurement. Closing on it. This issue named its own rejection condition: *"Count merge mismatches attributable to wording drift, from existing UAT.md files and verifier outputs across completed phases. If that count is near zero across the shipped history, this proposal is solving a hypothetical and should be rejected."* **The count is zero.** ## Corpus Every completed phase was recovered from git, since `/cad-milestone` prunes them and `.planning/phases` is empty. Complete history: **5 milestone cycles** (v1.1.0-rc.1, v1.1.0-rc.2, v1.2.0, v1.3.1, v1.4.0), **20 completed phases**, **181 UAT items**, **122 CONTEXT acceptance criteria**, **139 verifier findings merged** across the 18 phases where a deep pass ran. **0 of 139 merged findings (0%).** With zero events in 139 trials the honest bound is *below roughly 2%* (rule of three), not exactly zero. Of the 139, 123 matched an existing UAT item in place and 16 were appended; all 16 appends were inspected individually and every one is a genuinely new gap rather than a renaming. ## Why it is zero, which matters more than the count The premise is that six artifacts describe the same truth and a matcher treats rewordings as different truths. **Five of the six boundaries have no matcher at all.** ROADMAP to CONTEXT to PLAN to SUMMARY to UAT are LLM authoring steps. Nothing in `cadence-core/bin/` ever compares an acceptance-criterion string to a UAT item string. Exactly one automated matcher exists, at `cadence-core/bin/planning.mjs:485-488`: ```js const find = (ref) => uat.items.find((i) => { if (ref.k !== undefined && Number(i.k) === Number(ref.k)) return true; const name = usableName(i); return name !== null && name === usableName(ref); }); ``` It keys on `k` and only falls back to exact name. And `k` is not independently invented by two agents: `verify-deep.md` hands the verifier the existing UAT items and `cad-verifier.md` requires it to echo `k` back, so the verifier is annotating a list it was given rather than naming truths freely. **The stable ID this proposal asks for already exists on the only boundary where a merge can break.** That is a structural reason, and it does not change with more data. Prose paraphrase, meanwhile, is close to universal: only 3 of 122 UAT item names appear verbatim inside their acceptance criterion. None of it was counted, because no code reads it. A rewording no matcher ever sees cannot mismatch. For completeness, the one real merge mismatch in the repo's history was `5d4b07f`, where `find` compared names untrimmed while the append path trimmed. Real instance of this failure *shape*, but the cause was a normalizer asymmetry rather than drift, and it was closed structurally. ## What the measurement did find, which is a different issue Two acceptance criteria in v1.4.0 phase 1 got **no UAT item at all** on the first checklist build, appearing only in a second `/cad-verify` pass. That is 2 of 122 (1.6%) silently dropped. It is the "a dropped truth is indistinguishable from success" failure this issue uses in its rationale, but the cause is **omission during extraction, not drift during matching**, and IDs on truths and UAT items would not have caught it. IDs on CONTEXT acceptance criteria would. Filed as its own issue. ## Limits, stated plainly - **Verifier outputs are never persisted.** `cad-verifier.md` requires findings in the final message and written to no file, so the exact comparison this issue asks for is not directly available for any phase. The above is measured from its residue in UAT.md. Filed separately, because it is the precondition for ever re-testing this honestly. - **Small n, one project.** 139 merges over 13 days, one author, one model family, on Cadence's own repo. Distinguishing a 0% rate from a 3% rate needs roughly 300-500 merges. - **Dogfood bias runs toward this proposal, not away from it.** This repo's phases are unusually well specified. A user with thin CONTEXT files hits the verifier's derive-from-goal branch, where it invents truth names with no list to anchor to. That is the regime where drift would appear and it is untested here. ## Verdict Rejected as scoped, on this issue's own terms. Reopen if the thin-CONTEXT regime produces evidence, which is the one place the conclusion is only moderately confident.