self-verify: absent surfaces/budgets/internals are silently skipped, leaving checks green with zero coverage #44

New issue

Closed

opened 2026-07-23 14:50:18 +00:00 by crenshawdev · 0 comments

crenshawdev commented

2026-07-23 14:50:18 +00:00

(Migrated from github.com)

Severity: medium · Area: self-verify · surfaced in the post-v1.2.0 cross-model review sweep. self-verify is a dev/CI drift linter (not runtime), which caps the blast radius.

What happens

Several drift checks are guarded by "if this input exists, check it; otherwise skip" — and a skip is invisible and still counts as a pass. So a whole class of drift can silently stop being guarded while CI stays green:

A surface directory that is renamed or moved (e.g. cadence-core/workflows) is silently skipped, so its config-key / invocation / path checks run against nothing.
Deleting or renaming weight-budgets.json silently drops the entire context-weight enforcement.
A missing INTERNALS.md silently skips the repo-path citation check.

Mechanism (verified against source)

cadence-core/bin/self-verify.mjs:113-114: for (const d of dirs) { if (!existsSync(d)) continue; ... } — an absent surface dir contributes nothing and is not reported.
self-verify.mjs:246-247: if (existsSync(budgetPath)) { ... } — an absent budgets manifest skips check 4 entirely.
self-verify.mjs:222: if (existsSync(internals)) { ... } — same for the INTERNALS citation check.

All three still yield ok:true with no note that a check was skipped.

Not a defect (for contrast)

The config.schema.json load at self-verify.mjs:149 is deliberately unguarded and fails loud on a corrupt schema — correct, and the model to follow.

Fix direction

Distinguish "surface legitimately absent" from "surface expected but missing." For the always-expected inputs (the core surface dirs, the shipped budgets manifest), a missing input should surface as a problem (or at least a skipped-check note in the output) rather than a silent pass, so a class of checks can't drop to zero coverage while green.

**Severity:** medium · **Area:** self-verify · surfaced in the post-v1.2.0 cross-model review sweep. `self-verify` is a dev/CI drift linter (not runtime), which caps the blast radius. ## What happens Several drift checks are guarded by "if this input exists, check it; otherwise skip" — and a skip is invisible and still counts as a pass. So a whole class of drift can silently stop being guarded while CI stays green: - A surface directory that is renamed or moved (e.g. `cadence-core/workflows`) is silently skipped, so its config-key / invocation / path checks run against nothing. - Deleting or renaming `weight-budgets.json` silently drops the entire context-weight enforcement. - A missing `INTERNALS.md` silently skips the repo-path citation check. ## Mechanism (verified against source) - `cadence-core/bin/self-verify.mjs:113-114`: `for (const d of dirs) { if (!existsSync(d)) continue; ... }` — an absent surface dir contributes nothing and is not reported. - `self-verify.mjs:246-247`: `if (existsSync(budgetPath)) { ... }` — an absent budgets manifest skips check 4 entirely. - `self-verify.mjs:222`: `if (existsSync(internals)) { ... }` — same for the INTERNALS citation check. All three still yield `ok:true` with no note that a check was skipped. ## Not a defect (for contrast) The `config.schema.json` load at `self-verify.mjs:149` is deliberately **unguarded** and fails loud on a corrupt schema — correct, and the model to follow. ## Fix direction Distinguish "surface legitimately absent" from "surface expected but missing." For the always-expected inputs (the core surface dirs, the shipped budgets manifest), a missing input should surface as a problem (or at least a skipped-check note in the output) rather than a silent pass, so a class of checks can't drop to zero coverage while green.