verifier: abstain (UNCERTAIN) instead of guessing PASS/FAIL #15

New issue

Closed

opened 2026-07-18 22:05:16 +00:00 by crenshawdev · 1 comment

crenshawdev commented

2026-07-18 22:05:16 +00:00

(Migrated from github.com)

Tier: Improve Cadence (ranked #2) · cheap hardening

Teach the verifier to return UNCERTAIN on any check whose answer isn't derivable from the spec, rather than emit a confident PASS/FAIL.

Why: hardens the verify gate against the exact failure mode it exists to prevent, false confidence. Abstain, don't guess.

Cost: one paragraph in the agents/cad-verifier.md contract. (This is also the PORT "honest-verifier reference" item, folded in here.)

Source: internal GSD comparison note, 2026-07-18.

**Tier:** Improve Cadence (ranked #2) · cheap hardening Teach the verifier to return **UNCERTAIN** on any check whose answer isn't derivable from the spec, rather than emit a confident PASS/FAIL. **Why:** hardens the verify gate against the exact failure mode it exists to prevent, false confidence. Abstain, don't guess. **Cost:** one paragraph in the `agents/cad-verifier.md` contract. (This is also the PORT "honest-verifier reference" item, folded in here.) _Source: internal GSD comparison note, 2026-07-18._

crenshawdev commented

2026-07-28 17:02:50 +00:00

(Migrated from github.com)

Already shipped. The abstain contract this asks for is in agents/cad-verifier.md today, and predates the issue (the file has not changed since 14e8fb1, 2026-07-16, two days before this was filed against a GSD-core comparison note).

What is there:

UNCERTAIN is a first-class classification alongside VERIFIED and FAILED, defined as "only a human can settle it: visual, feel, external service, live behavior with no runnable probe".
The abstain rule is stated as a guardrail: "Evidence for every status, a truth without cited evidence is UNCERTAIN, not VERIFIED." That is exactly "abstain, do not guess a confident PASS".
Level 4 (Behaves) routes there by default: a truth hinging on runtime behavior upgrades to VERIFIED only on a named passing test or an observed spot-check, "otherwise the truth is UNCERTAIN and becomes a human check".
The verdict rule honors it: any UNCERTAIN truth or human-only check yields needs_human, and scoring counts UNCERTAIN toward neither side, so abstaining can never inflate a clean score.
The over-abstain direction is guarded too, in the stance block: "Marking UNCERTAIN when absence is observable, that is FAILED."

Closing as covered. The genuine verifier gap from that same comparison note is the L4 data-flow trace, which stays open as #14.

Already shipped. The abstain contract this asks for is in `agents/cad-verifier.md` today, and predates the issue (the file has not changed since 14e8fb1, 2026-07-16, two days before this was filed against a GSD-core comparison note). What is there: - UNCERTAIN is a first-class classification alongside VERIFIED and FAILED, defined as "only a human can settle it: visual, feel, external service, live behavior with no runnable probe". - The abstain rule is stated as a guardrail: "Evidence for every status, a truth without cited evidence is UNCERTAIN, not VERIFIED." That is exactly "abstain, do not guess a confident PASS". - Level 4 (Behaves) routes there by default: a truth hinging on runtime behavior upgrades to VERIFIED only on a named passing test or an observed spot-check, "otherwise the truth is UNCERTAIN and becomes a human check". - The verdict rule honors it: any UNCERTAIN truth or human-only check yields `needs_human`, and scoring counts UNCERTAIN toward neither side, so abstaining can never inflate a clean score. - The over-abstain direction is guarded too, in the stance block: "Marking UNCERTAIN when absence is observable, that is FAILED." Closing as covered. The genuine verifier gap from that same comparison note is the L4 data-flow trace, which stays open as #14.