SENTINEL SIGNAL

Sentinel Signal

Sentinel Policy M6: evidence and confidence (v1.0.679)

Source: docs/sentinel-policy-m6-evidence-and-confidence-v1.0.679.md

All documents Docs Home Home

Document Content

Sentinel Policy M6: evidence and confidence (v1.0.679)

What shipped

M6 (EVD-001..006, CNF-001..006, LED-001/003 minimal) — the exit criterion: "No event can enter PUBLISHED state without verified evidence."

  • A real bug fixed, found during M6 planning, not after deploy: M5's

    • _verify_evidence() only checked the CURRENT version's section text. A genuinely correct extraction of a removal cites text that existed in the PREVIOUS version and is gone from the current one — that quote could never verify, so a correct removal extraction was silently dropped into NO_VERIFIABLE_EVIDENCE. Fixed by checking both sides (extraction.py's new _verify_evidence_either_side), recording which side matched as evidence_side on ExtractedFact — this is EVD-004's previous_evidence/current_evidence distinction. A dedicated regression test proves it: a real phrase from NCD 108's v1 text (absent from v2) now verifies correctly instead of being dropped.

  • **EvidenceSpan** (EVD-001..006): formal, immutable promotion of a

    • verified ExtractedFact, with the document/hash lineage EVD-002 wants (document_asset_id, raw_sha256, canonical_content_hash, source_url) traced all the way back through policy_section → policy_version → normalized_document → document_asset.

  • **confidence.py** (CNF-001..006): seven deterministic, weighted

    • components — including a real one this data source uniquely supports: cross-checking the LLM's affected_codes against M4's independently- computed regex-based code_diffs for the same diff ("deterministic corroboration," CNF-002's own named component). Verified live against real NCD 110 data: a mocked extraction reporting the same real CPT-33246 removal M4 already found scored full corroboration (1.0); a mismatched fake code scored 0.0.

  • **ConfidenceThreshold** (CNF-004/006): DB-backed, tunable without code

    • changes, most-specific-match lookup (source+type > type > global). Migration seeds a global default (0.85 auto-publish / 0.6 review) plus stricter rows for the spec's named critical types (0.95/0.75).

  • **ChangeEvent** (LED-001/003, minimal): one immutable ledger row per

    • ExtractedFact, regardless of outcome — REJECTED rows are real, inspectable records too, not discarded. supersedes_change_event_id exists schema-only (LED-002's correction workflow is M8).

  • Runs automatically, inline, right after extraction (unlike M5's own

    • manual-trigger LLM call) — this stage is free and fully deterministic.

  • New GET /v1/change-events (optional ?status= filter) — the flat,

    • cross-policy query LED-004 asks for, alongside a nested change_event under each fact in the existing GET /v1/policies/{id} view.

Verified against real production data, two ways

  1. A high-confidence, real-corroboration case (NCD 110, the same CPT

    2. removal M4 already found live): confidence 1.0, AUTO_PUBLISHED.

  2. A deliberately low-confidence, critical-type case (coverage_removed,

    3. mismatched code, missing effective date, needed a schema-retry): confidence 0.725, correctly REJECTED under coverage_removed's stricter 0.75 review threshold (would have cleared the 0.6 global default). Confirms CNF-004's stricter-threshold requirement is real, not just seeded and unused.

Both confirmed through the actual GET /v1/change-events endpoint against real Postgres, not just direct DB queries.

Verification

  • PYTHONPATH=policy/src:policy/tests python -m pytest policy/tests -q —

    • 94 passed (up from 80). New: test_confidence.py (each component in isolation, threshold lookup including the real fallback-when-DB-empty path), test_ledger.py (the removal-evidence regression test, one ChangeEvent per fact regardless of outcome, idempotent re-extraction).

  • Migration verified against real local PostgreSQL, including confirming

    • the seeded threshold rows landed correctly.

  • Full pipeline verified end-to-end against real Postgres using the real

    • NCD 110 diff, both scenarios above, LLM call mocked (still no ANTHROPIC_API_KEY provisioned, per the M5 decision).

Still pending

The true exit-criterion check — everything above but with a real, non-mocked M5 extraction — is still gated behind provisioning SENTINEL_POLICY_ANTHROPIC_API_KEY, same as noted in M5. M7 (golden evaluation harness) is the next milestone.