Sentinel Policy M4: deterministic diff engine (v1.0.677)

What shipped

M4 (DIF-001..008) — deterministic text/date/numeric/code/table diffing between consecutive PolicyVersions, computed and persisted automatically whenever policyintel.py's resolve_policy_version() creates a new version. No LLM involved anywhere in this milestone (AGENT-004).

• **align_sections()**: matches PolicySections across versions by

source_coordinate (M3's CMS field name — a stable key for API-sourced content), falling back to difflib heading similarity (SEC-004) only when a coordinate doesn't find an exact match.

• **diff_text/diff_dates/diff_numeric/diff_codes**: stdlib-only

(difflib/re), operating on aligned section-pair text. Code detection deliberately follows the spec's stated precision-over-recall principle — only distinctive shapes (HCPCS, ICD-10-CM) or keyword-gated ambiguous shapes (CPT code NNNNN, modifier XX, revenue code NNNN, DRG NNN) are matched; a bare 5-digit number is never treated as a candidate CPT code.

• **diff_table()**: built and unit-tested against a synthetic fixture

only. Direct sampling of several real live CMS NCD detail responses during planning (ncdid 108, 9, 110, 372, 374) found zero HTML tables — every field is prose. The contract exists for a future table-bearing source; it is honestly not verified against real production content in this milestone.

• **CandidateDiff**: new table, one row per `(previous_version_id,

version_id) pair. compute_candidate_diff() fast-paths to has_material_change=False with empty diff lists when the two versions' canonical_content_hash` values match exactly — the real CMS "administrative republish, no policy changes" case (confirmed via M2's own captured fixture data) that VER-007's CMS-authoritative version numbering makes routine, not hypothetical.

• Minimal internal verification surface: GET /v1/policies/{policy_id}

now nests candidate_diff inside each PolicyVersion entry (null for a policy's first version). Not the M9 customer /v1/policies/{id}/diff contract — same minimal admin-gated pattern M3 established.

A real bug found via the diff_table unit test

difflib.SequenceMatcher requires hashable sequence elements; diff_table()'s row-major list[list[str]] input is unhashable as-is. Caught immediately by the first diff_table unit test (TypeError: cannot use 'list' as a dict key) before this ever reached integration testing — fixed by diffing a tuple view of the rows and indexing back into the original lists for output.

Verification

• PYTHONPATH=policy/src:policy/tests python -m pytest policy/tests -q —

66 passed (up from 48), including per-category unit tests, section alignment tests (exact match, fallback similarity, pure addition/ removal), and full-pipeline integration tests reusing M3's CMS fixtures: a materially different item_service_description produces a populated text diff; a synthetic administrative-republish pair (identical canonical content, bumped document_version) produces has_material_change=False; a policy's first version has no CandidateDiff row at all.

• Migration verified against real local PostgreSQL.
• End-to-end verification against the real live CMS API, not just

fixtures: found a real NCD (110, Implantable Cardioverter Defibrillators) with two real historical document_versions covering an actual 2003 Medicare coverage expansion, resolved both through the real pipeline against real Postgres, and got a materially correct, human-readable diff out the other end — real text replacements describing the coverage criteria change, real added/removed dates (October 1, 2003 added, January 24, 1986 removed), real numeric candidates (4 weeks, 3 months, 1 year), and a real code removal (CPT 33246). Re-ran the same two documents to confirm the CandidateDiff row count stayed at 1 (idempotent). Confirmed the nested candidate_diff renders correctly through the actual GET /v1/policies/{policy_id} endpoint.

Known limitations / deferred

• diff_table() has no real CMS data to validate against yet (see above)

— will need real verification once a table-bearing source (LCDs, Medicaid) is added, and CMSCoverageAdapter's HTML normalizer would need to preserve table structure before that source's data could even reach this function (currently flattened to plain text at normalize time).

• No taxonomy classification of what kind of change this is yet — that's

M5 (structured semantic intelligence), which will read these CandidateDiff rows as its bounded input rather than raw documents.