SENTINEL SIGNAL

Sentinel Signal

Agent Reliability — Implementation Summary (local-first core)

Source: docs/agent-reliability-implementation-summary.md

All documents Docs Home Home

Document Content

Agent Reliability — Implementation Summary (local-first core)

Companion to docs/agent-reliability-architecture-plan.md (the Phase 0 assessment). This is the first implementation pass, shipped 2026-08-11.

Scope of this pass

The architecture plan's own recommended sequencing (§39: "1. working CLI, 2. snapshot model, 3. semantic diff, 4. policy engine ... 5. GitHub App ... 9. billing") was followed literally. This pass implements the local-first core — everything a developer can use standalone, with no account and no database. The v1.0.554 stabilization update tightened this core without changing product scope. The v1.0.555 update adds the standalone developer distribution and local GitHub Action. The v1.0.556 update adds the hosted persistence/API skeleton behind AGENT_RELIABILITY_ENABLED=false. The v1.0.557 update adds hosted worker and webhook foundations. The v1.0.559 update adds the authenticated installation callback plus Check Run creation/update boundaries. The v1.0.560 update adds real GitHub App RS256 signing, installation-token exchange, and Checks API calls. The v1.0.561 update adds stale-run suppression. The v1.0.562 update adds baseline lifecycle enforcement. The v1.0.564 update adds hosted beta guardrails for cross-tenant access, repository policy override fail-closed behavior, and concurrency limits. The v1.0.566 update adds GitHub contents fetch for repository .verify-agent.yml overrides. The v1.0.567 update wires the Agent Reliability feature flags and GitHub App settings through IONOS verify-web and verify-worker so production route activation is controlled by /etc/sentinel-signal/prod.env. The v1.0.568 update also passes the existing API auth signing key into verify-web, which the authenticated hosted routes require. The v1.0.569 update adds operator runbooks, smoke tooling, /status visibility, and basic Prometheus alerts for the live hosted routes. The v1.0.570 update makes the smoke script use certifi when available for reliable TLS verification in operator Python environments. The v1.0.571 update mounts the IONOS secret directory into Verify web/worker containers and ensures unreadable GitHub App private-key paths degrade to no-op instead of crashing startup. The v1.0.572 update constrains GitHub webhook run enqueue to supported push and pull_request events. The v1.0.573 update hardens ignored webhook response serialization and removes the Check Run enqueue race found during live smoke testing. The v1.0.574 update adds the customer-facing install page, authenticated dashboard/run/evidence HTML, scheduled monitoring, and Ed25519 evidence-signing hooks:

  • Canonical snapshot model + deterministic fingerprinting (Phase 1)
  • Semantic contract diff engine, with independent severity and breaking-change axes (Phase 2)
  • CLI (verify-agent scan / test / diff / evidence) (Phase 3)
  • Repository policy engine (.verify-agent.yml) (Phase 4)
  • Evidence artifacts, JSON + Markdown (Phase 5, minus real signing)
  • Standalone sentinel-verify-agent distribution and actions/verify-agent composite Action
  • Hosted Agent Reliability SQLAlchemy/Alembic model, service layer, feature-flagged FastAPI router, and queued-run API skeleton
  • Hosted run worker execution, public-target safety checks, generic job queue linkage, and GitHub webhook signature/dedupe foundation
  • Hosted GitHub App installation callback and Check Run integration boundary
  • GitHub App Checks API client using mounted private-key configuration, in-memory installation-token caching, queued Check Run creation, and final Check Run updates
  • Stale GitHub run suppression so superseded queued/running jobs cancel before scanning and never overwrite newer Check Run state
  • Run-based baseline promotion, default-branch pass auto-promotion, PR/block/superseded/config/infra promotion guards, and default-branch-only push enqueue
  • Tenant-guarded repository/target/run access tests, repository override settings that replace hosted policy rather than merging, fail-closed override errors before scanning, and open-beta concurrency enforcement (AGENT_RELIABILITY_CONCURRENCY=2)
  • GitHub contents fetch for repository policy overrides at the evaluated commit. Enabling overrides requires contents:read; fetched policy replaces hosted policy entirely and installation tokens remain process-local only.
  • IONOS production wiring for AGENT_RELIABILITY_* environment variables on both web and worker containers.
  • IONOS verify-web auth wiring for the existing TOKEN_SIGNING_KEY so Agent Reliability routes return normal auth failures instead of server errors.
  • Production runbook, no-secret smoke script, /status readiness panel, and route/webhook Prometheus alerts for hosted Agent Reliability.
  • Read-only IONOS secret mount plus startup-safe GitHub private-key loading for the hosted Checks client.
  • Unsupported GitHub webhook events, such as check_suite, are recorded and ignored without enqueueing hosted runs.
  • GitHub Check Run IDs are attached before hosted worker jobs are enqueued, so fast workers can always update the queued Check Run to the final conclusion.
  • Public Agent Reliability onboarding, authenticated workspace dashboard, recent-run list, and evidence HTML views.
  • Per-target scheduled scans through existing target settings and worker scheduler enqueue, preserving beta quotas and the kill switch.
  • Optional Ed25519 evidence signing for the canonical artifact fingerprint when a mounted signing key is configured.

Explicitly deferred, not started: automated GitHub App registration/permission-upgrade UI/marketplace approval flow, external Stripe paid-plan product setup, and public verification-key publication/rotation ceremony. Repository override execution is now wired to GitHub contents fetch and still fails closed when content is missing, invalid, oversized, or unavailable, so hosted policy is never used as an accidental fallback. The new Action runs inside a user's workflow and has no hosted-account dependency.

What shipped

New package: verify/src/mcp_verify/agent_reliability/ — pure-domain modules (snapshots/, diff/, policy/, evidence/) with zero GitHub/CLI/HTTP-framework dependency, plus cli.py as the one integration layer, per the architecture plan's §73/§74 guardrail (verified: nothing in snapshots/diff/policy/evidence imports cli).

Reused from Verify, as identified in the architecture assessment: detect_tool_capabilities/detect_tool_risk_flags/infer_tool_risk_level (validation/service.py) directly classify every tool in a snapshot — this is why a delete_customer tool with destructiveHint: true correctly shows up as capability_class: ["delete", "write"] without any new classification logic. No DB, job queue, or GitHub code was touched — none of it was needed for this pass.

Genuinely new:

  • snapshots/model.py — Snapshot/Tool/AuthPosture/Transport/RuntimeInfo/VerifyMeta, versioned (SNAPSHOT_SCHEMA_VERSION = "1"), with strict Pydantic validation at external load boundaries (extra="forbid", supported-version checks, bounded tool/capability lists, unique normalized tool IDs, and fingerprint verification).
  • snapshots/canonicalize.py — sorted-key, fixed-separator JSON + SHA-256 fingerprinting. contract_fingerprint is computed over the deployment contract (tools/schemas/annotations/auth/transport type) and excludes endpoint, source, timestamps, evidence, latency, and Verify observations. snapshot_id fingerprints the complete sanitized observation.
  • snapshots/redaction.py — rejects endpoint userinfo, redacts sensitive endpoint query values, and converts secret-shaped defaults/examples/descriptions/metadata to [REDACTED] before fingerprinting or serialization.
  • protocol.py + snapshots/collect.py — scan_target() uses a database-free MCP protocol client for initialize, notifications/initialized, session propagation, JSON/SSE response parsing, and tools/list. It validates JSON-RPC result/error semantics rather than treating HTTP 200 as success.
  • diff/severity.py + diff/breaking.py — the two independent classification axes requirements §11/§12 insist must not be collapsed. Every rule is a small, named, directly-testable pure function (e.g. severity_for_new_tool, breaking_for_required_field_added).
  • diff/engine.py — compute_contract_diff(baseline, candidate) -> ContractDiff. Covers tool inventory, capability/risk changes, input/output schema diffs, authentication, transport, latency, Verify score/readiness, and similarity-based rename inference. Clear one-to-one renames are labeled tool_renamed; ambiguous matches remain remove-plus-add and get a tool_rename_ambiguous note.
  • policy/schema.py + policy/evaluate.py — parses .verify-agent.yml as a strict DeploymentPolicy and evaluates a diff into DeploymentPolicyEvaluation, producing pass/warn/approval_required/block with precedence block > approval_required > warn > pass. Legacy Policy/PolicyResult aliases remain for compatibility.
  • evidence/artifact.py — structured JSON artifact + Markdown report. Records engine_version, policy_fingerprint/policy_version, both snapshot_id values, and both contract_fingerprint values (§19/§61/§63). Hosted evidence remains explicitly signed: false unless an Ed25519 evidence-signing key is configured.
  • cli.py — verify-agent scan|test|diff|evidence, with --header-env HEADER=ENV_VAR for authenticated local scans/tests. Header values are read from the environment and are never accepted raw on the command line or serialized into snapshots.
  • verify/src/sentinel_verify_agent/ + packages/sentinel-verify-agent/ — standalone package source and distribution metadata for sentinel-verify-agent, retaining the verify-agent executable.
  • actions/verify-agent/ — composite GitHub Action that installs an exact package version, accepts one candidate source, emits JSON/Markdown evidence, writes a step summary, and maps local decisions into CI outcomes.
  • verify/src/mcp_verify/agent_reliability_hosted/ — hosted router/service/auth adapter and worker status helpers. Routes are registered only when AGENT_RELIABILITY_ENABLED=true; tenant identity is derived from auth context rather than request payloads.
  • verify/alembic/versions/0024_agent_rel_hosted_tables.py — the hosted persistence model for installations, repositories, targets, snapshots, baselines, deployment policy versions, runs, evidence artifacts, and webhook deliveries. Diff and policy-evaluation payloads are stored on runs, not split into separate tables.
  • verify/alembic/versions/0025_agent_rel_run_job_id.py — links hosted Agent Reliability runs to generic queue jobs.
  • agent_reliability_hosted/security.py and worker.py — hosted target safety checks plus queued-run execution. Worker scans use the same sentinel_verify_agent snapshot/diff/policy/evidence engine as the CLI and Action; config_error/infra_error remain status values, not policy decisions.
  • POST /v1/agent-reliability/github/webhook — HMAC-verified delivery receiver that dedupes GitHub deliveries and enqueues one run per active repository target.
  • POST /v1/agent-reliability/github/installations/callback and agent_reliability_hosted/github.py — authenticated installation/repository linking, a GitHub App Checks client with explicit GitHub conclusion mapping, and a GitHub contents fetcher for repository policy overrides. Installation access tokens are short-lived, process-local only, and never persisted.
  • AgentReliabilityRun.superseded_by_run_id — stale-run marker used by webhook enqueue and worker completion paths to prevent older GitHub completions from updating newer Check state.
  • POST /v1/agent-reliability/runs/{id}/baseline/promote — promotes the run's candidate snapshot only when the run is completed, non-superseded, non-PR, and decision-eligible. Default-branch pass push runs auto-promote through the same baseline history model.
  • Repository override settings — disabled by default, normalized through strict request models, require contents:read when using repository files, and are evaluated as a full policy replacement. Missing, oversized, invalid, or fetch-failed override content records config_error before scan and never falls back to hosted policy.
  • docs/agent-reliability-hosted-api.md — current hosted API surface and flag documentation.

Canonical demo (§70)

verify/eval/fixtures/agent_reliability/demo_baseline.json / demo_candidate.json — baseline has search_customer/update_customer; candidate adds a new delete_customer tool (destructiveHint: true) and makes update_customer's email field required. Running the full pipeline:

verify-agent scan --config demo_baseline.json --output baseline.json
verify-agent scan --config demo_candidate.json --output candidate.json
verify-agent diff --baseline baseline.json --candidate candidate.json

produces exactly the requirements doc's expected shape: delete_customer classified CRITICAL/non-breaking, the email field change classified MEDIUM/breaking, decision BLOCK, exit code 3, with violations naming both fail_on.new_destructive_tools and fail_on.breaking_schema_change. This is wired up as an automated integration test (test_agent_reliability_cli_integration.py::test_canonical_demo_end_to_end_scan_diff_evidence_blocks) covering the full local pipeline (scan → diff → policy → evidence) as a release gate, per requirements §76.

Testing

115 focused Agent Reliability tests are passing. Dedicated package coverage remains above the release gates for collector, CLI, policy, protocol, and hosted guardrail modules. Coverage now includes snapshot determinism/order-independence/contract fingerprint stability/version handling/forged-ID rejection/strict keys/secret redaction; protocol JSON-RPC error handling/session propagation/initialized notification/header non-persistence; diff tool-added/removed/rename/ambiguous-rename/schema/capability/auth/transport/score regression; policy pass/warn/approval-required/block/multiple violations/thresholds; evidence fingerprints/policy identity/engine identity/no secrets; hosted tenant isolation, stale-run suppression, baseline lifecycle, repository override fail-closed behavior, GitHub contents policy fetch, and beta quota enforcement.

Deliberate simplifications, documented rather than hidden

  • Tool identity remains name-keyed at the snapshot model level, but the diff engine now adds best-effort rename inference using schema fingerprint first and description similarity second.
  • No real cryptographic signing — hash-chain/fingerprint-based tamper-evidence only, explicitly labeled as such in every evidence artifact.
  • **test command is minimal** today (schema-validity + the same initialize/tools_list handshake scan does) — the fuller reliability-test catalogue from requirements §8 (timeout behavior, invalid-input behavior, fixture/contract tests) needs its own design pass; what exists never executes anything destructive by default, which was the one hard requirement for this pass.
  • **No baseline/policy/monitor/approve subcommands** — requirements §6 itself lists these as "potential later commands," not MVP.

Verification

  • PYTHONPATH=verify/src python3 -m pytest verify/tests -q --no-cov: current full Verify suite must pass before release.
  • python3 scripts/export_openapi.py --check: fresh (no FastAPI routes touched — this pass added no HTTP surface at all).
  • Manual end-to-end CLI run against the demo fixtures (scan/diff/evidence, default and custom policy, identical-snapshot pass case) — see the commit for full transcript; all outputs matched the requirements doc's own conceptual examples.