SENTINEL SIGNAL

Sentinel Signal

Agent Reliability production runbook

Source: docs/agent-reliability-production-runbook.md

All documents Docs Home Home

Document Content

Agent Reliability production runbook

Updated: 2026-08-11

This runbook covers the remaining operator steps for full hosted Agent Reliability go-live on IONOS.

Current production mode

The hosted API is live when:

AGENT_RELIABILITY_ENABLED=true

The REST API is authenticated. An unauthenticated request to /v1/agent-reliability/repositories should return 401, not 404 or 500.

The GitHub webhook endpoint is registered at:

https://verify.sentinelsignal.io/v1/agent-reliability/github/webhook

The webhook secret is stored only in /etc/sentinel-signal/prod.env on the IONOS host. Do not commit it or paste it into logs.

Environment variable reference

Consolidated from the public API docs page, which no longer lists these directly (per the Agent Reliability productization pass -- public docs should read like a product, not this runbook).

  • AGENT_RELIABILITY_ENABLED=false -- keeps all hosted routes unregistered.
  • AGENT_RELIABILITY_DISABLE_NEW_SCANS=true -- kill switch; blocks new hosted scan enqueue while preserving read paths.
  • AGENT_RELIABILITY_REPOSITORY_LIMIT=3 / AGENT_RELIABILITY_TARGETS_PER_REPOSITORY=5 / AGENT_RELIABILITY_MONTHLY_SCANS=100 / AGENT_RELIABILITY_CONCURRENCY=2 -- open-beta (free tier) caps.
  • AGENT_RELIABILITY_TEAM_TIER_REPOSITORY_LIMIT=25 / AGENT_RELIABILITY_TEAM_TIER_TARGETS_PER_REPOSITORY=25 / AGENT_RELIABILITY_TEAM_TIER_MONTHLY_SCANS=1000 / AGENT_RELIABILITY_TEAM_TIER_CONCURRENCY=10 -- paid Team-tier caps, applied once a workspace has an active/trialing billing account.
  • AGENT_RELIABILITY_GITHUB_WEBHOOK_SECRET -- must be configured before the webhook endpoint accepts deliveries.
  • AGENT_RELIABILITY_GITHUB_APP_ID, AGENT_RELIABILITY_GITHUB_PRIVATE_KEY_PATH (production; mounted secret file) / AGENT_RELIABILITY_GITHUB_PRIVATE_KEY (local/dev only), AGENT_RELIABILITY_GITHUB_APP_SLUG (enables the public install link).
  • AGENT_RELIABILITY_EVIDENCE_SIGNING_KEY_PATH / AGENT_RELIABILITY_EVIDENCE_SIGNING_KEY_ID -- see "Evidence signing" below.
  • MCP_VERIFY_STRIPE_AGENT_RELIABILITY_TEAM_PRICE_ID -- the Stripe Price ID for the Team plan; reuses the existing MCP_VERIFY_STRIPE_SECRET_KEY/MCP_VERIFY_STRIPE_WEBHOOK_SECRET already configured for Verify's own claimed-server billing, not a separate Stripe account.
  • AGENT_RELIABILITY_ALERT_SLACK_WEBHOOK_URL / AGENT_RELIABILITY_ALERT_PAGERDUTY_ROUTING_KEY / AGENT_RELIABILITY_ALERT_EMAIL_TO -- internal-ops alert destination(s); see "Operational alerts to monitor" below. Reuses MCP_VERIFY_PAGERDUTY_EVENTS_URL/MCP_VERIFY_SMTP_* for the actual PagerDuty/SMTP transport.
  • AGENT_RELIABILITY_ALERT_CHECK_INTERVAL_MINUTES=15 / AGENT_RELIABILITY_ALERT_COOLDOWN_MINUTES=60 -- health-check window and re-alert suppression window.

GitHub App setup

Create a GitHub App owned by the production GitHub organization/account.

Recommended app name:

Sentinel Verify Agent Reliability

Homepage URL:

https://verify.sentinelsignal.io

Webhook URL:

https://verify.sentinelsignal.io/v1/agent-reliability/github/webhook

Webhook secret:

ssh root@66.179.248.190 "grep '^AGENT_RELIABILITY_GITHUB_WEBHOOK_SECRET=' /etc/sentinel-signal/prod.env | cut -d= -f2-"

Required repository permissions:

  • Metadata: read
  • Checks: read/write
  • Contents: read
  • Pull requests: read

Required webhook events:

  • push
  • pull_request
  • installation
  • installation_repositories

Generate a private key from the GitHub App settings page. Save the downloaded PEM as a root-readable mounted secret on IONOS, for example:

scp /path/to/downloaded-private-key.pem root@66.179.248.190:/etc/sentinel-signal/secrets/agent-reliability-github-app-private-key.pem
ssh root@66.179.248.190 "chmod 0400 /etc/sentinel-signal/secrets/agent-reliability-github-app-private-key.pem"

Update /etc/sentinel-signal/prod.env:

AGENT_RELIABILITY_GITHUB_APP_ID=<numeric app id>
AGENT_RELIABILITY_GITHUB_PRIVATE_KEY_PATH=/etc/sentinel-signal/secrets/agent-reliability-github-app-private-key.pem

Then deploy or restart:

cd /opt/sentinel-signal
./deploy/ionos/deploy.sh

Installation and workspace linking

After the GitHub App is installed on a repository, link the installation to the authenticated workspace through:

POST /v1/agent-reliability/github/installations/callback

The callback is authenticated and never accepts a caller-supplied team ID. The authenticated workspace determines tenancy.

The request body must include the GitHub installation ID, account identity, permissions, setup action, and selected repositories. The GitHub App setup UI or an internal admin flow should call this endpoint after installation.

Live smoke tests

No-secret route registration smoke:

python3 scripts/smoke_agent_reliability.py --base-url https://verify.sentinelsignal.io

Authenticated REST smoke:

AGENT_RELIABILITY_BEARER_TOKEN=<token-with-agent-reliability-read-or-admin-scope> \
  python3 scripts/smoke_agent_reliability.py \
  --base-url https://verify.sentinelsignal.io \
  --bearer-env AGENT_RELIABILITY_BEARER_TOKEN

GitHub App smoke:

  1. Install the GitHub App on a test repository.
  2. Link the installation to a test workspace through the installation callback.
  3. Create one public HTTPS target for the repository.
  4. Push to the repository default branch.
  5. Confirm a GitHub Check Run is created.
  6. Confirm a hosted run is queued and completed.
  7. Confirm evidence is persisted.
  8. Confirm the Check Run conclusion matches the run decision.
  9. Add .verify-agent.yml, push again, and confirm the worker fetches the policy at the evaluated commit.
  10. Open a pull request from a test branch and confirm the Check Run is attached to the PR head SHA.
  11. Visit /agent-reliability and, with an authenticated bearer token, /v1/agent-reliability/dashboard.
  12. Enable a target schedule with PUT /v1/agent-reliability/repositories/{id}/targets/{target_id}/schedule and confirm the scheduler enqueues a due scan.

Expected conclusion mapping:

  • pass → success
  • warn → neutral
  • approval_required → action_required
  • block → failure
  • config_error / infra_error → action_required

Status checks

The public status page includes an Agent Reliability panel:

https://verify.sentinelsignal.io/status

It reports only booleans and limit values:

  • hosted API enabled/disabled
  • scan kill switch state
  • webhook secret configured/missing
  • GitHub App ID configured/missing
  • GitHub private key configured/missing
  • GitHub Checks ready/no-op
  • open-beta limits

It does not expose secret values, App IDs, or file paths.

Evidence signing

Technical beta evidence can remain unsigned. For GA/compliance-grade evidence, mount an Ed25519 PEM private key and configure:

AGENT_RELIABILITY_EVIDENCE_SIGNING_KEY_PATH=/etc/sentinel-signal/secrets/agent-reliability-evidence-ed25519.pem
AGENT_RELIABILITY_EVIDENCE_SIGNING_KEY_ID=<stable-key-id>

The worker signs the canonical artifact fingerprint. Raw signing key material is never stored in application tables or evidence responses.

Operational alerts to monitor

Automatic as of the operator-alerting pass. A scheduled health check (check_agent_reliability_operational_health, agent_reliability_hosted/alerting.py) runs on every schedule_tick and pages a configured internal-ops destination (Slack/PagerDuty/email -- reuses notifications.py's existing delivery transport) whenever one of the 8 failure classes below breaches its threshold in the trailing AGENT_RELIABILITY_ALERT_CHECK_INTERVAL_MINUTES window (default 15 min). Repeat breaches are suppressed for AGENT_RELIABILITY_ALERT_COOLDOWN_MINUTES (default 60 min) so an ongoing incident pages once, not every tick.

Configure at least one destination for alerts to actually fire (a breach with no destination configured is still recorded and visible via AgentReliabilityOperationalEvent, just not paged):

AGENT_RELIABILITY_ALERT_SLACK_WEBHOOK_URL=<slack incoming webhook url>
AGENT_RELIABILITY_ALERT_PAGERDUTY_ROUTING_KEY=<pagerduty routing key>
AGENT_RELIABILITY_ALERT_EMAIL_TO=<ops distribution address>

Failure classes covered, and their alert threshold (count within one check window):

  • hosted run infra_error -- 1 (alerts on first occurrence)
  • webhook signature failures -- 1
  • hosted run config_error -- 3
  • GitHub API create/update Check Run failures -- 3
  • scheduled scan enqueue failures -- 3
  • worker retry loops (Job.attempts >= 3 on a still-pending/running Agent Reliability job) -- 1
  • quota rejections -- 5
  • duplicate delivery spikes -- 10

Not yet automated: evidence-signing readiness when paid/compliance claims are enabled -- this is a live configuration-state check (is a signing key mounted), not an event count, and doesn't fit the same threshold model. Still requires manual review; see "Evidence signing" above.

The manual grep command below remains useful for ad-hoc debugging (e.g. investigating a specific delivery ID) but is no longer the primary detection mechanism for the classes above:

ssh root@66.179.248.190 \
  "cd /opt/sentinel-signal && docker compose --env-file /etc/sentinel-signal/prod.env -f deploy/ionos/docker-compose.yml --project-directory /opt/sentinel-signal logs --since=15m verify-web verify-worker | grep -Ei 'agent_reliability|github|webhook|traceback|error'"

Remaining external inputs

The codebase cannot create these safely without operator interaction with GitHub:

  • numeric GitHub App ID
  • downloaded GitHub App private key PEM
  • confirmation of the GitHub owner/account that should own the App
  • a bearer token with Agent Reliability scopes for authenticated API smoke testing
  • Ed25519 evidence signing key and public verification-key publication plan before paid/compliance-grade claims
  • GitHub Marketplace approval and Stripe product configuration before paid GA