Low-Cost Hybrid Architecture

This document proposes a cheaper operating model for Sentinel Signal and MCP Verify after the Fly.io apps were spun down. The goal is to move heavy compute, mutable state, validation, model refreshes, and batch processing onto a powerful Linux server, then publish a curated read-only snapshot to a minimal Fly.io web tier once per day.

Executive Summary

The proposed plan is reasonable if the cloud product is treated as a read-only published surface between daily syncs. It is not a drop-in replacement for the current production stack if the cloud must continue accepting arbitrary scoring requests, issuing API keys, processing Stripe webhooks, recording usage, storing feedback, or running MCP Verify workers continuously.

Recommended target:

• On-prem Linux server is the source of truth and runs all heavy work.
• Local Postgres remains the operational write database for the current app code.
• A daily publisher builds a curated SQLite snapshot plus static assets and uploads them to Fly/Tigris object storage.
• A single small Fly web app downloads the latest signed snapshot at startup, opens it read-only, and serves the public UI/API from local container disk.
• Fly Postgres, Fly Loki, Fly Grafana, Fly Prometheus, exporters, workers, watchdogs, staging apps, and token service stay removed or stopped unless explicitly needed.

Expected cloud cost target:

• Minimal always-on Fly web app: roughly the cost of one small shared CPU VM.
• Tigris object storage: likely near free to a few dollars for this workload, depending on snapshot size and request volume.
• No Fly Postgres volume, no monitoring volumes, and no always-on workers.

This should move the project from about $150/month to roughly low single digits to low teens per month, depending on whether the web app is always-on or allowed to autostop.

Current Repo Reality

The current repository has multiple cloud-facing services:

• Claim scoring API: app/main.py, deployed by fly.toml.
• Token service, billing, API keys, Stripe webhooks: token_service/main.py, deployed by fly.token-service.toml.
• Hosted MCP remote bridge: mcp_remote/app.py, deployed by fly.mcp-remote.toml.
• MCP Verify web and worker: verify/src/mcp_verify, deployed by verify/fly.toml.
• Monitoring stack: Fly Loki, Grafana, Prometheus, Postgres exporter, and log shipper configs.
• Fly Postgres apps for production and staging.

The claim scoring app currently expects a live SQLAlchemy database on startup:

• app/db/session.py defaults to Postgres and waits for database readiness.
• app/db/models.py uses Postgres-specific schema names (rcm), JSONB, Postgres UUIDs, regex check constraints, and enum types.
• app/persistence.py, app/auth.py, and app/usage_metering.py perform request-time database reads/writes for API key auth, usage limits, feedback, request tracking, and history.
• token_service/main.py is write-heavy by design: API key issuance, self-serve workspace sessions, checkout sessions, Stripe webhooks, billing status, and usage reporting.

MCP Verify is closer to SQLite-compatible:

• verify/src/mcp_verify/db/session.py already supports SQLite URLs.
• Verify still has long-running scheduler/worker behavior that should move on-prem for the cheap architecture.

Conclusion: SQLite publication is a good target, but the first production version should publish a curated read-only snapshot rather than trying to run every current endpoint against SQLite in the Fly app.

What Happens On-Prem

The Linux server becomes the active system.

Services

Run these locally with Docker Compose or systemd:

• Local Postgres:
• source of truth for claim API operational data
• source of truth for billing/account data if those features remain in use
• source of truth for MCP Verify catalog, validations, jobs, and analytics
• Claim scoring API:
• used for local/internal scoring
• can run heavier model artifacts and larger memory profiles
• Token service:
• optional, only if continuing self-serve keys/billing
• handles Stripe webhooks only if an ingress tunnel is configured
• MCP Verify scheduler and workers:
• registry ingest
• validation jobs
• materialization
• TrustOps/Agent Sprawl analytics
• Model training and simulation jobs:
• denial, prior-auth, reimbursement model refreshes
• validation reports
• support profile generation
• Publisher:
• exports a sanitized read-only SQLite database
• writes a manifest
• signs or hashes artifacts
• uploads artifacts to Tigris
• restarts or signals the Fly web app

Local Data Flow

1. Ingest source data and registry data into local Postgres.
2. Run model training, scoring simulations, MCP Verify registry sync, and validation workers locally.
3. Materialize public serving tables.
4. Build a SQLite publication database from those materialized tables.
5. Run PRAGMA integrity_check, row-count checks, freshness checks, and application smoke tests against the SQLite artifact.
6. Compress the SQLite file.
7. Upload manifest.json, snapshot.sqlite.zst or snapshot.sqlite.gz, static assets, and optional model metadata to Tigris.
8. Restart or signal the Fly web app to load the new snapshot.

Snapshot Contents

The snapshot should include only data the cloud app needs to serve public read surfaces. Do not publish secrets, raw customer claim payloads, API key hashes, Stripe payloads, session tokens, webhook events, or raw logs.

Recommended SQLite tables:

• publication_metadata
• snapshot_id
• generated_at
• source_git_sha
• schema_version
• data_freshness_at
• artifact_hash
• MCP Verify public catalog:
• servers
• current validation summaries
• score components
• capabilities/search facets
• generated insight summaries
• current badges/status
• Public Sentinel Signal metadata:
• workflow catalog
• model versions
• support profiles
• pricing/plan display metadata if needed
• docs/search index metadata
• Optional public demo fixtures:
• sample claims
• precomputed example scores
• benchmark summaries

Tables to exclude:

• billing accounts
• API keys and API key hashes
• control-plane sessions and identities
• Stripe checkout sessions and webhook events
• usage metering tables
• feedback events unless explicitly anonymized and aggregated
• raw claims with patient/provider identifiers
• raw validation payloads that may include private credentials or sensitive headers

What Happens In The Cloud

The cloud side should be boring and small.

Fly Apps To Keep

Keep one production web app:

• sentinel-signal-web-prod or reuse sentinel-signal-api-prod after refactoring its config
• one process only
• one small machine only
• no worker process
• no watchdog process
• no Fly Postgres
• no Loki/Grafana/Prometheus apps
• no staging apps unless actively testing

Optional second app:

• sentinel-signal-mcp-remote can remain as an autostopped bridge if hosted MCP is still important.
• It should call only cheap/read-only endpoints or require caller-provided API keys.

Fly Web Runtime

At container startup:

1. Read SNAPSHOT_MANIFEST_URL or Tigris bucket/object configuration.
2. Download the latest manifest.json.
3. Verify schema version and artifact hash.
4. Download the compressed SQLite snapshot if the local cached copy is missing or stale.
5. Decompress to /tmp/sentinel-snapshot/<snapshot_id>.sqlite.
6. Open SQLite in read-only immutable mode.
7. Serve public UI/API routes.

SQLite connection string:

sqlite+pysqlite:///file:/tmp/sentinel-snapshot/current.sqlite?mode=ro&immutable=1&uri=true

Runtime behavior:

• Cloud app is read-only.
• No request writes to SQLite.
• No request depends on Postgres.
• Snapshot refresh is atomic: download to a new path, verify, then swap a current.sqlite symlink or restart the app.
• If the latest snapshot is bad, keep serving the previous snapshot.

Fly Configuration

Use a new small Fly config instead of the current production config:

app = "sentinel-signal-web-prod"
primary_region = "iad"

[build]
  dockerfile = "Dockerfile.web-snapshot"

[env]
  APP_ENVIRONMENT = "production"
  LOG_FORMAT = "json"
  SNAPSHOT_BUCKET = "sentinel-signal-public"
  SNAPSHOT_MANIFEST_KEY = "prod/latest/manifest.json"
  PUBLIC_READ_ONLY_MODE = "true"

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]

  [http_service.concurrency]
    type = "requests"
    soft_limit = 50
    hard_limit = 100

[[vm]]
  cpu_kind = "shared"
  cpus = 1
  memory_mb = 512

For a public site that should avoid cold starts, set min_machines_running = 1. For maximum savings, keep it at 0 and accept cold starts.

Proposed Architecture

flowchart LR
    subgraph OnPrem[Linux Server]
        PG[(Postgres source of truth)]
        API[Claim API]
        Token[Token service]
        VerifyWeb[MCP Verify local web]
        Workers[Verify workers and schedulers]
        Models[Training and simulation jobs]
        Publisher[Daily snapshot publisher]
    end

    subgraph Storage[Fly/Tigris Object Storage]
        Manifest[manifest.json]
        Snapshot[snapshot.sqlite.zst]
        Assets[static assets/search indexes]
    end

    subgraph Fly[Fly.io Cheap Cloud Tier]
        Web[Single read-only web app]
        Cache[(SQLite cached in /tmp)]
    end

    PG --> API
    PG --> Token
    PG --> VerifyWeb
    PG --> Workers
    Models --> PG
    Workers --> PG
    PG --> Publisher
    Publisher --> Manifest
    Publisher --> Snapshot
    Publisher --> Assets
    Manifest --> Web
    Snapshot --> Web
    Assets --> Web
    Web --> Cache

Is The User's Plan Reasonable?

Yes, with constraints.

Reasonable parts:

• Moving compute-heavy work to the Linux server is the right cost move.
• Publishing one curated SQLite database per day is a good fit for read-heavy catalog/search/UI pages.
• Object storage is a good fit for snapshot artifacts.
• Restarting or bouncing the web app after publishing is simple and acceptable for once-daily updates.
• Caching SQLite into the container filesystem is appropriate because Fly Machines have ephemeral root filesystems and the snapshot can be rehydrated from object storage.

Important caveats:

• The current claim API is not read-only. Scoring requests write claim records, prediction records, usage counters, and request IDs for feedback.
• The current token service is not read-only. It requires durable mutation for API keys, sessions, Stripe lifecycle events, and usage.
• The current SQLAlchemy claim models are not SQLite-portable without work because they use Postgres-specific schema and type features.
• If public users need live scoring of arbitrary claims, that processing either remains in cloud, proxies securely to the Linux server, or is disabled in the cheap public tier.
• If public users need self-serve billing and API key issuance, either keep a tiny durable write store in cloud or move that product surface to an on-prem service reachable by a secure tunnel.

Recommended product decision:

• Phase 1 cheap cloud should serve public read-only surfaces and precomputed demos only.
• Keep live scoring and billing private/on-prem until there is enough revenue or demand to justify always-on cloud infrastructure.

Endpoint Classification

Good Fit For Cloud Read-Only Snapshot

• Public marketing and docs pages under /portal.
• OpenAPI/spec display if generated into static JSON.
• MCP Verify search/index pages.
• MCP Verify server detail pages.
• Model validation/governance report summaries.
• Workflow catalog and schema endpoints.
• Status page based on latest published snapshot metadata.
• Demo pages backed by precomputed fixtures.

Keep On-Prem Or Disable In Cheap Cloud

• POST /v1/score
• POST /v1/score/batch
• POST /v1/workflows/{workflow}/validate if it only validates payload shape can stay in cloud; if it records telemetry, disable writes.
• POST /v1/feedback
• history endpoints unless backed by sanitized public data
• /v1/usage
• /v1/limits for real API keys
• all token-service billing/key/session endpoints
• Stripe webhooks
• MCP Verify admin endpoints
• MCP Verify validation job endpoints

Optional Proxy Mode

If live scoring must remain public, add an explicit proxy mode:

• Fly web app authenticates the caller.
• Fly forwards scoring requests to the Linux server over Cloudflare Tunnel, Tailscale Funnel, WireGuard, or another locked-down tunnel.
• Linux server performs scoring and writes to local Postgres.
• Fly returns the response.

This preserves on-prem processing but introduces uptime, security, latency, and home-network reliability concerns. It is less cheap-operationally than read-only publication, but still cheaper than running all heavy compute on Fly.

Security Model

Snapshot Safety

• Export from allowlisted tables only.
• Never export secrets or raw billing tables.
• Strip request headers, tokens, raw logs, and customer-identifying fields.
• Hash or aggregate identifiers if public analytics are needed.
• Write a snapshot_manifest with row counts and SHA-256 hashes.
• Sign the manifest with an offline/local signing key, or at minimum use a SHA-256 hash stored separately from the artifact.
• The Fly app must refuse to serve a snapshot with an unexpected schema version or hash mismatch.

On-Prem Ingress

Default recommendation: no inbound public ingress to the Linux server for Phase 1.

If proxy mode is enabled later:

• Use a private tunnel with mTLS or signed service tokens.
• Allow only Fly egress identities or tunnel-authenticated requests.
• Rate limit at the tunnel and app.
• Log minimally and avoid PHI.
• Keep Stripe webhooks on a dedicated endpoint with raw-body signature verification.

Cost Plan

Costs To Remove

Remove or keep deleted/stopped after export:

• Fly Postgres production and staging machines.
• Fly Postgres volumes once data is safely backed up and migrated on-prem.
• Grafana volume.
• Loki volume.
• Prometheus volume.
• Monitoring apps.
• Postgres exporter.
• Log shipper.
• Always-on MCP Verify worker.
• API watchdog process.
• Staging apps except during short test windows.

Current provisioned Fly volumes found during analysis:

• sentinel-signal-db-prod: 30 GB Postgres volume.
• sentinel-signal-db-staging: 5 GB Postgres volume.
• sentinel-signal-grafana-prod: 10 GB Grafana volume.
• sentinel-signal-grafana-staging: 1 GB Grafana volume.
• sentinel-signal-loki-prod: 10 GB Loki volume.
• sentinel-signal-prometheus-prod: 10 GB Prometheus volume.

Fly bills volumes by provisioned capacity even when attached machines are stopped, so those volumes continue to matter until removed.

Costs To Keep

Keep:

• One small web app.
• One Tigris bucket.
• DNS/certificates.
• Optional autostopped MCP remote app.

Tigris standard storage and request pricing should be negligible for a daily SQLite artifact in the hundreds of MB or low GB range.

Daily Sync Protocol

Artifact Layout

s3://sentinel-signal-public/
  prod/
    latest/
      manifest.json
      snapshot.sqlite.zst
      public-assets.tar.zst
    snapshots/
      2026-07-22/
        manifest.json
        snapshot.sqlite.zst
        public-assets.tar.zst

Manifest

{
  "snapshot_id": "2026-07-22T05-00-00Z",
  "schema_version": "public-snapshot-v1",
  "source_git_sha": "replace-with-git-sha",
  "generated_at": "2026-07-22T05:00:00Z",
  "data_freshness_at": "2026-07-22T04:55:00Z",
  "sqlite_object_key": "prod/snapshots/2026-07-22/snapshot.sqlite.zst",
  "sqlite_sha256": "replace-with-sha256",
  "sqlite_size_bytes": 123456789,
  "required_app_min_version": "1.1.0",
  "row_counts": {
    "servers": 10000,
    "server_current_summaries": 10000,
    "workflow_catalog": 3
  }
}

Publisher Steps

1. Acquire a local publication lock.
2. Run MCP Verify registry sync and due validation jobs.
3. Run model/support/profile refreshes if scheduled.
4. Run materialization SQL into local Postgres public tables.
5. Export curated tables into a new SQLite file.
6. Add indexes optimized for web reads.
7. Run PRAGMA integrity_check.
8. Run snapshot contract tests.
9. Compress and hash the snapshot.
10. Upload versioned objects to Tigris.
11. Upload latest/manifest.json last.
12. Restart Fly web app or call an authenticated refresh endpoint.
13. Run cloud smoke checks.
14. Keep the previous N snapshots for rollback.

Development Plan

Phase 0: Product Boundary Decision

Decide which surfaces remain public in cheap mode:

• Public site and docs: yes.
• MCP Verify catalog/search: yes.
• Precomputed demos: yes.
• Live claim scoring: no for initial cheap mode.
• Self-serve billing/API keys: no for initial cheap mode.
• Hosted MCP remote: optional autostopped bridge.

Deliverable:

• CHEAP_CLOUD_MODE=true feature flag.
• A documented list of enabled/disabled routes.

Phase 1: Public Snapshot Schema

Add a new package:

app_publication/
  __init__.py
  schema.py
  export_sqlite.py
  manifest.py
  validate_snapshot.py

Build:

• SQLAlchemy or raw SQL export from local Postgres to SQLite.
• Public allowlist for tables/columns.
• SQLite indexes for search/detail routes.
• Snapshot manifest and hash generation.
• Tests that assert excluded tables are not present.

Suggested test commands:

PYTHONPATH=. pytest tests/unit/test_publication_snapshot.py
PYTHONPATH=. python -m app_publication.export_sqlite --output /tmp/sentinel-public.sqlite
PYTHONPATH=. python -m app_publication.validate_snapshot /tmp/sentinel-public.sqlite

Phase 2: Read-Only Web Runtime

Add a new web app entrypoint:

app_public_web/
  __init__.py
  main.py
  snapshot_loader.py
  repository.py
  routes.py

Build:

• Tigris/S3 snapshot loader.
• Read-only SQLite repository.
• Public UI routes.
• Public JSON routes.
• Disabled-route responses for live mutation endpoints.
• Startup fallback to previous cached snapshot.

Do not import app.main directly for this runtime unless it is refactored to avoid database startup and write-heavy dependencies.

Phase 3: Fly Cheap Web Deployment

Add:

• Dockerfile.web-snapshot
• fly.web-snapshot.toml
• minimal runtime requirements if possible
• secrets for Tigris access

Recommended production command:

flyctl deploy --remote-only --config fly.web-snapshot.toml --app sentinel-signal-web-prod
flyctl scale count 1 -a sentinel-signal-web-prod --yes

For maximum savings after smoke testing:

• keep one machine defined with flyctl scale count 1
• keep auto_stop_machines = "stop", auto_start_machines = true, and min_machines_running = 0
• let Fly stop the idle machine after traffic drains

Do not scale the web app to 0 with flyctl scale count 0 if you expect public traffic to wake it. Fly autostart works on existing machines; it does not create machines from zero.

Phase 4: Daily Publisher Automation

Run on the Linux server via systemd timer or cron:

02:00 run local ingest/validation/model jobs
04:30 build public snapshot
04:45 validate snapshot
04:50 upload versioned snapshot
04:55 promote latest manifest
04:56 restart Fly web app
05:00 run smoke checks

Add scripts:

scripts/publish_public_snapshot.sh
scripts/restart_fly_web_snapshot.sh
scripts/rollback_public_snapshot.sh

Phase 5: Remove Legacy Cloud Cost

After two successful daily syncs and a tested rollback:

1. Export final Fly Postgres backups.
2. Verify local Postgres restore.
3. Delete unused Fly volumes.
4. Delete unused monitoring apps or leave apps without volumes/machines.
5. Remove or archive old deploy workflows that would recreate expensive apps.
6. Update docs/fly-io-deployment.md to point to the cheap architecture.

Phase 6: Optional Live Proxy

Only if needed:

• Add an on-prem scoring gateway.
• Expose it through a private tunnel.
• Add Fly read-only web proxy routes for selected live APIs.
• Keep billing/auth simple at first: static admin keys or allowlisted customers before restoring full self-serve billing.

Implementation Notes For This Repo

Refactors Needed

• Extract pure route-independent catalog rendering from app/main.py.
• Add a read-only repository layer for public pages.
• Avoid importing token_service.main in the cheap public runtime.
• Avoid wait_for_database() in the cheap public runtime.
• Add route gating for live scoring/billing endpoints.
• Keep current Postgres models for on-prem writes.
• Do not convert all current SQLAlchemy models to SQLite in Phase 1.

Candidate Route Strategy

Use a new FastAPI app for cheap cloud:

• / and /portal/*: serve static/generated pages.
• /verify/* or verify.sentinelsignal.io: serve SQLite-backed MCP Verify pages.
• /v1/workflows: serve static workflow metadata.
• /v1/workflows/{workflow}/schema: serve static schema.
• /health: returns app and snapshot health.
• /readyz: returns whether a verified snapshot is loaded.

Return 503 or 410 with clear JSON for disabled mutation endpoints.

Observability

For cheap mode:

• Use Fly logs only.
• Keep /health and /readyz.
• Add a /snapshot endpoint with snapshot ID, generated time, and row counts.
• Do not run Grafana/Loki/Prometheus in Fly.
• Run local monitoring on the Linux server if desired.

Rollback Plan

Keep at least seven daily snapshots in Tigris.

Rollback steps:

1. Point prod/latest/manifest.json back to a previous versioned snapshot.
2. Restart the Fly web app.
3. Verify /readyz, /snapshot, and key UI pages.

If the cheap web runtime fails entirely:

1. Keep old Fly apps suspended but not destroyed until the new runtime has run for a week.
2. Temporarily scale the previous API or Verify app back up.
3. Restore local/Postgres connectivity only if needed.

Open Questions

• Should public live scoring be disabled, proxied to on-prem, or kept as a paid cloud-only feature later?
• Should token.sentinelsignal.io stay offline, become static/help-only, or proxy to on-prem?
• Which domain owns MCP Verify after consolidation: verify.sentinelsignal.io, sentinelsignal.io/verify, or both?
• What is the acceptable public data freshness SLA: daily, twice daily, or manual publish?
• How large is the curated SQLite file after removing private/raw tables?
• Does the Linux server have a reliable backup plan for local Postgres and model artifacts?

References

• Fly.io resource pricing: https://fly.io/docs/about/pricing/
• Fly.io autostop/autostart behavior: https://fly.io/docs/reference/fly-proxy-autostop-autostart/
• Fly.io app configuration: https://fly.io/docs/reference/configuration/
• Fly.io Tigris object storage: https://fly.io/docs/tigris/
• Tigris pricing: https://www.tigrisdata.com/pricing/