Sentinel Signal

IONOS Production Phases 1-4 Implementation Notes

Source: docs/ionos-production-phases-1-4.md

All documents Docs Home Home

Document Content

IONOS Production Phases 1-4 Implementation Notes

Date: 2026-07-31

Completed

  • Added deploy/ionos/docker-compose.yml for the full application rather than the slim public snapshot.
  • Added deploy/ionos/Caddyfile for public TLS termination and routing.
  • Added deploy/ionos/prod.env.example with production settings and secret placeholders only.
  • Added deploy/ionos/prometheus.yml to scrape API, token service, MCP remote, Verify, and PostgreSQL exporter metrics inside the Compose network.
  • Added deploy/ionos/postgres-init/create-verify-db.sh so the shared PostgreSQL instance creates the Verify database.
  • Added deploy/ionos/backup-postgres.sh for nightly encrypted database backups to IONOS Object Storage.
  • Added systemd units for stack startup and the nightly backup timer.
  • Added deploy/ionos/bootstrap-host.sh for host hardening, Docker installation, firewall setup, and mounting the dedicated PostgreSQL SSD.
  • Updated .dockerignore so production API builds include only the required model artifacts and contract snapshots.
  • Provisioned the separate IONOS production VDC, Memory Cube L, public LAN, public NIC, and 150 GB SSD Standard volume.
  • Executed host bootstrap on root@66.179.248.190.
  • Formatted the attached 150 GB PostgreSQL SSD, mounted it at /srv/sentinel-signal/postgres, and created /srv/sentinel-signal/postgres/data for Postgres.
  • Installed Docker, Docker Compose, AWS CLI, age, zstd, and git.
  • Enabled UFW with default deny incoming and explicit 22/tcp, 80/tcp, and 443/tcp rules.
  • Created IONOS Object Storage bucket sentinel-signal-prod-db-backups in us-central-1.
  • Installed generated production secrets and Object Storage credentials in /etc/sentinel-signal/prod.env on the IONOS host.
  • Started the full Docker Compose stack through sentinel-signal.service.
  • Enabled the nightly encrypted backup timer through sentinel-signal-db-backup.timer.
  • Ran a manual encrypted PostgreSQL backup and verified the uploaded object in IONOS Object Storage.
  • Ran a non-destructive restore validation from the encrypted Object Storage backup into a throwaway PostgreSQL 16 container.
  • Completed DNS cutover for all configured production hostnames.
  • Confirmed Caddy issued public Let's Encrypt certificates for all configured production hostnames.
  • Recovered readable provider values from currently running Fly machines and applied them to the IONOS production env file.

Provisioned IONOS Resources

| Resource | Name | ID | | --- | --- | --- | | VDC | sentinel-signal-prod | 5a41b5f8-263d-4248-9c56-8a03d3b2526e | | LAN | public-lan | 1 | | Server | sentinel-signal-prod-app-01 | dc40cc09-e056-40ab-a120-88998662c0fb | | Boot volume | sentinel-signal-prod-app-01-boot | 77742443-a081-44e0-adcb-491bb14c9148 | | PostgreSQL data volume | sentinel-signal-prod-pgdata-150gb | 044912a4-8fe8-4b68-87e8-f46d92b712b6 | | Public NIC | public-nic | ca096387-da7e-4cb4-b7fb-67d440cfe005 | | Object Storage bucket | sentinel-signal-prod-db-backups | n/a |

Public IPv4: 66.179.248.190

Firewall rules:

| Rule | ID | Port | | --- | --- | --- | | allow-ssh | f2298dac-e16f-43f2-af01-ca01c421249b | 22/tcp | | allow-http | 2f4479c4-f8e1-4c1d-a73b-051e31ddeb19 | 80/tcp | | allow-https | ea11fd42-cc14-4859-860e-f4abb2edaa23 | 443/tcp |

Host Bootstrap Verification

| Check | Result | | --- | --- | | OS | Ubuntu 24.04.4 LTS | | PostgreSQL block device | /dev/disk/by-id/virtio-07560f9ad547df41 | | Mounted filesystem | /dev/vdb at /srv/sentinel-signal/postgres, ext4, rw,noatime | | PostgreSQL data directory | /srv/sentinel-signal/postgres/data | | Filesystem UUID | dc5dc91d-c5e0-44f5-a4a0-3238a24ae59c | | Docker | Docker version 29.7.0, service active | | Docker Compose | Docker Compose version v5.3.1 | | AWS CLI | aws-cli/1.45.61 | | age | 1.1.1 | | UFW | Active, default deny incoming, allow outgoing |

Remaining Work

Object Storage credentials, the bucket, /etc/sentinel-signal/prod.env, and the required model artifact directories have been configured on the IONOS host.

The remaining work is external Stripe/WorkOS/SMTP credential configuration if those should be live immediately.

Current production credential status:

| Area | Current state | | --- | --- | | Claim API/token Stripe | Secret key, webhook secret, and usage reporting flag recovered from running Fly. Growth/pro checkout price IDs remain missing. | | MCP Verify Stripe | Secret key, webhook secret, and MCP Verify price IDs recovered from running Fly. | | WorkOS | Bridge shared secret and AuthKit domain remain missing; the only observed WorkOS values were staged on a suspended/no-image Fly app and could not be read back. | | SMTP | Alertmanager and MCP Verify SMTP settings recovered from running Fly and applied on IONOS. |

Live Validation

Validated on 2026-07-31:

| Check | Result | | --- | --- | | sentinel-signal.service | Active | | sentinel-signal-db-backup.timer | Active | | PostgreSQL container | Running and healthy | | API internal health | HTTP 200 | | Token service internal health | HTTP 200 | | MCP remote internal health | HTTP 200 | | Verify internal health | HTTP 200 | | Object Storage bucket access from host | Success | | Manual encrypted backup upload | postgres/20260731T051456Z/sentinel-signal-postgres-20260731T051456Z.tar.zst.age | | Backup restore validation | Decrypt, checksum, and restore succeeded in throwaway PostgreSQL 16 container | | Restored claim_risk_score metadata | 0 public tables, matching production | | Restored mcp_verify metadata | 39 public tables, matching production | | Public DNS | All configured hostnames resolve to 66.179.248.190; no AAAA records | | Public TLS | Caddy issued Let's Encrypt certificates for all configured hostnames | | https://sentinelsignal.io | HTTP 200 | | https://portal.sentinelsignal.io/health | HTTP 200 | | https://api.sentinelsignal.io/health | HTTP 200 | | https://token.sentinelsignal.io/health | HTTP 200 | | https://mcp.sentinelsignal.io/healthz | HTTP 200 | | https://verify.sentinelsignal.io/healthz | HTTP 200 | | https://grafana.sentinelsignal.io | HTTP 401 without basic auth, expected |

Operational Guardrails

  • Keep /etc/sentinel-signal/prod.env out of git and mode 0600.
  • Do not expose application containers directly. Only Caddy should publish host ports.
  • Use the IONOS NIC firewall plus ufw; both should allow only SSH, HTTP, and HTTPS.
  • Keep the age private key off the IONOS host. The server only needs the public recipient.
  • Re-run restore validation after any material database schema or backup-script change.