Skip to main content

​
Production Hardening Plan

Status: draft · Owner: eng · Last updated: 2026-07-07 This plan closes the gap between “the code is sound and known bugs are fixed” (true today) and “we can run real customers’ money and data at scale.” It is organized by the five gaps identified in the July 2026 audit. Each section ends with a decision, concrete steps, and an effort estimate. Read the roadmap at the bottom first — it sequences everything into launch-blockers vs. post-launch and tells you what to do in what order. Scope of stack (drives every decision below): Convex (backend functions + cron + HTTP), Clerk (auth), Stripe (billing), GatewayAPI (SMS), Cloudflare (dashboard/web/CDN worker), Expo (mobile), PostHog (product analytics).

​
1. Observability & alerting — you are currently flying blind

Problem. No error tracking on any surface. Dropped Stripe events, failed SMS sends, and credit-ledger drift do console.error(...); return — nobody is paged. Incidents surface via customer complaints, not monitoring. PostHog is product analytics, not ops observability. Decision: Sentry (free tier) for errors across all surfaces + a free uptime monitor on health endpoints. Sentry’s free tier (5k errors/mo) is the right “free to start” pick — best-in-class React/Vite, Expo, and Cloudflare Workers SDKs, source maps, release tracking, and free alert rules to Slack/email. Fully-free-forever self-host alternative is GlitchTip (Sentry-API-compatible), but it’s ops overhead we don’t want on day one; start on Sentry’s free tier and only migrate if volume forces it. Approach (three surfaces, one nuance):
  • Frontends + mobile (free, immediate): add the Sentry SDK to apps/dashboard, apps/web, and apps/mobile. Wire source-map upload into the Cloudflare/EAS builds. This alone gives crash and error visibility for the entire user-facing surface today.
  • Convex backend (two phases): Convex’s native exception-reporting/log-stream integrations (Sentry/Axiom/Datadog) are a Convex Pro feature, not on the free Starter plan.
    • Phase 1 (free): a thin reportError() helper that POSTs exceptions to Sentry via fetch from the Convex runtime, wrapped around mutations/actions (a withObservability wrapper) and called explicitly at the existing console.error sites in the billing/webhook/SMS paths.
    • Phase 2 (when on Convex Pro — you’ll want it for production anyway): enable the native Sentry log-stream integration for full function-level capture + log retention, and retire the manual wrapper.
  • Uptime (free): BetterStack (Better Uptime) or UptimeRobot free tier pinging the Convex HTTP health route, the dashboard, the web root, and the CDN worker. Pairs with the deploy smoke tests in §2.
Alert rules that must exist from day one (Sentry → Slack/email), tagged by area:
  • Any error in the Stripe webhook handler (http.ts billing routes) or organizationSettings subscription handlers — including the “event dropped” console.error branches.
  • SMS send failures / GatewayAPI errors.
  • Credit-ledger anomalies (debit failures after retries, negative-balance guard trips).
  • Spike detection: error-rate jump after a deploy (ties to rollback, §2).
Effort: frontends ~1 day; Convex Phase-1 wrapper ~1–2 days; uptime ~2 hours. Launch-blocker.

​
2. Deploy safety — modeled on adamtrip-solutions/fogos-portugal

Problem. deploy.yml pushes straight to prod on every merge to main, including live Convex schema pushes, with no staging gate, no post-deploy smoke test, and no documented rollback. One bad merge is a live, customer-facing incident with no early warning. What fogos-portugal does right (and we should adopt the principles of):
  • Release-gated CD, not push-to-main. Deploy triggers only on a published GitHub release (on: release: published) or manual workflow_dispatch — never on PR/merge. Deploys are deliberate and versioned (tags api-v* / web-v*).
  • environment: production on the deploy job — enables GitHub environment protection (required reviewer / wait timer) as a human gate.
  • Health-gated rollout + post-deploy smoke tests. docker compose up -d --wait plus explicit curl smoke steps (/healthz/ready, a real GraphQL query, the RSS feed, the web root) with retry/backoff — the deploy fails if the app isn’t actually serving.
  • Version pinning + rollback. A server-side versions.env pins the deployed version; rollback = re-pin a prior version and re-run.
Translation to CalGest’s serverless stack (we can’t copy the Docker/self-hosted model, but every principle maps):
  1. Switch to release-gated deploys. You already have release-please config + version-bump actions. Invert the current order: cut the release first, deploy on the release event, so every prod deploy corresponds to a versioned, reviewable release. Stop deploying on raw push to main.
  2. Add a staging Convex deployment + gate. Convex supports multiple deployments — deploy to a staging deployment first, run smoke tests against it, then promote to prod. Put prod behind environment: production with a required approval.
  3. Add post-deploy smoke tests (the biggest current gap), directly modeled on fogos’s smoke step: after each deploy, curl with retry/backoff against
    • the Convex HTTP health route (add a /healthz HTTP action that checks DB reachability),
    • the dashboard URL (expects 200 + an app shell marker),
    • the web root,
    • the CDN worker (a known asset path returns the right status). Fail the deploy job — and alert (§1) — if any smoke check fails.
  4. Document + script rollback. Cloudflare Workers/Pages keep deployment history: wrangler rollback (or the dashboard) is a one-click revert for dashboard/web/CDN. Convex: redeploy the prior release tag / use the Convex dashboard’s deployment history. Write this as a deploy/README runbook (fogos keeps a deploy/README.md — mirror that) so rollback is a known 2-minute path, not an improvised one under pressure.
  5. Keep concurrency deploy-production with cancel-in-progress: false (fogos does this) so deploys never race.
Migration-safety note (Convex-specific): schema changes push live. Adopt the widen → migrate → narrow discipline (the repo has migrations.ts and a convex-migration-helper skill) for any breaking field change, and test migrations against a production-like dataset in staging before promoting. This is part of “deploy safety,” not a separate workstream. Effort: ~3–5 days (health endpoint + smoke steps + staging deploy + release-gating rewire + runbook). Launch-blocker (staging gate + smoke tests + rollback runbook are the must-haves; release-please rewire can follow).

​
3. Expand security & test coverage — audit in progress

Problem. The July audit was ~95% and sampling-based; it found a class of bug (client-supplied refs written without org validation) that likely recurs elsewhere. The frontends were only diff-reviewed, mobile not reviewed at all, and there are no end-to-end tests. Completed (July 2026, both clean):
  • Backend mutation-ref sweep — exhaustive hunt for every remaining instance of the recurrence-class bug. Result: 115 public mutations/actions scanned, ~90 take reference ids, all validated, 0 flagged. The recurrence.ts fix was the last instance of its class; the ownership- check pattern (assertEntityRefsBelongToOrg / ensure*BelongToOrg / orgId-scoped index lookups) is applied uniformly, including in the batch/reorder/bulk paths where misses usually hide.
  • Frontend security review (dashboard + web) — no medium-or-higher findings. Convex is the authoritative authz layer; client guards are correctly cosmetic-only; no server secrets in the bundle (all VITE_-prefixed public keys); open-redirect defended; AI/markdown rendered through a hardened renderer; PostHog carries only operator data (no customer PII), consent-gated. One flagged item verified safe (getAdminNavBadges gates on requirePlatformStaff).
Two narrow follow-up sweeps the mutation/action sweep explicitly did not cover (do for completeness):
  • Public query read-IDOR sweep — notificationCredits and analytics expose public queries (outside the mutation/action scope); spot-checked as org-scoped but not exhaustively swept.
  • List-query field projection — confirm customer/employee list queries project minimal fields rather than full PII rows (frontend review’s one open low item).
  • (http.ts asset-download endpoints are signed-token based and the CDN worker’s token→key binding was already verified safe in the July audit — no further action.)
Still to schedule:
  • Mobile review + typecheck debt. Mobile has 23 pre-existing type errors and thin tests; it’s the least mature surface. Fix the type errors (so it can join the CI check-types job — see the CI work already merged), then a security pass on token handling (SecureStore), deep-link/OAuth handling, and the same ref-validation class on any mobile-initiated mutations.
  • End-to-end tests for the critical journeys: signup → subscribe → book → reminder fires → credit debits → cancel/refund. None exist today; these are what catch cross-subsystem regressions the unit tests miss.
  • Load test the 09:00 reminder fan-out (audit finding F4). The credit-ledger sharding serializes debits (safe, but retry-storm-prone under a concurrent fan-out). Prove it holds at realistic org volume; if it retry-storms, migrate to the Convex sharded-counter component.
Cadence: make the ref-validation sweep a recurring check (a lightweight lint/test that fails if a public mutation writes a v.id() arg without going through the ownership helper), so this class can’t regress. Effort: sweep remediation ~1–2 days once results land; frontend fixes TBD by findings; mobile ~1 week; E2E harness ~1 week; load test ~2 days. Ref-sweep remediation is launch-blocker; E2E and mobile are fast-follow.

​
4. GDPR / compliance — EU/PT market: likely a hard launch blocker

Problem. You are a data processor for the salons’ end-customer PII (names, phones, emails, plus aftercare notes and photos that may be special-category/health-adjacent for personal-services). Clerk’s deletion covers the account owner, not their customers. For an EU self-serve launch this is a legal blocker, not a nice-to-have. Phased implementation plan: Phase A — Data map & tooling (engineering):
  1. Data inventory (ROPA). Enumerate every store of personal data: Convex tables (customers, appointments, aftercare, photos, customFieldValues, tracking), Clerk (owners), Stripe (billing), GatewayAPI (phone numbers), PostHog (behavioral), Sentry (may capture PII in errors — scrub it). Document controller vs. processor role for each.
  2. Right to erasure. Extend/verify the existing customers.remove cascade to be a complete hard-delete: appointments unlink, custom field values, tracking, aftercare, and R2 photo objects (confirm storage objects are actually deleted, not just DB rows). Add an org-level account deletion (delete-my-business) that purges all tenant data and cancels Stripe.
  3. Right to access / portability. A function that exports all data for a given end-customer (and an org-level export) to a JSON/CSV bundle.
  4. Retention & minimization. Define retention windows and implement purge crons (old soft-deleted records, aged SMS logs, stale appointments). Scrub PII from Sentry payloads and limit what’s sent to PostHog (ties to §3 frontend review).
Phase B — Legal & policy (parallel, not eng-blocking): 5. Sub-processor DPAs. Sign DPAs with Convex, Clerk, Stripe, GatewayAPI, Cloudflare, PostHog, Sentry. Publish a sub-processor list + privacy policy. 6. Customer-facing DPA. You’re a processor for salons (controllers) — add a DPA to your ToS. 7. ROPA + breach-notification process (72-hour) documented. 8. Special-category assessment. Get a legal read on whether aftercare/photos for beauty count as special-category data; if so, add explicit-consent capture + extra safeguards. Consent (cookie consent already exists): ensure analytics/marketing consent is captured and respected end-to-end. Effort: Phase A tooling ~1.5–2 weeks; Phase B is legal-led, run in parallel. Launch-blocker for EU self-serve (erasure + export + DPAs + privacy policy are the minimum bar).

​
5. Abuse & cost safeguards on expensive operations

Problem. Booking spam is throttled and AI is now metered, but authenticated mutations are largely unthrottled, and SMS is real money — a compromised or malicious account can burn credits down to the safety floor, trigger bulk operations, or spam invitations. Good news: the infrastructure already exists — @convex-dev/rate-limiter is registered (convex.config.ts) and used for booking. Extend it into a coherent policy. Design — a limit taxonomy + a shared helper:
  • Public (unauthenticated): booking (done), any other public endpoint — strict per-IP/per-contact limits.
  • Authenticated read: generous per-user limits (cheap).
  • Authenticated write: moderate per-user + per-org limits.
  • Billable / expensive: SMS-triggering ops, bulk/batch mutations, exports, invitations, AI — tight per-user and per-org limits, plus a daily spend circuit-breaker per org (cap SMS/credit burn per day beyond the existing per-send safety floor; AI token cap already enforced).
  • Idempotency on any retry-able mutation with money/side effects (extends the refund/pack-purchase idempotency already in place).
Anomaly safeguard: alert (§1) when an org’s SMS/credit burn or AI spend spikes above its trailing baseline — catches a compromised account before the bill does. Implementation: a withRateLimit(kind) wrapper applied per endpoint by category, so limits are declarative and auditable rather than scattered. Start by covering the billable tier (highest risk), then write, then reads. Effort: ~3–4 days for the billable tier + circuit-breaker + anomaly alert; the rest incremental. Fast-follow (billable-tier caps are near-launch-blocker given real money is involved).

​
Sequenced roadmap

Launch-blockers (do before real customers + real money):
  1. Observability + alerting (§1) — Sentry on all surfaces + billing/webhook/SMS alert rules + uptime monitor. Highest priority: you cannot run billing blind.
  2. Stripe reconciliation + global idempotency (from the billing audit’s F1, folded here) — a processedStripeEvents dedup table + a nightly job reconciling DB subscription state against Stripe, alerting on drift. Second-highest: this is where silent revenue loss lives.
  3. Deploy safety (§2) — staging gate + post-deploy smoke tests + rollback runbook.
  4. GDPR Phase A tooling + Phase B minimum (§4) — erasure, export, DPAs, privacy policy (EU).
  5. Billable-tier rate limits + circuit-breaker (§5).
(The ref-validation sweep is done — 0 findings — so it is no longer a blocker; add the recurring lint/test guard from §3 so the class can’t regress.) Fast-follow (first weeks post-launch):
  • E2E tests for critical journeys (§3).
  • Mobile type-error cleanup → CI type-check + mobile security pass (§3).
  • Load-test the 09:00 fan-out; migrate to sharded-counter if needed (§3).
  • Release-please-gated deploys (§2) and full rate-limit taxonomy (§5).
Rough total to launch-ready: ~3–4 focused weeks, observability and Stripe reconciliation first. None of it is glamorous; all of it is what lets you sleep once real money flows.

​
Confidence note

Security and billing correctness assessments are high-confidence (code read directly). The ops, scale, mobile, and compliance assessments are based on checking what’s present, not stress-testing depth — treat effort estimates as planning-grade, to be refined per workstream.