> ## Documentation Index
> Fetch the complete documentation index at: https://docs.calgest.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Production hardening plan

# Production Hardening Plan

Status: draft · Owner: eng · Last updated: 2026-07-07

This plan closes the gap between "the code is sound and known bugs are fixed" (true today) and
"we can run real customers' money and data at scale." It is organized by the five gaps identified
in the July 2026 audit. Each section ends with a decision, concrete steps, and an effort estimate.

**Read the roadmap at the bottom first** — it sequences everything into launch-blockers vs.
post-launch and tells you what to do in what order.

Scope of stack (drives every decision below): Convex (backend functions + cron + HTTP),
Clerk (auth), Stripe (billing), GatewayAPI (SMS), Cloudflare (dashboard/web/CDN worker),
Expo (mobile), PostHog (product analytics).

***

## 1. Observability & alerting — *you are currently flying blind*

**Problem.** No error tracking on any surface. Dropped Stripe events, failed SMS sends, and
credit-ledger drift do `console.error(...); return` — nobody is paged. Incidents surface via
customer complaints, not monitoring. PostHog is product analytics, not ops observability.

**Decision: Sentry (free tier) for errors across all surfaces + a free uptime monitor on health
endpoints.** Sentry's free tier (5k errors/mo) is the right "free to start" pick — best-in-class
React/Vite, Expo, and Cloudflare Workers SDKs, source maps, release tracking, and free alert rules
to Slack/email. Fully-free-forever self-host alternative is GlitchTip (Sentry-API-compatible), but
it's ops overhead we don't want on day one; start on Sentry's free tier and only migrate if volume
forces it.

**Approach (three surfaces, one nuance):**

* **Frontends + mobile (free, immediate):** add the Sentry SDK to `apps/dashboard`, `apps/web`,
  and `apps/mobile`. Wire source-map upload into the Cloudflare/EAS builds. This alone gives crash
  and error visibility for the entire user-facing surface today.
* **Convex backend (two phases):** Convex's native exception-reporting/log-stream integrations
  (Sentry/Axiom/Datadog) are a **Convex Pro** feature, not on the free Starter plan.
  * *Phase 1 (free):* a thin `reportError()` helper that POSTs exceptions to Sentry via `fetch`
    from the Convex runtime, wrapped around mutations/actions (a `withObservability` wrapper) and
    called explicitly at the existing `console.error` sites in the billing/webhook/SMS paths.
  * *Phase 2 (when on Convex Pro — you'll want it for production anyway):* enable the native Sentry
    log-stream integration for full function-level capture + log retention, and retire the manual
    wrapper.
* **Uptime (free):** BetterStack (Better Uptime) or UptimeRobot free tier pinging the Convex HTTP
  health route, the dashboard, the web root, and the CDN worker. Pairs with the deploy smoke tests
  in §2.

**Alert rules that must exist from day one** (Sentry → Slack/email), tagged by area:

* Any error in the Stripe webhook handler (`http.ts` billing routes) or `organizationSettings`
  subscription handlers — including the "event dropped" `console.error` branches.
* SMS send failures / GatewayAPI errors.
* Credit-ledger anomalies (debit failures after retries, negative-balance guard trips).
* Spike detection: error-rate jump after a deploy (ties to rollback, §2).

**Effort:** frontends \~1 day; Convex Phase-1 wrapper \~1–2 days; uptime \~2 hours. **Launch-blocker.**

***

## 2. Deploy safety — *modeled on `adamtrip-solutions/fogos-portugal`*

**Problem.** `deploy.yml` pushes **straight to prod on every merge to `main`**, including live Convex
schema pushes, with **no staging gate, no post-deploy smoke test, and no documented rollback**. One
bad merge is a live, customer-facing incident with no early warning.

**What fogos-portugal does right (and we should adopt the principles of):**

* **Release-gated CD, not push-to-main.** Deploy triggers only on a published GitHub release
  (`on: release: published`) or manual `workflow_dispatch` — never on PR/merge. Deploys are
  deliberate and versioned (tags `api-v*` / `web-v*`).
* **`environment: production`** on the deploy job — enables GitHub environment protection (required
  reviewer / wait timer) as a human gate.
* **Health-gated rollout + post-deploy smoke tests.** `docker compose up -d --wait` plus explicit
  `curl` smoke steps (`/healthz/ready`, a real GraphQL query, the RSS feed, the web root) with
  retry/backoff — the deploy **fails** if the app isn't actually serving.
* **Version pinning + rollback.** A server-side `versions.env` pins the deployed version; rollback =
  re-pin a prior version and re-run.

**Translation to CalGest's serverless stack** (we can't copy the Docker/self-hosted model, but every
principle maps):

1. **Switch to release-gated deploys.** You already have `release-please` config + `version-bump`
   actions. Invert the current order: cut the release *first*, deploy *on the release event*, so
   every prod deploy corresponds to a versioned, reviewable release. Stop deploying on raw push to
   `main`.
2. **Add a `staging` Convex deployment + gate.** Convex supports multiple deployments — deploy to a
   staging deployment first, run smoke tests against it, then promote to prod. Put prod behind
   `environment: production` with a required approval.
3. **Add post-deploy smoke tests** (the biggest current gap), directly modeled on fogos's smoke
   step: after each deploy, `curl` with retry/backoff against
   * the Convex HTTP **health route** (add a `/healthz` HTTP action that checks DB reachability),
   * the dashboard URL (expects 200 + an app shell marker),
   * the web root,
   * the CDN worker (a known asset path returns the right status).
     Fail the deploy job — and alert (§1) — if any smoke check fails.
4. **Document + script rollback.** Cloudflare Workers/Pages keep deployment history: `wrangler
   rollback` (or the dashboard) is a one-click revert for dashboard/web/CDN. Convex: redeploy the
   prior release tag / use the Convex dashboard's deployment history. Write this as a `deploy/README`
   runbook (fogos keeps a `deploy/README.md` — mirror that) so rollback is a known 2-minute path,
   not an improvised one under pressure.
5. **Keep concurrency `deploy-production` with `cancel-in-progress: false`** (fogos does this) so
   deploys never race.

**Migration-safety note (Convex-specific):** schema changes push live. Adopt the
`widen → migrate → narrow` discipline (the repo has `migrations.ts` and a
`convex-migration-helper` skill) for any breaking field change, and test migrations against a
production-like dataset in staging before promoting. This is part of "deploy safety," not a separate
workstream.

**Effort:** \~3–5 days (health endpoint + smoke steps + staging deploy + release-gating rewire +
runbook). **Launch-blocker** (staging gate + smoke tests + rollback runbook are the must-haves;
release-please rewire can follow).

***

## 3. Expand security & test coverage — *audit in progress*

**Problem.** The July audit was \~95% and sampling-based; it found a *class* of bug (client-supplied
refs written without org validation) that likely recurs elsewhere. The frontends were only
diff-reviewed, mobile not reviewed at all, and there are no end-to-end tests.

**Completed (July 2026, both clean):**

* **Backend mutation-ref sweep** — exhaustive hunt for every remaining instance of the
  recurrence-class bug. Result: **115 public mutations/actions scanned, \~90 take reference ids, all
  validated, 0 flagged.** The recurrence.ts fix was the last instance of its class; the ownership-
  check pattern (`assertEntityRefsBelongToOrg` / `ensure*BelongToOrg` / orgId-scoped index lookups)
  is applied uniformly, including in the batch/reorder/bulk paths where misses usually hide.
* **Frontend security review** (dashboard + web) — **no medium-or-higher findings.** Convex is the
  authoritative authz layer; client guards are correctly cosmetic-only; no server secrets in the
  bundle (all `VITE_`-prefixed public keys); open-redirect defended; AI/markdown rendered through a
  hardened renderer; PostHog carries only operator data (no customer PII), consent-gated. One flagged
  item verified safe (`getAdminNavBadges` gates on `requirePlatformStaff`).

**Two narrow follow-up sweeps** the mutation/action sweep explicitly did not cover (do for
completeness):

* **Public query read-IDOR sweep** — `notificationCredits` and `analytics` expose public *queries*
  (outside the mutation/action scope); spot-checked as org-scoped but not exhaustively swept.
* **List-query field projection** — confirm customer/employee list queries project minimal fields
  rather than full PII rows (frontend review's one open low item).
* (`http.ts` asset-download endpoints are signed-token based and the CDN worker's token→key binding
  was already verified safe in the July audit — no further action.)

**Still to schedule:**

* **Mobile review + typecheck debt.** Mobile has 23 pre-existing type errors and thin tests; it's
  the least mature surface. Fix the type errors (so it can join the CI `check-types` job — see the
  CI work already merged), then a security pass on token handling (SecureStore), deep-link/OAuth
  handling, and the same ref-validation class on any mobile-initiated mutations.
* **End-to-end tests** for the critical journeys: signup → subscribe → book → reminder fires →
  credit debits → cancel/refund. None exist today; these are what catch cross-subsystem regressions
  the unit tests miss.
* **Load test the 09:00 reminder fan-out** (audit finding F4). The credit-ledger sharding serializes
  debits (safe, but retry-storm-prone under a concurrent fan-out). Prove it holds at realistic org
  volume; if it retry-storms, migrate to the Convex `sharded-counter` component.

**Cadence:** make the ref-validation sweep a recurring check (a lightweight lint/test that fails if a
public mutation writes a `v.id()` arg without going through the ownership helper), so this class
can't regress.

**Effort:** sweep remediation \~1–2 days once results land; frontend fixes TBD by findings; mobile
\~1 week; E2E harness \~1 week; load test \~2 days. Ref-sweep remediation is **launch-blocker**; E2E and
mobile are **fast-follow**.

***

## 4. GDPR / compliance — *EU/PT market: likely a hard launch blocker*

**Problem.** You are a **data processor** for the salons' end-customer PII (names, phones, emails,
plus aftercare notes and photos that may be special-category/health-adjacent for personal-services).
Clerk's deletion covers the *account owner*, not their customers. For an EU self-serve launch this is
a legal blocker, not a nice-to-have.

**Phased implementation plan:**

**Phase A — Data map & tooling (engineering):**

1. **Data inventory (ROPA).** Enumerate every store of personal data: Convex tables (`customers`,
   `appointments`, `aftercare`, `photos`, `customFieldValues`, tracking), Clerk (owners), Stripe
   (billing), GatewayAPI (phone numbers), PostHog (behavioral), Sentry (may capture PII in errors —
   scrub it). Document controller vs. processor role for each.
2. **Right to erasure.** Extend/verify the existing `customers.remove` cascade to be a *complete*
   hard-delete: appointments unlink, custom field values, tracking, aftercare, and **R2 photo
   objects** (confirm storage objects are actually deleted, not just DB rows). Add an **org-level
   account deletion** (delete-my-business) that purges all tenant data and cancels Stripe.
3. **Right to access / portability.** A function that exports all data for a given end-customer (and
   an org-level export) to a JSON/CSV bundle.
4. **Retention & minimization.** Define retention windows and implement purge crons (old
   soft-deleted records, aged SMS logs, stale appointments). Scrub PII from Sentry payloads and
   limit what's sent to PostHog (ties to §3 frontend review).

**Phase B — Legal & policy (parallel, not eng-blocking):**
5\. **Sub-processor DPAs.** Sign DPAs with Convex, Clerk, Stripe, GatewayAPI, Cloudflare, PostHog,
Sentry. Publish a sub-processor list + privacy policy.
6\. **Customer-facing DPA.** You're a processor for salons (controllers) — add a DPA to your ToS.
7\. **ROPA + breach-notification process** (72-hour) documented.
8\. **Special-category assessment.** Get a legal read on whether aftercare/photos for beauty count as
special-category data; if so, add explicit-consent capture + extra safeguards.

**Consent** (cookie consent already exists): ensure analytics/marketing consent is captured and
respected end-to-end.

**Effort:** Phase A tooling \~1.5–2 weeks; Phase B is legal-led, run in parallel. **Launch-blocker for
EU self-serve** (erasure + export + DPAs + privacy policy are the minimum bar).

***

## 5. Abuse & cost safeguards on expensive operations

**Problem.** Booking spam is throttled and AI is now metered, but **authenticated mutations are
largely unthrottled**, and SMS is real money — a compromised or malicious account can burn credits
down to the safety floor, trigger bulk operations, or spam invitations.

**Good news:** the infrastructure already exists — `@convex-dev/rate-limiter` is registered
(`convex.config.ts`) and used for booking. Extend it into a coherent policy.

**Design — a limit taxonomy + a shared helper:**

* **Public (unauthenticated):** booking (done), any other public endpoint — strict per-IP/per-contact
  limits.
* **Authenticated read:** generous per-user limits (cheap).
* **Authenticated write:** moderate per-user + per-org limits.
* **Billable / expensive:** SMS-triggering ops, bulk/batch mutations, exports, invitations, AI —
  tight per-user **and** per-org limits, plus a **daily spend circuit-breaker** per org (cap
  SMS/credit burn per day beyond the existing per-send safety floor; AI token cap already enforced).
* **Idempotency** on any retry-able mutation with money/side effects (extends the refund/pack-purchase
  idempotency already in place).

**Anomaly safeguard:** alert (§1) when an org's SMS/credit burn or AI spend spikes above its trailing
baseline — catches a compromised account before the bill does.

**Implementation:** a `withRateLimit(kind)` wrapper applied per endpoint by category, so limits are
declarative and auditable rather than scattered. Start by covering the billable tier (highest risk),
then write, then reads.

**Effort:** \~3–4 days for the billable tier + circuit-breaker + anomaly alert; the rest incremental.
**Fast-follow** (billable-tier caps are near-launch-blocker given real money is involved).

***

## Sequenced roadmap

**Launch-blockers (do before real customers + real money):**

1. **Observability + alerting** (§1) — Sentry on all surfaces + billing/webhook/SMS alert rules +
   uptime monitor. *Highest priority: you cannot run billing blind.*
2. **Stripe reconciliation + global idempotency** (from the billing audit's F1, folded here) — a
   `processedStripeEvents` dedup table + a nightly job reconciling DB subscription state against
   Stripe, alerting on drift. *Second-highest: this is where silent revenue loss lives.*
3. **Deploy safety** (§2) — staging gate + post-deploy smoke tests + rollback runbook.
4. **GDPR Phase A tooling + Phase B minimum** (§4) — erasure, export, DPAs, privacy policy (EU).
5. **Billable-tier rate limits + circuit-breaker** (§5).

*(The ref-validation sweep is done — 0 findings — so it is no longer a blocker; add the recurring
lint/test guard from §3 so the class can't regress.)*

**Fast-follow (first weeks post-launch):**

* E2E tests for critical journeys (§3).
* Mobile type-error cleanup → CI type-check + mobile security pass (§3).
* Load-test the 09:00 fan-out; migrate to sharded-counter if needed (§3).
* Release-please-gated deploys (§2) and full rate-limit taxonomy (§5).

**Rough total to launch-ready:** \~3–4 focused weeks, observability and Stripe reconciliation first.
None of it is glamorous; all of it is what lets you sleep once real money flows.

***

## Confidence note

Security and billing *correctness* assessments are high-confidence (code read directly). The ops,
scale, mobile, and compliance assessments are based on checking what's present, not stress-testing
depth — treat effort estimates as planning-grade, to be refined per workstream.