Skip to main content

​
Deploy & Rollback Runbook

CalGest ships with release-please driving versioned, per-component deploys.
  1. Conventional-commit PRs merge to main.
  2. The Release Please workflow (.github/workflows/release-please.yml) keeps one rolling release PR open, updating each component’s version + CHANGELOG.md.
  3. Merging that PR tags the bumped components (<component>@X.Y.Z) and creates a GitHub Release for each.
  4. By default, an operator then dispatches the Deploy workflow (.github/workflows/deploy.yml) for each released component, pinned to the new tag, with confirm_production: true. Use the order backend → cdn → dashboard → web → mobile. Automatic sequential rollout exists only when the repository variable AUTO_DEPLOY_RELEASES is explicitly set to true; it is disabled by default because this private repository’s current GitHub plan does not support required environment reviewers.
Deploy is workflow_dispatch only (component + version inputs): a merge to main never deploys a surface by itself — only the release workflow (or an operator) dispatches it. Each dispatch checks out the <component>@<version> tag, re-runs the full preflight (types, lint, test, build, doctor, audit), then runs the component’s deploy + a post-deploy smoke test — if the app isn’t serving, the deploy job fails and turns red. Manual deploy / rollback: dispatch Deploy with the target component, an already-released version (e.g. an older tag), and confirm_production: true to roll a surface back to that version. Convex backends do not truly reverse (migrations don’t unwind) — a backend “rollback” is a roll-forward redeploy of an earlier good version; see the Convex section below. This runbook covers how to tell something is broken and how to roll back each surface fast.

​
Surfaces

​
How to tell it’s broken

  1. The deploy pipeline is red. A failed smoke-test step is the first signal — check the failing job’s log for the ::error:: line.
  2. Manual smoke checks (same as CI — run any of these locally):
  3. Uptime / error alerting (Sentry, uptime monitor) firing on the surface.
If a deploy job stayed green but the app is misbehaving, roll back the affected surface below, then investigate.

​
Rollback — happy path (~2 min per surface)

​
Cloudflare — Dashboard & Web (Vite deploy)

Both are Cloudflare Workers/Pages-style deployments driven by Wrangler.
  • wrangler rollback with no ID rolls back to the immediately previous deployment.
  • Dashboard UI path: Cloudflare dashboard → Workers & Pages → select calgest-dashboard / calgest → Deployments → find the last-good version → ⋯ → Rollback.
  • Re-run the smoke curl above to confirm 200.

​
Cloudflare — CDN Worker

Confirm with curl -s -o /dev/null -w '%{http_code}\n' https://cdn.calgest.com/ (expect a non-5xx, normally 404).

​
Convex — Backend

Convex has no in-place “rollback” button; you roll forward by re-deploying a known-good commit.
  • Inspect deploy history & logs: Convex dashboard → your deployment → Functions / Logs / History to identify the last-good version and any failing function.
  • If the break is a bad schema/migration rather than code, coordinate with the backend owner — a forward-fix migration is usually safer than reverting schema.
  • Confirm with curl -fsS https://<deployment>.convex.site/healthz/ready → {"status":"ok"}.

​
Mobile — Expo EAS Update (OTA)

EAS Updates are immutable; you “roll back” by republishing a previous update to the production channel so clients pull the good bundle on next launch.
Alternative: eas channel:edit production --branch <good-branch> to point the production channel at a different branch entirely. Native binary changes cannot be rolled back via OTA — those require a new store build.

​
Native fingerprint gate (mobile)

The mobile app pins runtimeVersion.policy: "fingerprint" (apps/mobile/app.json). An EAS Update only reaches an install whose native binary carries the same fingerprint runtime version — an OTA cannot cross a native-runtime change (new native module, bumped native dep, changed native config, etc.). To stop a deploy from publishing an update that no shipped build can accept, deploy-mobile runs a Fingerprint gate step right before eas update:
  • For each platform (ios, android) it takes the latest finished production build (eas build:list --build-profile production --status finished) and compares its fingerprint to the checked-out commit’s project fingerprint (eas fingerprint:compare --build-id <id> --environment production --json, hashes compared with jq).
  • No production build yet for either platform → fails. Publishing an OTA that cannot reach a released binary is never considered a successful deploy.
  • Any platform’s fingerprints differ → the job fails with ::error::: the update would target a runtime no shipped build has.
  • All match → the OTA publishes.
When the gate fails — operator procedure:
  1. Dispatch the Mobile Build workflow (.github/workflows/mobile-build.yml) for the affected platform(s) — normally platform: all, profile: production, submit: true so EAS builds and submits to the stores in one go.
  2. Wait for the store release to go live (App Store review / Play Store rollout). Users must actually be able to install the new native build.
  3. Re-dispatch Deploy with component: mobile and the same released version. With a matching native build now live, the gate passes and the OTA publishes.
Reviewer heads-up. The PR checks include a Native fingerprint check job (pr.yml → fingerprint) that fingerprints the PR base vs. head. When a PR changes the fingerprint it labels the PR needs-native-build and posts a sticky comment listing what moved — so you know at review time that merging + releasing will trip this gate and a native build must be planned. It never fails the PR; it is purely informational.

​
Mobile version floor (force update)

Independent of the fingerprint gate, the backend can hard-block clients running a JS bundle older than a minimum supported version — the escape hatch for a breaking backend change that old app versions cannot tolerate.
  • The backend exposes a public query mobileVersion.check. The app calls it on launch and, when its running JS version is below the floor, shows a blocking “Update required” screen (with a store button when a store URL is configured).
  • Configured via Convex env vars on the production deployment:
    • MIN_SUPPORTED_MOBILE_VERSION — semver floor, e.g. 1.2.0. Unset = no floor (every version is supported).
    • IOS_APP_STORE_URL, ANDROID_PLAY_STORE_URL — App Store / Play Store listing URLs for the update button. Unset = the button is hidden on that platform.
Operator procedure for a breaking backend change:
  1. Keep the old backend functions in place until the floor covers every client that still calls them (widen-migrate-narrow — never remove first).
  2. Ship the new app version to the stores / OTA and give it an adoption window.
  3. Raise the floor:
  4. Wait an update-adoption window so users on supported versions have moved up.
  5. Only then remove the deprecated functions.
⚠️ Raising the floor hard-blocks everyone below it — they cannot use the app until they update. Treat it as a last resort, and only raise it once the new version is genuinely available (store-released and/or OTA-published). Set it too high and you lock out users who have no version to update to. To relax it, set a lower semver, or clear it entirely with npx convex env remove MIN_SUPPORTED_MOBILE_VERSION --prod.

​
Production configuration gates

Every deployment now validates required variable names before it can mutate the target. Values are never printed. The backend also runs productionConfig:validate against the deployed Convex environment, and mobile runs its app-local validator through eas env:exec --environment production. Before the first mobile build, populate the EAS production environment from the Expo dashboard or eas env:create. Do not copy development Clerk/Convex values. At minimum configure Convex, Clerk, Google OAuth, and Sentry variables documented by apps/mobile/.env.example. Use EAS Sensitive visibility for SENTRY_AUTH_TOKEN (and Plain text or Sensitive for the public build variables), not Secret visibility: eas env:exec cannot read Secret-visibility variables, so the production preflight would correctly fail.

​
Notes & required account-side steps

  • CDN isolation. Production uses the EU calgest-prod R2 bucket and dev uses calgest-dev. Create calgest-prod before the first deployment. The production workflow only writes the production worker’s ASSET_TOKEN_SECRET; configure a separate secret directly on calgest-cdn-dev when deploying the dev worker.
  • Environment protection. Each deploy job targets the production GitHub Environment. Required reviewers were attempted but GitHub returned HTTP 422 because the current private-repository plan does not support them. The enforced fallback is manual dispatch plus confirm_production: true; keep AUTO_DEPLOY_RELEASES unset. Enable a required reviewer if the plan changes.
  • Staging. A separate staging Convex deployment, Cloudflare workers, EAS channel, DNS, and secrets require account-side provisioning and cannot be fabricated from repository code. Until those targets exist, preview builds plus the full preflight and the manual production confirmation are mandatory. Once provisioned, promote only after the same readiness, signed-CDN, app-marker, and mobile artifact checks pass in staging.