Runbook

Day-two operations for a deployed portal. Commands run from cloud/reform-assistant/ and use the local Wrangler (npx wrangler). Add --remote to any D1 or R2 command that must act on the deployed resources; without it, Wrangler may read the local emulated copy instead.

Before production

Work through this list before sending real families to the portal. Each item links to the page that explains it.

• [ ] APP_URL. Set vars.APP_URL in wrangler.jsonc to the HTTPS origin the portal is served from (the https://reform-assistant.<your-subdomain>.workers.dev URL until the custom domain exists) and redeploy. The default http://localhost:8787 would send families back to localhost after Stripe Checkout and the billing portal. It does not control the cookie's Secure flag, which follows the request scheme. See Cloudflare deployment.
• [ ] Real D1 database_id. Run npx wrangler d1 create reform-assistant, replace the all-zero placeholder database_id in wrangler.jsonc with the printed id, create the R2 bucket (npx wrangler r2 bucket create reform-assistant-files), and apply the schema with npx wrangler d1 migrations apply reform-assistant --remote. See Cloudflare deployment.
• [ ] Secrets. Set all six with npx wrangler secret put: STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, STRIPE_PRICE_MONTHLY, STRIPE_PRICE_QUARTERLY, STRIPE_PRICE_ANNUAL, and HERMES_LINK_TOKEN (generate it with openssl rand -hex 32 and store the same value in the Hermes token file). Confirm with npx wrangler secret list. STRIPE_SECRET_KEY must be set wherever webhooks arrive, because the webhook handler uses it to re-fetch the current subscription.
• [ ] Stripe keys: sandbox first, then live. Test everything with the test-mode (sandbox) secret key sk_test_... and test-mode Prices. For live, create a separate live-mode secret key, three live-mode recurring Prices ($8.00 per month, $21.60 every 3 months, $81.60 every 12 months), and save the Customer portal settings in live mode. The key, the three Price ids, and the webhook endpoint with its signing secret must all come from the same mode, because test and live objects are separate; mixing them produces 502 stripe_error. See Stripe setup and Going live.
• [ ] Webhook endpoint. In the Stripe Dashboard (same mode as the key) add an endpoint for https://<your-origin>/api/stripe/webhook that sends checkout.session.completed, checkout.session.expired, and customer.subscription.created|updated|deleted. Store its signing secret as STRIPE_WEBHOOK_SECRET (it differs from the stripe listen secret), then check that deliveries show 200 in the Dashboard. See Stripe setup.
• [ ] Hermes host. Create the token file, the wrapper script with REFORM_PORTAL_URL and PYTHONPATH, and the no-agent cron job, then run it once by hand and confirm exit code 0. See Hermes bridge.
• [ ] Custom domain after the domain transfer. Once the DNS zone is on the Cloudflare account, attach the domain to the Worker (Custom Domain), change APP_URL to https://<domain> and redeploy, edit the Stripe webhook endpoint URL to https://<domain>/api/stripe/webhook, and change REFORM_PORTAL_URL in the Hermes wrapper. Families will need to log in once more because the session cookie is host-scoped. See Cloudflare deployment.
• [ ] Smoke test. curl -s https://<your-origin>/api/plans returns the plans; sign up over HTTPS and check the Set-Cookie header includes Secure; pay with a test card in the sandbox and watch the inmate go from Connecting... to Connected after the bridge runs.

Rotate the Hermes link token

HERMES_LINK_TOKEN is a shared secret between the Worker and the Hermes host. The Worker rejects every /api/hermes/* request whose bearer token does not match, so rotate both sides together.

1. Generate a new value:

   openssl rand -hex 32

2. Set it on the Worker (this prompts for the value and takes effect for new requests):

   npx wrangler secret put HERMES_LINK_TOKEN

3. On the Hermes host, replace the contents of the token file (default $HERMES_HOME/securus/portal_link.token, or whatever --token-file the cron job uses) with the same value:

   printf '%s' 'NEW_TOKEN' > "$HOME/.hermes/securus/portal_link.token"
   chmod 600 "$HOME/.hermes/securus/portal_link.token"

4. Run the bridge once and read its output:

   hermes cron run <job_id>

   or run portal_sync.py by hand as shown in Hermes bridge. A correct token gives a JSON summary and no "Could not retrieve pending portal connections." message on stderr.

Between steps 2 and 3 the bridge's requests get 401 unauthorized. The bridge logs these as request failures and exits 1, so Hermes may deliver an error alert for those runs; nothing is lost because pending connections and usage ledger entries are retried on the next successful run.

Replay a Stripe event

The Worker records each processed event id in stripe_events and answers duplicates with {"received":true,"duplicate":true} without reprocessing. What to do depends on why you want a replay.

The event failed on first delivery. If archiving or handling threw, the Worker already deleted the event id from stripe_events and returned an error status. Stripe retries failed deliveries on its own; nothing to do on the portal side. Check the Worker logs (npx wrangler tail) for the cause.

You want to reprocess an event that already succeeded.

1. Find the archived payload in R2 if you need to inspect it:

   npx wrangler r2 object get reform-assistant-files/stripe/events/evt_XXXX.json --remote --file evt.json

2. Release the event id so the Worker will accept it again:

   npx wrangler d1 execute reform-assistant --remote \
     --command "DELETE FROM stripe_events WHERE event_id = 'evt_XXXX'"

3. Ask Stripe to resend it: in the Dashboard open Developers, Events, select the event, and use Resend for the endpoint; or from the Stripe CLI, stripe events resend evt_XXXX --webhook-endpoint we_XXXX (for local testing with stripe listen, resend while the listener runs).

With STRIPE_SECRET_KEY set, subscription events are not applied from the payload: the handler re-fetches the subscription from Stripe and writes its current state, so replaying an old customer.subscription.updated after a newer one does not roll the state back. An event for a different Stripe subscription than the account's live one is never applied: it is recorded at stripe/duplicates/<subscription_id>.json and that subscription is cancelled in Stripe (a replay of such an event cancels it again, which is harmless). Without STRIPE_SECRET_KEY (local setups), the payload is applied as-is and an old event does overwrite newer state, so replay only the most recent event for a subscription. A replay never re-queues a connection that is already connected while the subscription stays active or trialing (see Stripe webhooks); a connection that is failed or pending stays or becomes pending.

Inspect D1

Use wrangler d1 execute with --remote for the deployed database (omit it, or use --local, for the local one):

npx wrangler d1 execute reform-assistant --remote --command "SELECT COUNT(*) AS users FROM users"

Useful read-only queries:

Question	SQL
Subscriptions and their state	SELECT user_id, plan_id, status, current_period_start, current_period_end, cancel_at_period_end, stripe_subscription_id FROM subscriptions ORDER BY updated_at DESC LIMIT 20
Connections waiting for Hermes	SELECT c.id, i.inmate_number, c.status, c.period_start, s.status AS subscription_status FROM connections c JOIN inmates i ON i.id = c.inmate_id JOIN subscriptions s ON s.id = c.subscription_id WHERE c.status = 'pending'
Failed connections	SELECT c.id, i.inmate_number, c.detail, c.updated_at FROM connections c JOIN inmates i ON i.id = c.inmate_id WHERE c.status = 'failed'
Recent usage for an inmate	SELECT occurred_at, kind, stamps, message_id FROM usage_events WHERE inmate_id = (SELECT id FROM inmates WHERE inmate_number = '12345' LIMIT 1) ORDER BY occurred_at DESC LIMIT 20
Recently processed Stripe events	SELECT event_id, event_type, received_at FROM stripe_events ORDER BY received_at DESC LIMIT 20

Pass a query with --command "..."; wrap it in single quotes for the shell if it contains double quotes. Avoid selecting password_hash, password_salt, and session token_hash values, and avoid writing to those tables by hand.

Re-queue a failed connection

A connection acknowledged as failed is not returned by GET /api/hermes/connections?status=pending. Once the cause (usually the Securus CLI or the Hermes store) is fixed, put it back in the queue. Both updates are needed, and the connection is returned only while its subscription is active or trialing:

npx wrangler d1 execute reform-assistant --remote --command \
  "UPDATE inmates SET connection_status = 'pending', connection_detail = NULL WHERE id = (SELECT inmate_id FROM connections WHERE id = 'CONNECTION_ID'); UPDATE connections SET status = 'pending', detail = NULL WHERE id = 'CONNECTION_ID'"

The next bridge run picks it up. A new customer.subscription.created or updated event with active or trialing status for the subscription does the same.

Common errors

Symptom	Cause	Fix
Console says "Payments are not switched on yet (sandbox setup pending)." or the API returns 503 billing_not_configured	STRIPE_SECRET_KEY, the plan's STRIPE_PRICE_* secret, or (for the webhook) STRIPE_WEBHOOK_SECRET is not set on this environment	Set the missing secret (Cloudflare deployment); locally, edit .dev.vars and restart wrangler dev
502 stripe_error from checkout or portal	Stripe rejected the request: key and Price from different modes, a wrong Price id, or no Customer portal configuration saved in that mode	Check the secrets against the Dashboard mode; save the Customer portal settings; look at the request in Stripe's Developers, Logs
Webhook deliveries show 400 invalid_signature	STRIPE_WEBHOOK_SECRET belongs to a different endpoint than the one sending (the stripe listen secret differs from a Dashboard endpoint's), the body was altered, or the timestamp is more than 300 seconds from the Worker's current time	Use the secret of the endpoint that sends the events; resend a fresh event
Webhook deliveries show 503 billing_not_configured	STRIPE_WEBHOOK_SECRET not set	Set it and redeploy or restart
Payment succeeded but the inmate stays Connecting...	Hermes has not run or cannot reach the portal; or the webhook never produced a pending connection (no inmate at the time of the event, user could not be resolved, or the event was treated as a duplicate because the account has a different live subscription)	Run the bridge by hand; query pending connections; check stripe_events for the event and the R2 archive for its payload (and stripe/duplicates/ for a duplicate record)
Family was charged twice	Two payable Checkout Sessions existed, or STRIPE_SECRET_KEY was not set when the second subscription's event arrived, so it was recorded but not cancelled	Look for stripe/duplicates/<subscription_id>.json in the R2 bucket (canceled: true means Stripe already cancelled it). Otherwise cancel the extra subscription in the Dashboard and refund. Checkout normally expires the previous open session before creating a new one
Inmate shows Connection failed	The bridge acknowledged failed because the Securus CLI call failed	Fix the Hermes side, then re-queue
Bridge exits with code 2	REFORM_PORTAL_URL is missing or invalid, or the token file is unreadable or empty	Fix the environment in the wrapper script or the token file
Bridge exits with code 1	At least one operation failed in that run: portal unreachable or 401, a Securus CLI failure or timeout, a failed refill or revoke, a rejected usage post, an unreadable store or state file, or an unexpected exception (the cron boundary prints a full traceback to stderr and the summary shows failed: 1). The JSON summary and stderr say which	Read the stderr messages and fix the cause; the next run retries (see Hermes bridge). A traceback mentioning hermes_constants means the repository root is missing from PYTHONPATH
Bridge prints "Could not retrieve pending portal connections." and nothing is connected	Portal unreachable, wrong origin, or 401 from a token mismatch	Check REFORM_PORTAL_URL, then rotate or re-copy the token
409 already_subscribed from checkout	The account's own subscription is active, trialing, or past_due	The family should use Manage billing (the billing portal) to change plans
409 inmate_taken from checkout	Another account has a live subscription for the same inmate_number	Check both accounts' inmate numbers; contact the families or fix a mistyped number
Bridge usage post fails with 404 unknown_inmate	No account has a live (active, trialing, or past_due) subscription for that inmate number, for example the subscription was canceled before the spend was reported	Check subscriptions for the inmate number; the bridge only reports usage for inmates listed in /api/hermes/allowances, so this is rare
409 inmate_locked	The family tried to change the inmate number, or remove the inmate, while the subscription is active, trialing or past_due	Cancel the plan first, or fix the data by hand if the number was entered wrongly
409 email_taken on signup	An account with that normalized (trimmed, lowercased) email exists	The family should log in instead
Users are logged out unexpectedly on a new origin	The session cookie is host-scoped, and it is Secure on every HTTPS request (the flag follows the request scheme, not APP_URL)	After changing domain, expect one re-login. Make sure the site is served over HTTPS and APP_URL is the real HTTPS origin so Stripe redirects land on it
Deploy fails on D1 with an invalid database id	wrangler.jsonc still has the all-zero placeholder database_id	Run wrangler d1 create reform-assistant and paste the id (Cloudflare deployment)
404 not_found JSON for a page	The path starts with /api/ and is not a route, so the Worker (not the static assets) answers	Fix the URL; static pages live outside /api/