Frozen. A record of a decision at a point in time, not living documentation. Do not update it — supersede it with a new record instead. For how this works today, see docs/explanation/ and docs/generated/.

Payment Charge-Gating & Resilience Plan

Status: Phases 0–4 implemented (2026-07-21, build-verified; needs runtime QA on a test user). Phase 5 partial — see §5. Owner: Farhan

Implementation note (2026-07-21). Phases 0–4 are code-complete for the customer Initial (processPayment) path and build green; Airtable is removed from the whole payment/bundle flow (Initial, Addition, Reconnect, admin add-to-one-sub, passport provisioning). The executor auto-charge path is deleted, so every run now parks in AWAITING_ADMIN and only triggerPaymentForRun moves money. Not yet runtime-tested end-to-end (build + unit specs only). Phase 5: CPX already has a 30s HTTP timeout; charge-idempotency key and AsyncLocalStorage userId log-stamping are deferred as isolated, runtime-tested follow-ups (both touch live money code / the global logging pipeline and should not ship unverified).

Builds on and partially supersedes docs/bundle-run-redesign.md. That doc designed the BundleRun model as a parallel source of truth with charges auto-firing on fulfillment and Airtable left untouched. This plan makes the BundleRun the primary, durable object, gates all charging behind an explicit admin trigger, and removes Airtable from the payment flow. It exists because a real prod payment was silently lost (see §1).

Supersedes in bundle-run-redesign.md:

• Decision #3 (autoChargeOnFulfillment default = ON) → admin-trigger-only (hard): the auto-charge-on-fulfillment path is removed. Charging happens only via triggerPaymentForRun.
• Decision #9 ("Airtable left exactly as-is") → Airtable is removed from the payment/bundle-run flow (utilities/feedback/OTP Airtable untouched — separate migration).

---

1. Why — the incident

On 2026-07-20 20:40 (prod), user Tolu Adetayo called processPayment. The synchronous balance check passed and the API returned 200 with "Payment processing started…" — but no charge, no BundleRun, no email, and nothing in admin. Confirmed from prod logs + DB (no backgroundjobs / bundleruns for that date). See memory processpayment-silent-background-failure.

Root cause is structural, in payment-orchestration.service.ts:

• Gap A — the BundleRun is created last, in the fragile spot. createRunForBundul runs at ~L867, after ~24 Airtable calls + Passport/CPX provisioning, all inside a fire-and-forget void (async () => {})() block (begins ~L367). It "swallows its own errors and returns null," so any early hang/throw = no run, silently. Tolu's block died at the first Airtable await.
• Gap B — Airtable is on the critical path. ~24 awaited updateProcessingRecord audit writes gate run creation; each is a hang point. Pure telemetry — nothing reads them back.
• Gap C — no admin notification. AWAITING_ADMIN is set in 3 places (bundle-run.service.ts:530, bundle-run-executor.service.ts:313/316); none notify.
• Plus: the 200 is returned before anything durable exists; the per-run chargePolicy.autoChargeOnFulfillment schema default (true) contradicts the global BundleRunSetting default (false).

---

2. What already exists (do NOT rebuild)

Capability	Where
BundleRun + BundleRunItem state machine, AWAITING_ADMIN status	src/bundle-run/schemas/*, enums/bundle-run.enums.ts
Per-run + global charge-hold flag (autoChargeOnFulfillment, global default already false)	bundle-run-setting.schema.ts, bundle-run.service.ts:157-163
Admin "charge now" = triggerPaymentForRun(runId, itemIds?) (also retry; per-user lock; skips CREATED; re-derives Initial/Addition from live state)	bundle-run-admin.resolver.ts:34, bundle-run-admin.service.ts:125-188, bundle-run-payment.service.ts:66-174
Mint-on-reveal VC at admin card reveal	bundle-run-admin.service.ts:673-748 (_revealCardForSub)
Admin actions: claim, mark-handled, reveal creds/card, convert-to-manual, mark-manual-outcome, retry, rerun, per-run toggle	bundle-run-admin.service.ts
Event-driven executor (concurrency 3, never throws) + 10-min reconcile cron for stuck runs	bundle-run-executor.service.ts, bundle-run-reconcile.job.ts
Admin alert path (email → OPS_ALERT_EMAILS)	NotificationsService.sendAdminAlert, notifications.service.ts:252
Durable job model + status + reconcile pattern	payment/schemas/background-job.schema.ts

The target flow is ~80% present. This plan is wiring + hardening + removals, not a rebuild.

---

3. Locked decisions (this effort)

1. Charge gating = admin-trigger-only (hard). Remove/disable the executor's auto-charge-on-fulfillment branch (bundle-run-executor.service.ts:206-208). Charging is possible only through triggerPaymentForRun. Align the schema default to false and drop the true default to kill the mismatch. Manual items already never auto-charge — this makes automated items match.
2. Durability = reuse Mongo BackgroundJob + reconcile cron. No Redis/queue. Persist the job before returning 200; a worker resumes it; the cron is the backstop.
3. Airtable removal = payment/bundle-run flow only. Strip it from payment-orchestration, payment-admin, and the passport charge path. Leave utilities/feedback/OTP Airtable for a later, separate migration.

---

4. Target end-state flow

customer processPayment / admin addToOneSub
  → [sync] validate user, subs, account ownership, balance   (read-only, no money)
  → [sync] persist a durable BackgroundJob (PROCESS_BUNDUL)  ← durability point
  → return 200 "we're on it"
  ─────────────────────────────────────────────────────────
  → [worker, resumable] create BundleRun FIRST (status collecting)   ← never silent
  → [worker] provision: Passport account, CPX link, ACH stamp
             (each tracked as run/item state + history; failure → AWAITING_ADMIN + reason)
  → [worker] executor swaps cards (automated) / awaits admin (manual)
  → run lands in AWAITING_ADMIN   →  sendAdminAlert  →  admin sees it in dashboard
  ─────────────────────────────────────────────────────────
  ADMIN: reviews, reveals creds/cards, edits VCs, fixes items
  ADMIN clicks "trigger payment"  →  triggerPaymentForRun  →  MONEY MOVES (only here)
  → run finalizes completed / completed_with_failures

No charge occurs anywhere before the admin trigger.

---

5. Phases (each independently shippable)

Phase 0 — Stop silent loss (ship first).

• Persist a BackgroundJob synchronously before the 200; replace the fire-and-forget void (async)() with a worker that resumes from that job. On process death the reconcile cron re-picks it up.
• Reconciliation alert: any bundul attempt with a persisted job but no BundleRun within N minutes → sendAdminAlert. Permanent backstop.

Phase 1 — Create the run first, durably.

• Reorder so createRunForBundul is the first durable action after the sync checks, before provisioning.
• Make it non-swallowing on the critical path: if the run can't be created, fail loudly + alert; never proceed silently / return null.
• Move Passport/CPX/ACH provisioning to run/item state + history entries. A provisioning failure parks the run in AWAITING_ADMIN with a clear reason.

Phase 2 — Remove Airtable from the flow (see §6 for the call-site inventory).

Phase 3 — Delete the auto-charge path.

• Remove the if (autoChargeOnFulfillment) chargeItem() branch in the executor; align schema default to false. Every run now lands in AWAITING_ADMIN. Keep triggerPaymentForRun as the sole charge entry point.

Phase 4 — Notify admin on AWAITING_ADMIN.

• Emit an event on run creation / AWAITING_ADMIN; a listener calls NotificationsService.sendAdminAlert (user, services, amount, reason). Register a new NOTIFICATION_AUTOMATIONS key bundle_run_awaiting so it shows in the admin automations overview. Optionally add push/Google-Chat to match claim/mark-handled.

Phase 5 — Resilience hardening.

• Item-level idempotency key on the Passport charge (atop the existing per-user lock + skip-CREATED) so retries can't double-charge — protects the single-fire Bundul fee.
• Hard timeouts on every remaining external call in the flow (CPX especially; Passport already has one).
• userId log-stamping (AsyncLocalStorage + pino mixin) so every run/charge line is filterable by user in BetterStack.
• Extend the reconcile cron to also catch runs stuck in collecting/provisioning, not just swapping.

---

6. Airtable removal inventory (payment/bundle flow)

Delete these call sites (audit telemetry; replaced by BundleRun status + item history + structured logs). getBundulPricing stays — it already reads the DB first and only falls back to Airtable when unseeded.

• payment-orchestration.service.ts: getAirtableRecordIDOnCustomersTable L373; createProcessingRecord L386, L497; updateProcessingRecord L455, L476, L590, L632, L648, L662, L700, L721, L853, L890, L947, L959, L974, L994, L1017, L1025 (+ the addition path L1101, L1108, L1169, L1247, L1263, L1303 and reconnect L1566, L1588).
• payment-admin.service.ts: createProcessingRecord L173.
• payment.resolver.ts: createProcessingRecord L266, L411.
• passport/services/user-passport.service.ts: updateProcessingRecord L284, L296, L308, L320.

Out of scope (leave as-is): utility-invoice Airtable (email-parser, user-subscription, subscription-write), feedback.service.ts, the airtable/triggers/* OTP subsystem, and the admin passthrough resolver. The inbound Airtable "Burger" trigger controller (airtable/triggers/controllers/airtable-trigger.controller.ts) starts payments from Airtable — confirm it's unused for new work before retiring that entry point.

---

7. Sequencing, risk, verification

• Low-risk / independently shippable: Phase 0, 2, 4.
• Behavioral change: Phase 1 + 3 (charge gating). The admin board already supports the target model, so the surface change is small; the risk is ordering/provisioning-as-state.
• No new infrastructure (decision #2).
• Verification per phase: unit + a scripted end-to-end on a test user (betatesting@bundul.io) — trigger processPayment, assert (a) a durable job exists immediately, (b) a BundleRun in AWAITING_ADMIN appears, (c) no Passport charge until triggerPaymentForRun, (d) the admin alert fires. Re-run the BetterStack funnel query (processPayment called vs AWAITING_ADMIN vs charge) to confirm no silent drops.
• Highest-risk number remains the Bundul fee — must fire exactly once per One Sub, guarded by the per-user lock + live-state Initial/Addition routing + the new idempotency key (Phase 5).