Frozen. A record of a decision at a point in time, not living documentation. Do not update it — supersede it with a new record instead. For how this works today, see docs/explanation/ and docs/generated/.

Payment Resilience Hardening

Audit + fixes done 2026-07 (branch payment-resilience; admin UI on bundul-admin branch financials-spent-vs-collected). Each item is Issue → Problem → Fix. The deep audit and rationale live in payment-resilience-research.md; this doc is the short "what was wrong and what shipped."

The through-line: the collect side (getting paid by the customer) and the disburse side (paying their providers) were decoupled — nothing stopped Bundul paying out for a customer who stopped paying. Most of this work reconnects them and makes failures visible.

---

Backend — collection failures & dunning

1. A failed monthly collection was silently dropped

• Issue: When a customer's monthly ACH collection FAILED, the webhook handler only logged and returned.
• Problem: No retry, no notification, no suspension — and VC top-ups / utility payments kept running, so Bundul kept paying providers with money it never collected. Indefinitely.
• Fix: A delinquency state machine on UserPassportSubscription (paymentStatus, collectionFailureCount, disbursementPaused): record the failure, mark past_due, schedule a backed-off retry, notify the customer + ops, and gate disbursement (the VC top-up loop skips paused users). — src/payment/orchestration/payment-retry.service.ts, src/webhook/webhook.service.ts, src/subscriptions/orchestration/virtual-card-orchestration.service.ts

2. Only one charge type could retry; no backoff, no error classification

• Issue: Retry logic existed only for "addition one-time" charges; recurring collections had none. Retries had no backoff and didn't look at why a charge failed.
• Problem: A failed month-3 collection never retried; and a permanent failure (closed account) was treated like a transient blip.
• Fix: RECURRING_CHARGE_RETRY background job with a +1d / +3d / +7d backoff (runAfter on the job), plus transient-vs-hard classification (NSF/closed/R0x → hard). Re-issued as an idempotent one-time catch-up charge. — src/payment/orchestration/payment-retry.service.ts, src/payment/schemas/background-job.schema.ts

3. Terminal state sent the wrong email

• Issue: On the final failed attempt (retries exhausted), the code still sent the "we'll retry" email.
• Problem: Customer told we'd retry when we wouldn't; no clear "your services are paused" message.
• Fix: When no retry remains, send the terminal "services paused, reactivate anytime" email instead. — src/payment/orchestration/payment-retry.service.ts
• Superseded 2026-07-10: the original decision here was pause, never cancel (lodge cards keep their number so reactivation just resumes top-ups; cancelling forces card re-issuance + re-registration per provider). That tradeoff no longer holds given the CPX top-up bug ([[vc-topup-newcard-bug]]) — see the 6-month-prefund workaround for live VC customers.

4. Reactivation waited for the next monthly cycle

• Issue: A paused customer only recovered when their next monthly charge happened to succeed — potentially weeks away.
• Problem: Customer fixes their payment method but services stay off for days.
• Fix: When a customer (re)links their bank, emit payment.method_updated → immediate catch-up charge for past-due users → on success, un-pause. Decoupled via event + a single listener (no module cycle). — src/passport/services/user-passport.service.ts, src/payment/orchestration/payment-reactivation.listener.ts

5. One-time / catch-up charges could double-charge

• Issue: createOneTimeChargeForUser was called with allowDuplicate: 'true', and retries carried no idempotency token.
• Problem: A duplicated retry job or a re-delivered FAILED webhook could charge the customer twice.
• Fix: Optional externalId on the charge (Passport dedups; allowDuplicate off when present). Retries pass a per-attempt stable key; duplicate FAILED webhooks are deduped by transactionId. — src/passport/services/user-passport.service.ts, src/payment/orchestration/payment-retry.service.ts

---

Backend — external API resilience

6. No HTTP timeout on any Passport/CPX call

• Issue: Every outbound Passport (ACH) and CPX (virtual card) call had no timeout (axios default = infinite).
• Problem: One hung upstream request blocks forever; in the serial per-user top-up cron, a single hang stalls every remaining user.
• Fix: Hard timeout (PAYMENT_HTTP_TIMEOUT_MS, default 30s) at the module level and in the Passport request funnel. — src/passport/passport.module.ts, src/virtual-card/virtual-card.module.ts, src/passport/passport.service.ts, src/common/configuration.ts

7. No retry on transient API failures

• Issue: Money-movement calls were fire-once; a transient 5xx / 429 / network blip failed the charge outright.
• Problem: Recoverable failures became hard failures (and, mid-creation, triggered rollback).
• Fix: The Passport request funnel retries on 429/5xx (always) and on network errors for GET only — never blindly re-POSTing a charge that may have already applied. Exponential backoff. — src/passport/passport.service.ts

---

Backend — virtual cards

8. VC top-up failures were only caught 2 days late

• Issue: The only detector was a daily reconcile that flags unfunded top-ups at T+2. For customers billed on the 1st–3rd, the alert fired after the card already declined.
• Problem: No ahead-of-time signal; also missed under-funding and initial-funding failures.
• Fix: A daily pre-charge sweep (D−2 lead): for cards due within the window, flag not-funded (red) / under-funded (amber) and auto-top-up the at-risk cards (idempotency-guarded), before the charge. — src/subscriptions/orchestration/virtual-card-orchestration.service.ts, src/jobs/jobs/vc-precharge-alert.job.ts

9. Missed CPX webhooks left permanent gaps

• Issue: VC charges/refunds were recorded only from CPX webhooks; a missed webhook meant a silently wrong per-card balance/ledger.
• Problem: The webhook-only pipeline can't detect its own gaps.
• Fix: A daily balance-drift reconcile — fetch each active card's live CPX balance; if it diverges from the cached value with no matching ledger activity, alert (likely missed webhook) and refresh the cache. — src/subscriptions/orchestration/virtual-card-orchestration.service.ts, src/jobs/jobs/virtual-card-reconcile.job.ts

10. A pre-funded card could be charged for an uncollected month

• Issue: If a card was topped up before the customer's collection failed, a provider could still charge it — the gate can't un-fund a loaded card.
• Problem: Bundul fronts that cycle's spend for a non-payer (recovered later via dunning + True-Up, but still exposure).
• Fix: Opt-in strict "collect-first" policy — VC_TOPUP_SKIP_PAST_DUE=true also skips any past-due user's top-up (default off keeps "first miss is fronted"). The gate always skips fully-paused users. — src/subscriptions/orchestration/virtual-card-orchestration.service.ts

---

Backend — reconciliation & stuck states

11. Records could get stuck forever

• Issue: Payment records stuck PENDING (no terminal webhook), orphaned PROCESSING retry jobs, and True-up records stuck approved-but-never-executed had no sweeper.
• Problem: They sit silently — money state never resolves and nobody's told.
• Fix: A 4-hourly stuck-state sweep: alert on stale PENDING records (>3d), reset orphaned (idempotent) retry jobs, and alert on approved true-ups (>1h). — src/payment/orchestration/payment-retry.service.ts, src/trueup/services/trueup.service.ts

12. The hourly ledger sync could wedge or silently drop entries

• Issue: An in-memory "in progress" flag could wedge all future syncs; per-user errors and page-cap truncation were swallowed.
• Problem: Reconciliation could quietly stop or miss entries with no signal.
• Fix: Watchdog forces a fresh run if the flag wedges >2h; ops alert on error spikes and page-cap truncation. — src/ledger/services/ledger-sync.service.ts

---

Backend — fee changes

13. A fee change couldn't reach existing customers, and the two fee carriers couldn't move together

• Issue: Changing the global Bundul fee only affected new signups. The fee is carried in two places (embedded in the main charge + a separate BOOK sweep) and the BOOK-sweep amount had no update method at all.
• Problem: No way to re-price existing customers; any attempt would move one carrier and not the other → corrupts True-Up.
• Fix: Admin toggle applyTo: 'new_only' | 'all_existing'. Built the missing BOOK-sweep updater; changeFeeForUser updates both carriers + Mongo atomically (idempotent, audited); an event-driven, resumable migration re-prices all existing customers with an ops summary. — src/pricing/pricing.service.ts, src/payment/records/user-payment.service.ts, src/passport/services/user-passport.service.ts, src/payment/records/fee-change.listener.ts

---

Notifications & customer emails

14. New ops alerts pointed at a Slack channel that doesn't exist

• Issue: The new resilience alerts were written to a #payments Slack channel.
• Problem: #payments doesn't exist; alerts would go nowhere.
• Fix: All ops alerts route through NotificationsService.sendAdminAlert → email to OPS_ALERT_EMAILS and recorded to NotificationLog so they surface on the admin dashboard. 9 new automations registered in the catalog. — src/notifications/*, and the alerting call sites in the services above.

15. Dunning customer emails had no templates

• Issue: The dunning flow referenced Infobip templates that didn't exist.
• Problem: Customer-facing failure/paused/recovered emails wouldn't send.
• Fix: Created 4 Infobip templates in the existing house style (cloned from a live template via POST /email/1/templates; {$placeholder} syntax): payment failed (soft/hard), services paused, payment recovered. Wired into EmailTemplates, sent via sendEmailWithTemplate. — src/infobip/infobip.service.ts, scripts/create-dunning-templates.cjs

---

Admin dashboard (bundul-admin)

16. No always-on view of spent vs collected per customer

• Issue: Collected-vs-spent only existed inside the quarterly True-Up.
• Problem: Ops couldn't see, at a glance, which customers Bundul is exposed on right now.
• Fix: Backend GET /admin/ledger/financials[/:userId] (collected / fees / spent / delta + dunning status, sorted by exposure). Admin Financials page + per-customer detail with the month breakdown and dunning card. Live screenshot-verified. — src/ledger/controllers/ledger-admin.controller.ts; bundul-admin FinancialsPage.tsx / FinancialsDetailPage.tsx.

---

Deferred (dedicated, tested PRs)

17. Full Mongo multi-document transactions

• Issue: Several flows do multiple DB writes (webhook handlers, the Addition Passport→Mongo sequence) with no atomicity, so a mid-way crash leaves partial state.
• Why deferred: Needs replica-set verification + session threading across many money-write sites; risky to batch. Contained mitigations shipped instead (real-time drift alert on the Addition seam; the ledger sync + stuck-state sweepers catch most partial states after the fact).

18. Distributed cron lock (multi-instance safety)

• Issue: @nestjs/schedule fires on every ECS task; with >1 task (or during a rolling deploy's task overlap) money crons can double-fire. No distributed lock.
• Why deferred: Per decision to skip the ECS item for now. Fix would be a Mongo advisory-lock doc (unique key + TTL) around each money cron + a unique index to make the top-up guard atomic. desiredCount is a service-level setting (not in the repo).

19. Webhook authentication

• Issue: /webhook/passport and /webhook (Plaid) are unauthenticated and forgeable (CPX card-event is guarded).
• Why deferred: Parked per decision; revisit when security work is scheduled.

---

Config to set for production

Env var	Purpose	Default
OPS_ALERT_EMAILS	Recipients for all ops alerts	olamide@ , farhan.s@
PAYMENT_UPDATE_URL	"Update payment" link in dunning emails	app.bundul.io/account/payment
PAYMENT_HTTP_TIMEOUT_MS	Passport/CPX call timeout	30000
PAYMENT_HTTP_MAX_ATTEMPTS	Passport retry attempts	3
VC_TOPUP_SKIP_PAST_DUE	Strict collect-first (skip past-due top-ups)	off
VC_PRECHARGE_WINDOW_DAYS	Pre-charge alert lead time	2