Frozen. Research and findings as they stood on 6 July 2026, not living documentation. Twenty-five commits have landed on the payment code since, so treat specifics as historical. It is kept because the reasoning is still useful. For how payments work today see docs/explanation/financial-system.md and docs/generated/.

Payment Resilience — Research & Findings

Status: Research only. No code changed. This document maps how the payment system behaves under failure today, answers the specific questions raised, and proposes a prioritized hardening roadmap.

Companion doc: financial-system.md describes the intended architecture. This doc audits the actual implementation against it and focuses on what happens when things go wrong.

Method: Five parallel code audits (retry/failure, fee-change propagation, external-API resilience, webhook/reconciliation, cron/operational) plus a focused VC top-up failure/alerting deep-dive. All claims below are cited to file:line in the current main.

Decisions locked (from review)

• Delinquency: retry + notify, then gate disbursement after N failures. See §2.
• Fee changes: admin chooses per-change whether it applies to all existing customers or new customers only, going forward. See §3.3.
• VC top-ups: proactive ahead-of-time admin alerting is a priority — confirmed D−2 lead time and persist nextChargeDate. Design in §8.
• Webhook auth: parked for now (still logged as a finding in §6.1, not on the near-term roadmap).
• ECS desiredCount: not in the repo (service-level); cron-lock is needed regardless because rolling deploys transiently run two tasks. See §7.1.

---

0. TL;DR — The Ten Things That Matter

Ranked by risk to money correctness / customer trust.

#	Finding	Severity	Section
1	A failed monthly collection is silently dropped — no retry, no notification, no suspension. Bundul keeps funding VCs / paying utilities with money it never collected.	🔴 Critical	§2
2	/webhook/passport and /webhook (Plaid) are unauthenticated — anyone who can reach them can POST a forged COMPLETED/FAILED and move the system's state. (Parked per review — kept here for the record.)	🔴 Critical	§6.1
2b	VC top-up failures are only detected T+2 days (lag, not lead) via Slack — for customers billed on the 1st–3rd the alert fires after the card already declined. No ahead-of-time alerting.	🟠 High	§8
3	Multi-instance crons have no distributed lock — if ECS runs >1 task (or during rolling deploy overlap), the monthly top-up and every money cron fire on each instance.	🔴 Critical	§7.1
4	No HTTP timeout on any Passport/CPX call — one hung upstream request hangs the whole monthly top-up cron; every user after the stall goes unprocessed.	🟠 High	§4.1
5	A fee change cannot be pushed to existing customers, and the fee is carried in two places (embedded charge + BOOK sweep) that have no lockstep update path.	🟠 High	§3
6	One-time charge retries are not idempotent (allowDuplicate: 'true') — a duplicate FAILED webhook or a settle/fail race double-charges the customer.	🟠 High	§5.3
7	Top-up funding is not sequenced before the provider charge — users with early billing days can be charged before their card is funded (async, next-business-day).	🟠 High	§7.3
8	No retry/backoff on money-movement API calls — a single transient 5xx/timeout aborts a charge (then a best-effort, log-only rollback).	🟠 High	§4.2
9	Reconciliation blind spots — no CPX-side poll (missed card webhooks = permanent per-card gap); ledger sync can permanently miss back-dated / >1000-entry windows; per-user sync errors are swallowed with no alert.	🟡 Medium	§6.3
10	Stuck states have no sweepers — PENDING payment records, approved-not-executed true-ups, stale pending ledger entries, never-funded initial PIFs all can sit forever.	🟡 Medium	§6.4

The single most important gap is #1 combined with #7/#8: the collection side and the disbursement side are decoupled. Nothing gates paying the user's providers on whether Bundul actually collected from the user. A customer can stop paying and keep receiving fully-funded services indefinitely, with no alert to ops.

---

1. Mental Model — Two Rails, Two Sides, Loosely Coupled

        COLLECT side (money IN)                 DISBURSE side (money OUT)
        ───────────────────────                 ─────────────────────────
  Passport ACH COLLECT from user's bank    VC path:  CPX PIF funds lodge card → provider charges card
   • BUNDUL-SUB-{userId}  (services)        ACH path: provider debits user's Passport-linked bank
   • BUNDUL-FEE-{userId}  (fee sweep, BOOK)
   • BUNDUL-ONE-TIME-{userId} (catch-up)
   • BUNDUL-TRUEUP-{id}   (quarterly)

              └──────────────── NOT gated on each other ───────────────┘
                    (disbursement runs on its own monthly cron,
                     regardless of whether collection settled)

Everything reconciles after the fact through the hourly LedgerSyncService → CustomerCashLedger (the financial source of truth) and the quarterly True-Up. There is no real-time coupling between "did we get paid" and "should we pay out."

---

2. THE KEY SCENARIO: Passport fails on month 3

"What happens when a customer's Passport charge fails on the 3rd month? Can we retry?"

Traced answer: nothing happens, and no — it is not retried.

The month-3 charge is the recurring subscription charge (passportRecurringChargeId, externalId BUNDUL-SUB-{userId}). When Passport sends transaction.ach.update status=FAILED:

1. handleRecurringTransactionExecution hits the isFailed branch at webhook.service.ts:579-582 — which only logs and returns. (Verified: the COMPLETED branch right below it sends a customer confirmation email; the FAILED branch sends nothing.)
2. handlePassportPaymentRecordUpdate runs but finds no PassportPaymentRecord — recurring charges are never recorded there (records are only created for one-time charges, payment-recurring-subscription.service.ts:308). So no retry is enqueued.

Concretely, for that failed month-3 collection:

Question	Answer	Evidence
Does it retry?	No. Only ADDITION_ONE_TIME_CHARGE has retry wiring. Recurring charges have none.	payment-retry.service.ts:20, webhook.service.ts:496-504
Does it notify anyone?	No customer email, no ops alert, no push. (CPX card declines DO alert ops — different rail.)	webhook.service.ts:579-582 vs card path webhook.service.ts:1033
Does it suspend the subscription?	No. There is no past_due / delinquent / dunning state anywhere in the schema.	(no field exists)
Does it keep funding VCs / paying utilities?	Yes. Top-up and utility pulls run on their own schedules keyed off the active subscription, independent of collection success.	virtual-card-topup.job.ts:14
Is there any backstop?	The daily onesub-reconcile job checks amount drift only, not payment success — it will not flag an uncollected month.	onesub-reconcile.job.ts

Design (DECISION LOCKED): retry + notify, then gate

A delinquency state machine on UserPassportSubscription, driven by collection webhooks:

COLLECTION FAILED  (BUNDUL-SUB or BUNDUL-FEE, status=FAILED)
   → record the failure: paymentStatus='past_due', failureReason, failedAt, attemptCount++
   → classify:
        transient (5xx / timeout / rate-limit)  → auto-retry on a short cadence
        hard      (NSF / closed / invalid acct) → notify user to fix funding, slower cadence
   → schedule progressive retries with backoff, e.g. +1d, +3d, +7d (config-driven)
   → notify the CUSTOMER on each failure ("update your payment method / add funds")
   → notify OPS (Slack #payments + daily email digest of past-due accounts)

   ── GATE (after N failed attempts, N configurable, default e.g. 3) ──
   → set disbursementPaused=true on the user
   → VC top-up cron SKIPS paused users; ACH utility setup SKIPS paused users
   → optionally set subscription status → 'suspended'

   ── RECOVERY ──
   → on a successful catch-up charge: clear past_due + disbursementPaused,
     resume disbursement, notify customer ("you're back on track")

Reused building blocks: the ADDITION_ONE_TIME_CHARGE_RETRY job pattern (generalize it to RECURRING_CHARGE_RETRY), the BackgroundJob schema, NotificationsService (customer email + sendAdminAlert), SlackService.sendNotification.

New pieces to build:

1. Record recurring failures. Today webhook.service.ts:579-582 just logs — replace with a call into the delinquency service.
2. paymentStatus / attemptCount / disbursementPaused fields on the subscription/user.
3. A retry policy (transient vs hard classification + backoff schedule).
4. The gate: the VC top-up loop (virtual-card-orchestration.service.ts:693-708) and ACH setup must check disbursementPaused and skip. This is the piece that actually stops Bundul paying out uncollected money.
5. Customer + ops notifications on each failure and on gate/recovery.

Grace-period knobs (product-configurable): number of retry attempts before gating, the backoff schedule, and whether to suspend vs merely pause disbursement while past-due. Defaults above are a starting proposal, not a final policy.

---

3. What happens when the Bundul fee changes

"If Bundul fee changes, how does it affect others?"

3.1 How the fee is modeled

• The Bundul fee is a single global flat value in one Mongo Pricing doc (admin-managed), Airtable as legacy fallback — pricing.schema.ts:10-11, pricing.service.ts:22-36.
• It is snapshotted per One Sub at signup onto UserPassportSubscription.bundulFee and grandfathered forever — user-passport-sub.schema.ts:130-131. (Comment confirms some customers are on a legacy $5.99 while global is $8.99 — user-payment.service.ts:149-154.)
• The fee is carried in two places at once, not either/or:
  1. Embedded in the main recurring ACH charge (fullRecurringAmount = services + fee) — payment-recurring-subscription.service.ts:513-518.
  2. A separate BOOK transfer (passportRecurringFeesChargeId) that sweeps the already-collected fee from the user's Passport wallet to Bundul — this is not a second ACH pull from the customer — user-passport.service.ts:1119-1125.

3.2 The answer: a global fee change reaches nobody who already exists

PricingService.update() mutates one number and nothing else — pricing.service.ts:38-54. It does not iterate subscriptions, does not update Passport charges, does not touch the BOOK sweep. The one cron that ever adjusted charges post-hoc is a disabled no-op — user-payment.service.ts:577-578.

Consequences of changing the global fee (say $8.99 → $9.99):

• Existing customers: unchanged. They keep their snapshotted bundulFee; the main charge and BOOK sweep keep using the old value. Internally consistent (reconcile stays green) but the fee change simply never reaches them.
• Additions (existing customer adds a service): even a new add reuses the snapshot — existingSub.bundulFee ?? bundulFee — payment-recurring-subscription.service.ts:400.
• New signups: get the new fee (Initial reads the live global).

3.3 Design (DECISION LOCKED): admin toggle — all customers vs new-forward

The requirement: when an admin changes the Bundul fee, they choose at change-time whether it applies to (A) all existing customers or (B) new customers only, going forward.

The hard part first — the two-carrier problem. The fee lives in two carriers that cannot be updated in lockstep today: updateRecurringChargeAmount can move the embedded total (only called from the Addition path — user-passport.service.ts:645-716), but there is no update method for the BOOK-transfer amount at all — only create/cancel. Change one without the other and the customer is charged one fee while a different amount is swept to Bundul, which corrupts True-Up (it reads the actual swept bundul_fee ledger entries, not a formula — trueup.service.ts:184-191). So mode (A) is impossible to do correctly until we can update both carriers together.

Proposed shape:

PUT /admin/pricing  { bundulFee, applyTo: 'new_only' | 'all_existing' }

applyTo = 'new_only'   (current behavior, made explicit)
   → update the global Pricing doc ONLY
   → new signups pick it up; existing customers keep their snapshot
   → no per-customer work; safe today

applyTo = 'all_existing'   (new capability)
   → update the global Pricing doc
   → enqueue a background migration: for each active UserPassportSubscription,
     run a single atomic changeFeeForUser(userId, newFee):
        1. recompute fullRecurringAmount = servicesTotal + newFee (+ split)
        2. update the Passport MAIN recurring charge to the new full amount
        3. update the BOOK sweep (passportRecurringFeesChargeId) to the new fee   ← must build
        4. persist bundulFee + fullRecurringAmount + a history[] entry in Mongo
        5. reconcile-check: live Passport amount == stored fullRecurringAmount
     → per-user try/catch; failures collected and reported (Slack/email digest)
     → idempotent + resumable (skip users already at newFee)

New pieces to build:

1. applyTo field on the admin pricing endpoint/UI (admin-pricing.controller.ts:45-68, pricing.service.ts:38-54).
2. An update method for the BOOK sweep amount — the missing lockstep half. Without this, mode (A) cannot be correct.
3. changeFeeForUser() — the single atomic path across Mongo + main charge + BOOK sweep, with a post-write reconcile check and a history[] audit entry (fee_changed).
4. A background migration runner (idempotent, resumable, per-user isolated, with a completion report) for the all_existing path.

Cleanup: delete the dead needsBundulFeeUpdate / bundulFeeChargePending flags (user-passport-sub.schema.ts:91-108) so nobody assumes a fee-bump mechanism exists when it doesn't.

Note on grandfathering: the all_existing path erases grandfathered legacy fees (e.g. $5.99) for whoever it touches — intended, but the admin UI should say so, and the history[] entry preserves the prior fee for audit.

3.4 Related: subscription amount / service price changes

• Addition recomputes the full amount from all line items via the shared recomputeOneSubAmounts helper (never amount += x — the historical duplicate-fee bug) — one-sub-amounts.util.ts. Good.
• But Addition is not atomic: Passport updateRecurringChargeAmount (:425) then Mongo save() (:455) with no transaction and no rollback. A failure between them → Passport charges the new total while the stored breakdown shows the old. Only the daily reconcile detects it, and it alerts only, never self-heals.
• Service price changes never propagate at all: line-item prices are snapshotted at bundling (user-passport-sub.schema.ts:16). If a service's price later changes, the One Sub keeps collecting the old total forever.
• VC 15% buffer (virtual-card-orchestration.service.ts:218-220) is applied to the service price only, independent of the fee. But VC funding uses a live-resolved price while collection uses the frozen snapshot — divergent price sources can surface in True-Up as fake customer over/under-payment.

---

4. External API Resilience (Passport & CPX)

4.1 No timeouts — hang-forever risk

No timeout is configured on any Passport or CPX call. Both HttpModules are imported bare (passport.module.ts:46, virtual-card.module.ts:17) — no register({ timeout }), no per-request timeout, no AbortController. Axios default is timeout: 0 (infinite). (Note: non-payment services like Brandfetch do set timeouts, so this is an inconsistent omission, not a house style.)

The worst consequence is head-of-line blocking in the monthly top-up cron: topUpVirtualCardForAllUsers iterates users serially over a cursor (virtual-card-orchestration.service.ts:693-714), each making un-timed CPX calls. One hung CPX request stalls the entire month's top-ups; every subsequent user goes unfunded. The per-user try/catch does not help — it only catches returned errors, not a hang.

4.2 No retry on money-movement

The core money calls — Passport recurring/one-time charges (passport.service.ts:23-34) and the CPX PIF (virtual-card.service.ts:256-277) — are fire-once. A single transient 5xx / 429 / ECONNRESET fails the charge. The only transient-aware retry in the whole payment path is _fetchBuyerBankAccounts (bank-account GET), which correctly gates on !status || status >= 500 || status === 429 (virtual-card.service.ts:646).

Other resilience gaps at the API layer:

• No 429 / Retry-After handling anywhere except that one GET.
• No token caching — getBearerToken re-authenticates on every CPX operation (virtual-card.service.ts:42), tripling call volume during the cron burst (3 × N users × M cards) — prime 429 territory, with no 429 handling to absorb it.
• No circuit breaker (no opossum/cockatiel in deps).
• _fetchCardDetailsWithRetry (5× backoff, :963-1008) is an eventual-consistency poll for async card provisioning, not a fault-retry — it retries on any error and returns null (non-fatal) after exhausting attempts.
• Errors mask their status: Passport errors are re-wrapped to new Error(message), losing response.status, so downstream 409/duplicate detection degrades to string matching. CPX errors are thrown as ForbiddenException (misleading 403 for real 5xx/timeouts).

4.3 Partial-failure / half-written state

• VC creation deliberately writes the VirtualCard row before submitting the PIF (:105), so a PIF failure leaves a card row with no cpxCardIdForDetails and no funding. It also stamps vcStableId on the subscription before card details are confirmed (:141) — a subscription can be marked VC-ready pointing at an unfunded card.
• Initial PIF is not idempotency-guarded at submit time: a crash between a successful PIF and _storePifResult causes a re-run to mint a duplicate CPX lodge card ("More than 1 active card" — the documented VC-topup bug). The top-up path IS guarded (hasTopUpSince — :351); initial creation is not.

---

5. Retry, Idempotency & Background Jobs

5.1 What actually retries

Only ADDITION_ONE_TIME_CHARGE retries: max 3 attempts (payment-retry.service.ts:20), driven by a 10-minute cron (payment-retry.job.ts:12), no backoff — pacing depends entirely on how fast Passport re-emits a FAILED webhook for the new transaction. Recurring subscription charges, the Bundul fee, and true-ups have no retry.

5.2 Transient vs permanent

No distinction is made for Passport charges. The retry branches purely on record.type and retryCount — it never inspects the failure reason. NSF (permanent) is retried identically to a network blip (transient), both burning attempts up to 3.

5.3 Idempotency gaps

• Only createRecurringSubscription is idempotency-key guarded (payment-recurring-subscription.service.ts:60). It fails open on key-creation error (:114-122) and does not short-circuit a prior failed attempt — a retry re-executes the whole body, which after a partially-failed rollback can create a duplicate recurring schedule (double monthly billing).
• The one-time charge is explicitly non-idempotent: createOneTimeTransaction is called with allowDuplicate: 'true' (user-passport.service.ts:839), and the retry re-issues it with no idempotency token. A duplicate FAILED webhook or a settle/fail race → double charge.
• CPX card events, by contrast, are properly idempotent (unique eventId index + 11000 catch).

5.4 Background jobs are not a real queue

BackgroundJob is mostly an audit/error log. The main payment job runs inline in a fire-and-forget void (async () => {…})() inside the request handler (payment-orchestration.service.ts:321-419). Only ADDITION_ONE_TIME_CHARGE_RETRY is ever polled/dispatched. There is no dead-letter, no stuck-job detection, no reclaim — a job orphaned in PROCESSING (crash mid-execution) is never re-picked; it just sits until the 7-day TTL. A crash after the Passport charge succeeds but before the DB write leaves a live charge at Passport with no local record, invisible to retry and reconcile.

---

6. Webhooks & Reconciliation

6.1 Webhook authentication

Endpoint	Guard	Risk
POST /webhook/passport	none	🔴 Anyone reachable can POST a forged COMPLETED/FAILED → trigger true-up settlement, payment-record flips, retry enqueues, customer emails
POST /webhook (Plaid)	none	🔴 Forgeable Plaid events
POST /webhook/card-event (CPX)	BasicAuthGuard	OK (static cred, plain === compare — minor timing concern)

Verified at webhook.controller.ts:26-33. SNS signature verification exists only as dead commented-out code and never actually validated the signature. This is the highest-severity security finding. Fix: verify Passport's webhook signature (or SNS SigningCertURL/Signature), and put an auth guard on both open endpoints.

6.2 Idempotency & ordering of webhooks

• CPX: deduped by unique eventId index — good. But writeLedgerEntry dedup is a soft read-then-write with no unique DB index on the VC ledger, so a true concurrent double-delivery could double-write.
• Passport: no dedup at all. A re-delivered event re-runs every handler and can re-send the "monthly charge confirmation" customer email.
• Out-of-order CPX: if Settled arrives before VCN Generated (before cpxCardTid is stored), _resolveVirtualCard returns null and the charge is silently dropped from the VC ledger (money side can still be recovered by the hourly sync; per-card attribution is lost).
• CPX partial-processing trap: the event is persisted before ledger routing. A mid-routing 500 → CPX retries → _isCardEventDuplicate short-circuits → ledger row never written, event permanently marked processed.
• Passport handlers always return 200 even on internal error (each swallows its own exceptions) → Passport will not retry → that event is lost (money side relies on the hourly sync; payment-record/true-up side is not recovered).
• No DB transactions anywhere — Airtable + multiple Mongo writes run sequentially with no atomicity; partial processing is possible on both rails.

6.3 Reconciliation completeness

The hourly LedgerSyncService is the main safety net (upserts Passport ledger into CustomerCashLedger, idempotent on passportLedgerEntryId). Blind spots:

• CPX charges are NOT independently reconciled — no job lists CPX transactions to backfill missed charge/refund/funding webhooks. The VC ledger is 100% webhook-dependent. (virtual-card-reconcile only flags unfunded top-ups.)
• Permanent-miss windows: entries back-dated >2 days before the newest synced entry fall outside the cursor and are never revisited; >90-day first-run history is ignored;

  1000 entries in one window drop pages 21+.

• Per-user sync errors are swallowed with only a counter — no alert, no retry, no dead-letter. A user whose Passport account 500s every hour is silently never reconciled.
• In-memory syncInProgress flag can wedge all future syncs if a run hangs (no watchdog), and offers no cross-instance locking.

6.4 Stuck states

State	Sweeper?
TrueUpRecord executing	✅ checkStuckExecutingRecords every 4h
TrueUpRecord approved-not-executed	❌ none (crash between approve and execute → stuck forever)
CustomerCashLedger stale pending	❌ none
PassportPaymentRecord stale PENDING (terminal webhook never fired)	❌ none
Initial pif_submitted never funded	❌ none (top-up reconcile only covers top-ups)
VirtualCard status never confirmed	❌ none (no CPX status poll)
BackgroundJob orphaned in PROCESSING	❌ none (TTL only)

---

7. Cron / Operational Resilience

7.1 Multi-instance double-execution

@nestjs/schedule schedules jobs inside every Node process with no leader election. There is no distributed lock anywhere — the codebase explicitly notes this in two comments (keyed-mutex.ts, ledger-sync.service.ts) but never implemented it. So if the ECS service runs N tasks, every cron fires N times at the tick — including the monthly VC top-up and the quarterly true-up (which move money).

desiredCount is not in the repo — confirmed by checking both task defs and the deploy workflow. It is an ECS service-level setting (bundul-api-prod), and the deploy action (prod.yml:53-59) deploys to that service without setting a count, so it just preserves whatever the service holds. Read the live value with aws ecs describe-services --cluster bundul-api-prod --services bundul-api-prod --query 'services[0].desiredCount'.

The critical point: this is a real risk even if desiredCount == 1. The deploy uses ECS defaults (maximumPercent 200%), so every rolling deploy transiently runs two tasks — the new one starts before the old drains. If a money cron (VC top-up, quarterly true-up) happens to tick during that overlap window, it double-fires regardless of steady-state count. And the moment the service scales past 1 task, every cron double-fires every tick. The hasTopUpSince guard is check-then-act with no atomic constraint, so it does not fully protect against a simultaneous double-fire.

Fix: a Mongo advisory-lock doc (unique key + TTL) around each cron entry, or a dedicated single-task scheduler service. Add a unique index on the top-up ledger key (virtualCardId, billingPeriod) to make hasTopUpSince atomic.

7.2 Failure isolation & resumability

• Isolation is good: every per-user loop wraps the user in try/catch and continues.
• No resumability: batch crons restart from user 0 on crash (fresh cursor, no checkpoint). VC top-up and ledger sync survive via per-item idempotency; true-up quarterly's per-quarter idempotency is unverified and it moves money — a mid-run crash + re-run is a potential double-true-up.

7.3 Timing: top-up vs charge date

No ordering guarantee. The top-up fires once, on the 1st at 00:00 UTC (virtual-card-topup.job.ts:14), but each user's recurring charge dayOfMonth is derived from their signup date (any day 1–31). CPX funding settles asynchronously (next business day). So a user with an early billing day can have the provider charge the card before it is funded → decline. The 15% buffer is a cushion, not a guarantee. The only backstop is the daily reconcile, which detects an unfunded top-up after the fact.

7.4 Observability gaps

• 6 of 14 crons lack @SentryCron (including bundle-run reconcile and the true-up stuck-record check).
• Partial completion is invisible — failCount/errors are logged only; Sentry cron fires only on total failure/missed check-in, not on "23 of 400 users failed." A user silently skipped every month would go unnoticed.
• OneSub reconcile detecting drift only does logger.error — no direct Slack/email despite being "the backstop."

---

8. VC Top-Up — Proactive Failure Detection & Alerting (PRIORITY)

"We need to know when VC top-ups fail. Alerting admin ahead of time is super important."

8.1 What exists today (and why it's too late)

The only VC top-up failure detector is virtual-card-reconcile.job.ts (:23-49), daily at 6 AM. It calls reconcileUnfundedTopUps(slaDays=2) — finds top_requested ledger entries with no matching top_up/funding confirmation after 2 days, writes an unfunded marker, and posts to Slack #virtual-cards (PROD only, no email):

":warning: N virtual-card top-up(s) never funded (CPX did not confirm within SLA).
 These cards were NOT reloaded and may decline: …"

Why it's a lag, not a lead: the 2-day SLA matches Good Funds' next-business-day settlement, so the earliest it can flag anything is T+2. But the top-up cron fires on the 1st, and each user's provider charge day is derived from their signup date (any day 1–31, user-passport.service.ts:502-531). So for anyone billed on the 1st–3rd, the "may decline" alert arrives after the card has already declined. There is no ahead-of-time signal.

Additional blind spots in today's detector:

• Only inspects monthly top_requested — a failed initial pif_submitted is never reconciled.
• Does not detect under-funding (any top_up clears the flag regardless of amount vs. the real bill).
• Does not correlate late funding with a decline.
• False positives when CPX funded but the webhook was unroutable (_resolveVirtualCard returns null → no top_up row → looks "unfunded" though it's funded — webhook.service.ts:877).
• PIF-returns-no-record (:380-382) writes no ledger entry and no alert — silent.
• Good Funds (bank debit) failure the next business day has no representation anywhere in the VC domain.

8.2 The core missing data

Ahead-of-time alerting needs to answer, per card: "must be funded by day D, is it funded, is the balance enough?" Today none of that is queryable:

Needed	Status today
Per-card charge day ("due by D")	Not stored. Only in Passport's recurring config; derivable from signup startDate. No field on UserSubscription or VirtualCard.
Funding confirmed this period?	Inferable via hasTopUpSince(vc._id, monthStart) (virtual-card.service.ts:888-915) — presence of top_up/funding. Works.
Real-time balance	safeGetBalance(vc.cpxCardIdForDetails) exists (:805-816) but is never polled; lastKnownBalance is a stale cache between webhooks.
Expected bill amount to compare vs. balance	Not stored. Only the funded amount (price × 1.15) is known; the true bill arrives post-charge. Under-funding is undetectable pre-charge.
Explicit top-up status field	None — progress is inferred from ledger-row presence.

8.3 Proposed design — a pre-charge verification sweep

A new daily job that alerts before the charge, reusing existing primitives. Confirmed: lead time = D−2 (alert 2 days before the charge day), and we will persist nextChargeDate on the subscription so the "due within N days" query is a cheap indexed lookup rather than a per-user recompute.

For each active VirtualCard:
   1. Resolve the user's charge day D — persisted as nextChargeDate (indexed),
      backfilled from getDayOfMonthForRecurring(startDate).
   2. Only consider cards whose charge day is within the ALERT WINDOW
      (D within the next 2 days — the D−2 lead time we lacked).
   3. Funded?  hasTopUpSince(vc._id, billingPeriodStart())
        · NOT funded  → RED alert: "card for {service}/{user} charges on {D},
                        top-up not confirmed — will likely decline."
   4. If funded, verify AMOUNT: force a live safeGetBalance()
        · balance < expectedBill (or < lastFundingAmount)  → AMBER alert: "under-funded."
   5. Batch findings into ONE digest → Slack #virtual-cards + sendAdminAlert email.

This turns the T+2 lag into a D−2 lead. It also catches the cases the reconcile job misses: under-funding (step 4), and initial funding (it looks at all active cards, not just top_requested).

Complementary hardening (close the silent gaps):

• Capture the silent PIF-no-record failure (:380-382) as an error ledger entry and an immediate admin alert.
• Make the CPX error webhook alert (webhook.service.ts:1024-1040, already wired for declines) also fire on top-up/funding error statuses.
• Add a dead-letter / replay for _resolveVirtualCard returning null so unroutable funding webhooks aren't silently lost (fixes the false-positive unfunded).
• Optionally persist nextChargeDate on the subscription so "which cards are due in N days" is a cheap indexed query instead of a per-user recompute.

Reused delivery infra (no new plumbing): SlackService.sendNotification('#virtual-cards', …) (common/slack.service.ts:15-31), NotificationsService.sendAdminAlert({ key, subject, html, … }) (notifications.service.ts:252-293), renderAdminAlert() for consistent HTML, @SentryCron for cron monitoring.

8.4 Sequencing note

This detection pairs with roadmap item "sequence top-up before charge" (§7.3): the real fix is to fund each card ahead of its own charge day (not a blanket 1st-of-month run) and/or gate the provider charge behind confirmed funding. Proactive alerting is the safety net; per-user-timed top-up is the prevention. Both are worth doing.

---

9. Recommended Hardening Roadmap

Ordered by risk-reduction per unit of effort. Nothing here is implemented yet — this is the proposed plan for discussion.

Reflects the locked decisions: webhook auth parked; VC proactive alerting and the fee-change toggle promoted to Phase 1.

Phase 1 — Stop the bleeding

1. Delinquency + dunning for failed collections (§2). Record recurring failures; add paymentStatus: 'past_due'; progressive retry with backoff; notify customer + ops; gate disbursement for delinquent users after N failures. This closes the #1 money-loss hole. (Decision locked.)
2. VC pre-charge verification + ahead-of-time alerting (§8.3). Daily sweep: for cards due within the alert window, flag not-funded (RED) and under-funded (AMBER) to Slack + admin email before the charge. Plus close the silent PIF-no-record / unroutable-webhook gaps. (Priority per review.)
3. Distributed cron lock (§7.1) — Mongo advisory-lock doc (unique key + TTL) around every money cron; confirm current desiredCount. Add unique index to make hasTopUpSince atomic.
4. HTTP timeouts on all Passport/CPX calls (§4.1) — small change, removes the cron-hang risk immediately.

Phase 2 — Make failures self-heal & enable fee control

5. Fee-change toggle (all vs new-forward) (§3.3) — applyTo on the admin endpoint; build the BOOK-sweep update method (the missing lockstep half), changeFeeForUser(), and the idempotent migration runner. (Decision locked.)
6. Retry with backoff + transient/permanent classification on money-movement calls (§4.2, §5.2).
7. Idempotency keys on one-time charges (drop allowDuplicate: 'true'), and short-circuit failed/processing idempotency states correctly (§5.3).
8. Sequence top-up before charge (§7.3) — top up per-user ahead of their actual billing day, or gate the charge behind confirmed funding. (Prevention that complements the alerting in #2.)
9. CPX-side reconciliation poll (§6.3) — list CPX transactions daily and backfill missed card webhooks.

Phase 3 — Close the long-tail

10. Stuck-state sweepers (§6.4) — approved-not-executed true-ups, stale PENDING payment records, never-funded initial PIFs, orphaned PROCESSING jobs.
11. Reconciliation robustness — alert on per-user sync errors and partial cron completion; watchdog for the wedged syncInProgress flag; widen/guard the ledger sync window and page cap.
12. DB atomicity / rollback on multi-step flows (Addition, webhook handlers) and a Passport→DB reconciliation job for the fire-and-forget crash window (§5.4).
13. Webhook authentication (§6.1) — parked per review; revisit when security work is scheduled.

---

10. Open Questions for Product / Ops

• Delinquency thresholds: policy locked (retry + notify, then gate) — remaining knobs: how many attempts / how many days before gating, and suspend vs. merely pause disbursement while past-due?
• VC alert lead time / nextChargeDate: Resolved — D−2 lead, and persist nextChargeDate (indexed). See §8.3.
• Fee migration blast radius: for an all_existing fee change, run it as one big migration or roll it out in batches? Any customers explicitly exempt from re-pricing?
• ECS desiredCount: not in the repo — it's a service-level setting; read it with aws ecs describe-services --cluster bundul-api-prod --services bundul-api-prod. Note the cron-lock (§7.1) is needed regardless — rolling deploys already run two tasks transiently. The count only decides whether it's also an every-tick bug in steady state.
• SCG $1.65 fee & VC buffer leakage: currently absorbed by Bundul. Acceptable, or should it flow into True-Up?