Email invoice parsing

How Bundul reads a customer's utility bill out of their inbox: the pipeline, the GraphQL queries and mutations, the responses, and the states the app polls.

---

1. Why it exists

Utilities (ComEd, PG&E, Verizon…) give us no API for "what does this customer owe this month". So we read the customer's mailbox instead, find the bill email, and have Claude read the amount and the due date out of it.

That number is what gates and drives the utility payment. It also feeds bill history and utility insights (the "your bill is higher than normal" screen).

---

2. The three levels

The pipeline is split into three steps on purpose, because they cost wildly different amounts. The split lives in src/email-parser/interfaces/email-provider.interface.ts.

Level	What it does	Cost
L1	Pull candidate emails from the mailbox: from:<senderDomain> after:<24 months ago>, max 30 messages, bodies fetched concurrently	cheap API calls
L2	Batch-classify subject + snippet: "is this a bill, or a receipt / marketing?" — 8 emails per Claude call	one cheap AI call per 8 emails
L3	Read each bill in full: one Claude call per email, run sequentially	expensive

Which method runs which levels:

• findInvoiceCandidates() → L1 + L2 only.
• extractInvoiceDetails() → L3 only, over whatever candidates you hand it.
• listMostRecentEmailsForService() → both, back to back.

Providers

Three implementations of that interface, chosen by emailAccess.provider in src/email-parser/email-provider.factory.ts (defaults to gmail):

• src/email-parser/providers/gmail.provider.ts — Gmail API, q: from:<domain> after:<date>.
• src/email-parser/providers/microsoft.provider.ts — Graph. Tries KQL $search="from:domain" first and falls back to an OData contains(...) filter, because Exchange Online rejects one style or the other depending on the mailbox. $search can't combine with $orderby/$filter, so date filtering and sorting happen client-side.
• src/email-parser/providers/imap.provider.ts — IMAP search({ from, since }), capped at 200 UIDs.

Which utility to search for — the fingerprint

src/email-parser/parser.constants.ts resolves a UtilityFingerprint:

{
  name: string;              // display name, for logs
  senderDomain: string;      // the ONLY field that actually filters the mailbox query
  subjectMarker?: string;    // logging only
  senderName?: string;       // logging only
  additionalFilter?: string; // logging only
  merchantMarkers: string[]; // Plaid merchant substrings, for the already-paid check
}

It prefers the admin-editable invoiceDetection on the catalog item (BundulSupportedSubscription), walking userBundulSupportedSubId → slot → bundulSupportedSubscription. It falls back to the legacy hardcoded per-utility maps keyed by name. If neither yields a sender domain there is nothing to search on, and the run short-circuits to invoiceStatus = 'none'.

Gotcha. subjectMarker, senderName and additionalFilter are not used to filter anything. Only senderDomain goes into the mailbox query; L2 classification does the real filtering. Those three fields only appear inside log and error strings, which makes the "Search markers – sender=…, filter=…, subjectMarker=…" diagnostic misleading when a pull finds nothing — it reads as though we searched on them.

Which emails are worth paying to read

src/email-parser/extraction-policy.ts. Finding 30 emails is cheap; reading them is one AI call each, so we only pay where it adds something:

1. Always read the newest 3. They drive the notification and the payment path, and only the invoice itself carries the billing period, usage, and the split between this month's charges and money carried over.
2. Skip any older month a cheaper source already covered. Before deciding, the run awaits two backfills — our own ledger (free) and the Plaid bank feed (one API call for the whole payment history). A historical bill only needs to be an amount to serve as a comparison point, so if the ledger already told us what May cost, re-reading May's email buys nothing.
3. Skip emails we already turned into a stored bill (matched on message id).
4. Cap at 12 reads per run. Anything beyond that is deferred, not dropped — newest-first ordering means the deferred ones are the oldest and least urgent, and the next run picks them up. A two-year first pull spreads over ~2 runs instead of blowing the five-minute per-utility budget.

Lookback window is 24 months / 30 messages (src/email-parser/lookback.constants.ts) — two years, not one, so a seasonal comparison has the same month a year ago, and so last-12-vs-previous-12 drift can be computed at all.

---

3. The GraphQL surface

3.1 Connect the mailbox

Three mutations, one per provider:

connectEmailForMailParser(serverAuthCode: String!, emailConnected: String!): GooogleAuthResponse
connectMicrosoftForMailParser(serverAuthCode: String!, emailConnected: String!): MicrosoftAuthResponse
connectImapForMailParser(
  emailConnected: String!, imapHost: String!, imapPort: Float!,
  imapUser: String!, imapPassword: String!, imapSecure: Boolean
): ImapAuthResponse

All three are @UseGuards(JwtAuthGuard) and return the same shape:

{
  "error": false,
  "message": "Successfully connected email for mail parser",
  "emailAccess": { "id": "<emailAccessId>", "email": "user@gmail.com" }
}

That emailAccess.id is the emailAccessId every downstream call keys on. It points at a UserEmailMailAccess document (src/email-parser/schemas/email-access.schema.ts) holding the provider, tokens, expiry, reconnect flags, and IMAP credentials.

3.2 Create the utility subscription

createUserSubscription(input: CreateUserSubscriptionInput!): UserSubscription

emailAccessId is required for UTILITY-type subs — omitting it throws 400 emailAccessId is required for UTILITY type subscriptions.

On the request thread the mutation runs detection only — detectUtilityInvoiceCandidates(), i.e. L1 + L2, no AI reads — then emits UTILITY_INVOICE_EXTRACT_REQUESTED and returns immediately (src/subscriptions/user-subscriptions/user-subscription.service.ts:403-419).

Response fields that matter:

hasInvoice:     true                  // OPTIMISTIC — a candidate was seen, amount not yet known
invoiceStatus:  "candidate_found"
invoiceMessage: "Found 3 candidate invoice email(s); fetching details."

Detection never throws. It is best-effort; the background run is authoritative and re-derives everything regardless of what detection returned.

3.3 Poll for the terminal state

getUserSubscription(id: String!): UserSubscription
getUserSubscriptions: [UserSubscription!]!
getUserSubscriptionsWithFilter(filter: SubscriptionFilterInput): [UserSubscription!]!

Poll invoiceStatus, not hasInvoice.

invoiceStatus	Meaning	hasInvoice
detecting	L1/L2 running	unchanged
candidate_found	Bill exists; L3 extraction still running — non-terminal	true (optimistic)
ready	Extracted; amounts and due dates written	true
none	No candidate emails, or emails found but no valid due_date / total_amount	false
failed	Email access missing, or provider error	false

3.4 Read the parsed bill

src/utility-insights/resolvers/utility-insights.resolver.ts, both JWT-scoped to the caller:

utilityBillDetail(utilitySubId: String!, billId: String): UtilityBillDetail
utilitiesWithBills: [String!]!

utilityBillDetail returns null (not someone else's bill) if the sub isn't the caller's. The comparison and quarter blocks are independently nullable — a null block means hide it, never render a zero.

3.5 REST alongside it

Route	Purpose
GET /admin/subscriptions/:subscriptionId/invoice-email	Raw invoice emails (subject, sender, date, content, invoiceDetails) for debugging a customer's pull
POST /admin/subscriptions/refresh-utility-invoices	Idempotent manual refresh; kicks off in background, returns the run record
GET /admin/subscriptions/utility-refresh-status	Latest run for the admin UI to poll
GET /email-parser/aqua-finance/invoice?userId=	One-off Aqua Finance pull-and-send

---

4. The background half

UtilityInvoiceListener (src/email-parser/listeners/utility-invoice.listener.ts) picks up UTILITY_INVOICE_EXTRACT_REQUESTED and runs the authoritative path, EmailParserService.initializeUtilityForSubscription() (src/email-parser/email-parser.service.ts:124). In order:

1. Resolve the sub, the email access, and the fingerprint. Any miss → write failed / none and return.
2. Await the ledger and bank-history backfills. Awaited, unlike the fire-and-forget calls later, because their answer changes what we pay to read. Both are idempotent.
3. findInvoiceCandidates() → getCoverage() → decideWhatToExtract() → extractInvoiceDetails() on the chosen subset.
4. Keep only emails where both due_date (or date) and total_amount parse. If none survive → invoiceStatus = 'none', hasInvoice = false.
5. Sort by due date descending. Newest → nextDueDate / nextAmountDue; second newest → lastDueDate / lastAmountDue. Those two feed the payment path.
6. On the first pull only, cross-check Plaid recurring streams: merchant-marker substring match plus amount within 5%, preferring the stream whose last_date is closest to the invoice email date. A hit marks the bill Paid with paymentSettledAt. No new Plaid calls.
7. Persist every parsed bill to bill history via recordParsedBills() — fire-and-forget, de-duplicated, and everything except the newest flagged isBackfill so a first-time connection doesn't fire ten notifications about last year's bills.
8. Mirror to the customer-utility record, then write the terminal hasInvoice: true / invoiceStatus: 'ready'.

What the AI returns

ClaudeBedrockService.extractEmailContent() returns { parsed, usageMetadata }, where parsed.invoice is:

{
  "number": "…", "date": "YYYY-MM-DD", "due_date": "YYYY-MM-DD",
  "total_amount": "…", "currency": "USD",

  "service_period_start": "…", "service_period_end": "…",  // a 34-day bill is bigger than a 30-day one
  "current_charges": "…",   // this period only, excluding carry-over
  "previous_balance": "…",  // money carried over
  "late_fees": "…",
  "usage_amount": "840", "usage_unit": "kWh",
  "prior_year_usage": "…", "prior_year_amount": "…",  // from the bill's 13-month chart, when printed
  "is_budget_billing": "boolean"  // flat/levelled plans vary by design — never comment on them
}

The prompt says never hallucinate: absent fields come back as empty strings. Parsing is tolerant — parseMaybeNumber strips $ and commas, parseMaybeBoolean treats anything uncertain as false (calling a normal bill a flat plan would silence us on a customer we should have warned).

---

5. Payment gate

Because hasInvoice is optimistic at candidate_found, it is not sufficient on its own. src/payment/orchestration/payment-orchestration.service.ts:152-186 gates separately:

• invoiceStatus is detecting or candidate_found → block, and surface a distinct "try again shortly" message. The amount isn't confirmed and this is transient, so it must not read as a permanent exclusion.
• invoiceStatus === 'failed' → excluded, reason "Could not read invoices from the connected email".
• no emailAccessId → excluded, reason "No email connected".
• hasInvoice !== true → excluded, reason "No invoice found in connected email" or "Invoice status not verified yet".

---

6. Refresh cadence

src/jobs/jobs/user-subscription-utility.job.ts:

• 0 2 */5 * * — full re-pull for every bundled utility that has an email connection. Five-daily rather than daily; utility invoices don't change intra-week, and the change cut Bedrock parse volume ~80%.
• hourly self-heal — the fixed-day cron has no missed-run catch-up, so a deploy at 02:00 would leave invoices silently 5-10+ days stale. This sweep re-pulls anything past STALE_DAYS and alerts ops past ALERT_DAYS.

Both go through a UtilityRefreshRun record:

• its partial-unique index on status: 'running' is the multi-instance leader lock (prod runs several instances and every one fires the cron),
• it is also the persistent status the admin UI polls, so progress survives browser reloads,
• a 30s heartbeat keeps a long-but-live run from being reaped; a stale run past a 5-minute heartbeat TTL is marked failed,
• each utility is wrapped in a Promise.race timeout so one hung Gmail/Claude call can't stall the whole run.

---

7. Other entry points into the same pipeline

Caller	Path taken
createUserSubscription (UTILITY)	detection sync, full run via event
updateUserSubscription when emailAccessId changes	full run, awaited
refreshUtilityInvoicesDaily / refreshStaleUtilityInvoices / admin trigger	full run per utility, sequential, timeout-capped
processExistingCustomerSubscriptions	full run, awaited, per utility
GET /email-parser/aqua-finance/invoice	resolves the user's Aqua Finance sub, then full run

---

8. File map

File	Role
src/email-parser/email-parser.service.ts	Orchestrator: initializeUtilityForSubscription, detectUtilityInvoiceCandidates, resolveFingerprint, setInvoiceStatus, recordParsedBills
src/email-parser/interfaces/email-provider.interface.ts	The L1/L2/L3 contract
src/email-parser/email-provider.factory.ts	provider string → implementation
src/email-parser/providers/*.ts	Gmail / Microsoft / IMAP
src/email-parser/parser.constants.ts	Fingerprint resolution + legacy hardcoded markers
src/email-parser/extraction-policy.ts	What's worth an AI call
src/email-parser/lookback.constants.ts	24-month window, 30-message cap
src/email-parser/listeners/utility-invoice.listener.ts	Runs the slow path off the request thread
src/email-parser/schemas/email-access.schema.ts	UserEmailMailAccess
src/email-parser/auth/{google,microsoft,imap}/	Connect mutations, token exchange, refresh
src/ai/ai-services/claude-bedrock.ai.ts	classifyEmailsAsInvoiceBatch (L2), extractEmailContent (L3)
src/utility-insights/	Bill history storage + the customer-facing bill screen
src/jobs/jobs/user-subscription-utility.job.ts	The two crons