Batch notify · wiki guide

What this is

Batch customer notification

Batch notify is how Mike reaches a customer cohort without copy-pasting fifty emails. A filter defines the cohort (e.g. "all NYC DOE customers on bid B5875"), a template defines the body, and the runner personalizes per recipient, gates approval, and sends with idempotency.

It exists because hand-rolling bulk emails was the #1 source of "did we send that to the right customer?" anxiety. The v2 contract upgrade in R551 added the idempotency key + circuit breaker + per-recipient HITL option that closed the last gap.

Risk level 4 (high) — sending the wrong message to a cohort is an externally-visible mistake. The 50-recipient cap, kill switch, and per-recipient idempotency exist precisely for that reason.

When to use it

Trigger conditions

Pricing change cascading to a customer cohort — bid_award_notification chains into this.
Holiday schedule or facility-closure notice that needs to reach a list.
Recall or food-safety notice (rare but high-stakes — coordinate with food-safety log).
Annual roll customer cohort notification (annual_price_roll fans into this).
New product launch within a shared brand line.

Heuristic

The 48-hour no-spam check catches the most common accidental double-send: re-running yesterday's batch by mistake. Recipients already emailed in the last 48 hours surface a warn, not a block — Mike confirms.

Step-by-step what happens

The 3 beats

01

Personalize per recipient

The runner loops the recipient list and stages one proposed_actions row per customer with the personalized body. Variables (customer name, account ref, school year) substitute from context.recipients.

Writes proposed_actions

Time ~80ms per recipient

Kind loop_over_recipients

Status stub
02

Mike reviews per-recipient (or cohort-wide)

On /proposed-actions.html, Mike can approve each row individually or use bulk-decide to approve the whole cohort. R560's atomic-claim prevents double-approve race.

Writes proposed_actions.status

Time Mike-paced

Kind hitl_review

Status stub

Note cohort-wide bulk approve available
03

Send with idempotency + circuit breaker

Each approved recipient triggers an outbound email send. Idempotency key is workflow_id+customer_id — a duplicate trigger no-ops. After 5 consecutive failures the circuit breaker aborts the batch.

Writes outbound_email_log

Time ~1–3s per send

Kind send_email

Status stub

Note circuit breaker · abort after 5 fails

Outcomes

What's different after the workflow runs

Recipient cap

≤ 50

precondition

Idempotency

Enforced

workflow+customer key

Circuit breaker

5 fails

auto-abort

HITL

Per-recipient

or cohort bulk

Each approved recipient has a row in outbound_email_log stamped with the workflow_run_id.
Any unapproved recipient remains in proposed_actions with status=pending or rejected.
The workflow_runs row records the cohort filter, approved count, and send count.
reflexion_log carries a batch_notify tagged row for next-cohort comparison.

Failure modes

What can go wrong and how to recover

Circuit breaker trips (5 consecutive fails)

The batch aborts. Already-sent recipients are not affected; remaining staged rows stay in proposed_actions. Mike investigates the underlying email-send failure, then resumes with POST /admin/batch/resume?run_id=<id>.

Idempotency hit on retry

A second approve on the same recipient no-ops cleanly. Logged but harmless.

Recipient cap exceeded

Precondition blocks the launch. Mike either tightens the filter or explicitly sets a higher max_recipients in the input (audited).

Kill switch tripped mid-batch

Already-sent recipients remain sent. Pending stage rows stay pending. Mike investigates, then resumes or cancels manually.

Adjacent workflows + diagrams

For developers

Code paths + invariants

Concern	Where
Workflow contract	workflow_definitions WHERE workflow_type='batch_notify'
Idempotency key	workflow_id + customer_id
Circuit breaker	5 consecutive failures → abort
Recipient cap	50 default, override via max_recipients input
No-spam window	48 hours since last batch_notify send
Kill switch	kill:high_risk_ops
Risk level	4
Expected duration	~variable (cohort-size driven)
Trigger	hitl_approval · proposed_actions approved with action_type=bulk_customer_email

// Idempotency key (R551) const key = `${workflow_run_id}:${customer_id}`; if (sentKeys.has(key)) return { skipped: 'idempotent_noop' }; // Circuit breaker if (consecutiveFails >= 5) throw new Error('circuit_breaker_tripped');

Changelog

Dated trail · spot stale claims

Dated trail of when this doc was last touched, what changed, and what to look at if it feels stale.

Date	Round	Change	Touched by
`2026-05-26`	`R586`	Added CHANGELOG · SCHEMA · RUNBOOK · BACKLOG sections — wiki became best-in-class operating documentation.	Mike + Claude
`2026-05-25`	`R584/R585`	Wiki originally shipped — 8-section structure (hero / what / when / steps / outcomes / failure-modes / related / for-developers).	Mike + Claude

If today is more than 60 days past the latest changelog row, treat live system behavior as the source of truth. The doc may have drifted — verify against the workflow contract in workflow_definitions WHERE workflow_type='batch_notify' before acting on these claims.

Schema · data contract

The machine-readable spec

Canonical fields, table names, endpoint signatures. What code should match, what tests should assert. workflow_type · batch_notify · risk_level · 2.

Inputs (required + optional)

Field	Type	Description
`batch_id`	`string`	Unique batch identifier. Required.
`recipients`	`json`	Array of {customer_id, channel, template}.
`template_id`	`string`	Notification template. Required.
`send_after`	`datetime?`	Optional delayed-send.

D1 tables written

Table	Operation	Trigger
`outbound_email_log`	INSERT (one per recipient)	All status=pending_review
`batch_notify_runs`	INSERT	Run state
`events`	INSERT (batch_notify.staged)	audit

Endpoints called

Method	Path	Purpose
`POST`	`/api/batch-notify/stage`	Stage a batch
`POST`	`/api/batch-notify/:id/send`	Mike approves whole batch
`GET`	`/api/batch-notify/:id`	Batch status

Events fired

event_type	When	Subscribers
`batch_notify.staged`	On stage	audit + queue UI
`batch_notify.sent`	Per-recipient send	delivery tracking
`batch_notify.bounce`	Bounce received	customer_health

Runbook · when it breaks

It broke at 2am — what now

Different from "how do I use this." This is the page Mike pulls up when something is wrong: logs to check, recovery steps, who to escalate to.

Scenario · Batch staged 500 emails but Mike only approved 1 — what happened to the rest?

Batch approval is all-or-nothing today. If you click approve on one row in /proposed-actions, you only approve one.

Use batch-send: POST /api/batch-notify/<id>/send approves the entire batch.
Check status: SELECT status, COUNT(*) FROM outbound_email_log WHERE batch_id='<id>' GROUP BY status
Discard rest: If only one was wanted, bulk-reject the others.

Scenario · Bounce rate spiked after a batch send

Stale email addresses or content flagged as spam.

Inspect bounces: SELECT customer_id, bounce_reason FROM outbound_email_log WHERE batch_id='<id>' AND status='bounced'
Update emails: Patch customer.email_primary where bounced.
Throttle: If > 5% bounce rate, pause future batches until cleanup.

Scenario · Batch_notify cron didn't pick up scheduled batch

send_after window passed but emails still queued.

Check cron: npx wrangler tail during cron tick
Manual send: POST /api/batch-notify/<id>/send
Verify schedule: Cron expression in wrangler.toml — should be */5 or similar.

Logs to check

workflow_run_log · top-level run audit
workflow_step_log · per-step trace
workflow_verify_results · post-window verify outcomes
cron_locks · stuck cron lock detection
events · workflow.completed / workflow.failed event trail
reflexion_log · per-run narrative (if reflexion_enabled)
npx wrangler tail · live Worker logs

Kill switch · emergency stop

If this workflow is misbehaving in a high-impact way (creating bad proposed_actions in volume, pushing wrong things to NS), flip a kill switch:

kill:ns_writes · stops every NS push platform-wide
kill:proposed_apply · stops HITL approvals from executing fan-out
kill:high_risk_ops · stops risk_level >= 4 fan-out

See kill-switches-state-machine.html for the full state machine + recovery procedure.

Escalation

Primary: Mike Levine (single-admin) · mikelevine@globalfoodsolutions.co. For prolonged outage during business hours, notify warehouse lead + accounting lead so they can defer dependent work.

Backlog · open questions

What's not done · what's uncertain

What's not done, what's uncertain, what we punted. Captured so it survives context switches and doesn't die in someone's head.

OPEN
Per-recipient personalization
Today template is single-pass with mail-merge. More advanced personalization (per-customer pricing context) is manual.
STUB
Open-rate tracking
Today we know sends and bounces. No tracking on opens or clicks.
DECISION
Throttling policy
How many emails/hour can the worker send before mail provider rate-limits?
DEFER
Multi-channel (SMS, Slack)
Today email-only. recipients.channel allows for future expansion.