← back to index

System intake unified — every entry point, one workflow runner SubstrateR552 + R553

10 source surfaces · event ledger · workflow runner · HITL · 6 fan-out outcomes

Mike asked: intake unification — emails + chat go through the same path so I only have one place to look. This shows how every inbound signal (5 email mailboxes, chat, admin pages, cron, webhook, direct API) flows through the event ledger and subscription engine into the workflow runner and the HITL gate. The bottom lane is the fan-out: NS push queue, D1 writes, KV invalidations, notifications, Vectorize writes, event re-fire.

0 · Visual flow 5 swimlanes · 25 nodes

System flow
01 / Entry surfaces (every way a request can arrive) 02 / Authn + identity (each surface has a different proof) 03 / Event ledger + subscription engine (single convergence point) 04 / Workflow runner → HITL gate (every write needs Mike) 05 / Fan-out outcomes (where the work lands) Inbound email mailbox routed by Cloudflare Email Routing. Per-mailbox handler dispatch (R36) via mailbox_routes table. DKIM enforced when require_dkim=1. SOURCE: email mailbox address: bids@ai-globalfoodsolutions.co role: bid_intake handler_function: handleBidIntake purpose: PDF vision → bid_lines proposal authn: sender domain → known_sender lookup; DKIM gate when required i bids@ bid_intake Inbound email mailbox routed by Cloudflare Email Routing. Per-mailbox handler dispatch (R36) via mailbox_routes table. DKIM enforced when require_dkim=1. SOURCE: email mailbox address: pricing@ai-globalfoodsolutions.co role: price_request_intake handler_function: handlePriceRequest purpose: PDF vision → price_request authn: sender domain → known_sender lookup; DKIM gate when required i pricing@ price_request_intake Inbound email mailbox routed by Cloudflare Email Routing. Per-mailbox handler dispatch (R36) via mailbox_routes table. DKIM enforced when require_dkim=1. SOURCE: email mailbox address: customer@ai-globalfoodsolutions.co role: customer_onboarding handler_function: handleCustomerInquiry purpose: Minimal parse → forward authn: sender domain → known_sender lookup; DKIM gate when required i customer@ customer_onboarding Inbound email mailbox routed by Cloudflare Email Routing. Per-mailbox handler dispatch (R36) via mailbox_routes table. DKIM enforced when require_dkim=1. SOURCE: email mailbox address: vendor@ai-globalfoodsolutions.co role: vendor_cost_intake handler_function: handleVendorCost purpose: PDF vision → cost update authn: sender domain → known_sender lookup; DKIM gate when required i vendor@ vendor_cost_intake Inbound email mailbox routed by Cloudflare Email Routing. Per-mailbox handler dispatch (R36) via mailbox_routes table. DKIM enforced when require_dkim=1. SOURCE: email mailbox address: pricerequest@ai-globalfoodsolutions.co role: price_request_intake handler_function: handlePriceRequest purpose: alias for pricing@ authn: sender domain → known_sender lookup; DKIM gate when required i pricerequest@ price_request_intake Browser chat UI at chat.html. Sends POST /api/chat with X-Role-Id header. Role mapped to tool palette via tool_role_palettes (R556). Mike + 10 role users. SOURCE: chat endpoint: POST /api/chat authn: X-Role-Id header (10 roles) palette: filterToolsForRole() default_mode: council_v2 i chat.html role-gated chat UI Admin button clicks from static pages (ops.html, proposed-actions.html, cost-ingestion.html, etc). Every write button posts with X-Edit-Token. Pages stitch results from chat tools. SOURCE: admin pages surfaces: ops.html, proposed-actions.html, cost-ingestion.html, training.html, review.html authn: X-Edit-Token (checkEditToken) count: 20+ static HTML pages i admin HTML ops/proposed-actions/cost-ingestion Cloudflare Workers scheduled events. 22 cron expressions in wrangler.jsonc. Tiered hot (5m) / warm (15m) / cold (60m) + weekly + monthly. KV-mutexed so a long run doesn't double-fire. SOURCE: cron count: 22 schedules tiers: hot 5m, warm 15m, cold 60m authn: none (CF-internal trigger) mutex: KV cron-lock keys i cron triggers 22 scheduled events NetSuite SystemNote webhook fires on record changes. HMAC-verified. Drives Change Data Capture (CDC) into sync_log and downstream re-syncs. SOURCE: webhook endpoint: POST /api/ns-webhook authn: HMAC verify (shared secret) emitter: NetSuite SystemNote effect: invalidates D1 mirror row i webhook /api/ns-webhook NS SystemNote CDC Direct programmatic invocation. Used by tests, admin scripts, codex agents. Requires CF Access JWT + X-Edit-Token. SOURCE: direct api endpoints: /api/workflow/execute, /api/tools/invoke authn: CF Access JWT + X-Edit-Token i direct API /api/workflow/execute Inbound email sender domain matched against customers/vendors. Unknown senders are allowed but downgraded priority. DKIM-required mailboxes additionally enforce DKIM=pass. AUTHN: email mailbox proof: sender domain → customer/vendor row fallback: parse_status='manual_review_needed' if domain unknown optional: DKIM enforce i Sender-domain known_sender check Chat header X-Role-Id maps each request to a tool palette. R556 enforces this at filterToolsForRole. 10 distinct roles (admin, pricing_admin, ar_admin, bid_admin, etc). AUTHN: chat header: X-Role-Id roles: admin, pricing, ar, bid, nutrition, production, ops, relationship, order_mgmt, all i X-Role-Id tool palette gate Every write endpoint requires X-Edit-Token. Validated via checkEditToken against EDIT_TOKEN env var. Read endpoints don't need it. AUTHN: admin page header: X-Edit-Token validator: checkEditToken(request, env) source_of_truth: env.EDIT_TOKEN i X-Edit-Token write-gate (admin pages) Cron entry points are NOT reachable externally. Cloudflare invokes scheduled() handler directly. No JWT, no token. Mutex via KV prevents double-fire. AUTHN: cron proof: none (CF-internal trigger) protection: KV cron-lock mutex i (cron has no auth) CF-internal only Webhook authenticity verified with HMAC-SHA256 against shared secret stored in env.NS_WEBHOOK_SECRET. Rejects request if signature mismatch. AUTHN: webhook algorithm: HMAC-SHA256 secret: env.NS_WEBHOOK_SECRET header: X-NS-Signature i HMAC verify NS-shared secret Single convergence point for all inbound signals. Every source writes to events table (append-only) with idempotency_key per (producer, key) so retries collapse. Drives downstream subscriptions. CONVERGENCE: event ledger table: events (R553) schema: event_type, entity_type, entity_id, payload_json, caused_by, source_system, occurred_at idempotency: (producer, key) UNIQUE current_producers: email, workflow_runner, sync_engine, chat, cron i Event ledger append-only events table (R553) 120+ events recorded · 5 producers wired Reads events table by cursor, matches each event against event_subscriptions (event_type + filter_json), invokes the bound workflow contract via the runner. R560 cursor-fix. CONVERGENCE: subscription engine table: event_subscriptions matcher: event_type prefix + JSONPath filter cursor: per-drainer last_seen_event_id effect: invokes executeWorkflowContract i Subscription engine event_subscriptions + cursor table fires matching workflow contracts The single function every workflow contract runs through. 7 stages: trigger → context → preconditions → HITL gate → fan-out → post → verify. Implements R560 hardening + idempotency. RUNNER: workflow entry: executeWorkflowContract(env, contract, input) contracts: 22 (workflow-*.html) stages: 7 reflexion: writes reflexion_log when contract.reflexion_enabled=1 i Workflow runner executeWorkflowContract (R552) 22 contracts on this substrate Risk-gated approval queue. Risk-tier >= 3 stages a proposed_actions row with status='pending' and blocks fan-out until Mike approves. ADR-031. R560 atomic-claim race fix. HITL: gate table: proposed_actions trigger: risk_level >= 3 approver: Mike (single-admin) claim: atomic UPDATE...RETURNING (R560 race-fix) ADR: 031 i HITL gate proposed_actions (R367/R560) Mike approves in /proposed-actions.html Cloudflare Queue carrying NS writes. Drainer consumer dequeues, calls NS RESTlet, flips ns_pending_pushes.status to 'applied' or 'failed' (DLQ after retries). FANOUT: NS push queue: NS_PUSH_QUEUE stages: ns_pending_pushes → queue → NS RESTlet → status='applied' retry: 3 then DLQ i NS push queue NS_PUSH_QUEUE Direct D1 INSERT/UPDATE/UPSERT on the local mirror. Used for derived data + platform state (proposed_actions, reflexion_log, workflow_run_log, customer_health_scores, etc). FANOUT: D1 writes tables: 162 (R563) patterns: INSERT OR REPLACE on NS id for synced tables; explicit derived writes for platform state i D1 writes 162 tables KV keys flushed for cached query results, role contexts, auto-context, suiteql cache. Invalidation key set in env.KV. FANOUT: KV uses: query cache, auto-context cache, mutex locks ttl: varies (5m to 1h) i KV invalidations cache flush Outbound emails staged as proposed_actions (action_type='email_send') for Mike's HITL review. Cloudflare Email Routing API sends after approval. Plus DLQ alerts. FANOUT: notifications channels: outbound email (CF Email Routing), DLQ alerts HITL-gated: yes (no auto-send) i Notifications email drafts + alerts Reflexion entries, decision_corpus rows, and knowledge-chunk additions are embedded and inserted into Vectorize indexes (ns_knowledge, decision_corpus, suiteql_corpus). FANOUT: vectorize indexes: ns_knowledge (3,360), decision_corpus, suiteql_corpus trigger: post-HITL approval for decision_corpus; nightly cron for knowledge i Vectorize writes embeddings index Runner emits workflow.completed | partial | failed event to ledger. Idempotency_key per run_id. Downstream subscriptions can react. FANOUT: event re-fire events: workflow.completed, workflow.partial, workflow.failed idempotency_key: run_id consumers: downstream subscriptions, audit i Event re-fire events.workflow.completed

1 · What this is

goal
Show every entry point feeding into the workflow runner — no matter how a request arrives, the same gate logic applies.
layout
horizontal swimlanes, top→bottom convergence
lanes
5 lanes · 25 nodes
substrate
R552 workflow runner + R553 event ledger + R367/R560 HITL
completeness
All 5 mailboxes (bids@/pricing@/customer@/vendor@/pricerequest@) verified against migrations/033_outbound_email.sql

2 · Sources at a glance 5 inbound mailboxes + 5 other surfaces

SourceSurfaceAuthn proofDefault priority
bids@ai-globalfoodsolutions.coemailsender-domain + optional DKIMhigh
pricing@ai-globalfoodsolutions.coemailsender-domainhigh
customer@ai-globalfoodsolutions.coemailsender-domainnormal
vendor@ai-globalfoodsolutions.coemailsender-domainnormal
pricerequest@ai-globalfoodsolutions.coemail (alias for pricing@)sender-domainhigh
chat.htmlchat UIX-Role-Idinline
ops.html, proposed-actions.html, cost-ingestion.html, ...admin pagesX-Edit-Tokeninline
22 cron schedulesscheduled handlernone (CF-internal)background
/api/ns-webhookNS SystemNote CDCHMAC verifyreactive
/api/workflow/execute, /api/tools/invokedirect APICF Access JWT + X-Edit-Tokeninline

3 · How the convergence works

Every surface, regardless of authn proof, lands a row in the events table (R553) as the first persistent write. The events row carries event_type (e.g. email.parsed, chat.tool_invoked, cron.tick, ns.systemnote) + a JSON payload + idempotency_key. The subscription engine reads forward from its cursor, matches event_type + filter against event_subscriptions, and dispatches the bound contract through executeWorkflowContract. This is the substrate guarantee: one place to instrument, one place to gate.

4 · How to read it

ColorMeaning
frontendUser-facing surface (chat UI, admin HTML pages)
backendWorker logic / agent code / business rules
databaseD1 table / R2 object / KV key / Vectorize index
cloudExternal system (NetSuite, Anthropic, etc.)
securityGate / policy / HITL approval / kill switch
messagebusEvent ledger, Queues, async fan-out
externalInbound source (email, webhook, cron tick, user input)
→ solidSynchronous call (request → response)
→ greenApproved / happy-path
→ red dashedPolicy or security check
→ grey dashedOptional / conditional / async