Anomaly remediation

What this is

Anomaly remediation is the substrate workflow that closes the loop between "the AI noticed something weird" and "Mike has a one-tap remediation in his queue." It runs every 15 minutes on the existing R174 cron, so the latency between an anomaly being detectable and Mike seeing the prompt is bounded.

The scan reuses the find_anomalies_today chat tool internals, then classifies findings by category and severity. For each anomaly, a remediation row stages in proposed_actions with action_type=anomaly_response.

Risk is 2 (low) — nothing pushes to NS from this workflow. Worst case is a noisy queue, which is recoverable.

When to use it

Trigger conditions

Triggered by the 15-minute R174 cron — no manual invocation needed in steady state.
Manual re-run when a vendor cost spike is suspected outside the cron window.
After a deploy that changed the anomaly heuristics — useful to validate new findings.

Heuristic

Anomalies are de-duplicated against rows already open in anomaly_investigations within the last 2 hours. Repeated detections of the same anomaly do not generate duplicate proposed_actions.

Step-by-step what happens

The 2 beats

01

Propose one remediation per anomaly

For each fresh anomaly the runner stages a proposed_actions row with action_type=anomaly_response, the structured payload, and a suggested next step. Mike can approve, ignore, or escalate.

Writes proposed_actions

Time ~50ms per anomaly

Kind stage_proposed_action

Status real

Note one_per anomaly
02

Surface to admin dashboard

New rows insert into anomaly_investigations (the persistent record) and the weekly aggregate weekly_anomalies is updated. The admin-dashboard pulls live from both.

Writes anomaly_investigations, weekly_anomalies

Time ~80ms

Kind d1_write

Status stub

Outcomes

What's different after the workflow runs

Detection latency

≤ 15 min

cron window

Proposals

Staged

one per anomaly

Duplicates

Suppressed

2h dedupe window

Risk

Read-only push

no NS writes

New anomalies appear on /admin-dashboard.html within the next cron tick.
Each open anomaly has at least one proposed_action waiting in Mike's queue.
weekly_anomalies aggregates the trailing week for trend review.
reflexion_log holds a tagged anomaly_remediation row per cron run.

Failure modes

What can go wrong and how to recover

Anomaly scanner times out

The cron tick logs a failure in workflow_runs but doesn't retry — next tick (15 min) picks it up. Acceptable for risk-2 work.

Duplicate anomaly rows

If the 2h dedupe window is too tight for a slow-moving anomaly, manual cleanup is DELETE FROM anomaly_investigations WHERE .... Rare.

No anomalies found

A successful run that produces zero rows is normal. The reflexion entry still records the no-op tick.

Adjacent workflows + diagrams

For developers

Code paths + invariants

Concern	Where
Workflow contract	workflow_definitions WHERE workflow_type='anomaly_remediation'
Cron schedule	22,37,52 * * * * (R174)
Detector	find_anomalies_today chat tool internals
Output tables	anomaly_investigations, weekly_anomalies
Reflexion tag	anomaly_remediation
Risk level	2
Expected duration	~15 min
Trigger	cron · 22,37,52 * * * * (R174 cron)

// Dedupe key — anomaly_type + entity_ref + 2h window const existing = await db.prepare( "SELECT id FROM anomaly_investigations WHERE anomaly_type=? AND entity_ref=? AND created_at > datetime('now','-2 hours') AND status='open'" ).bind(type, ref).first(); if (existing) return { skipped: 'dedupe_hit' };

Changelog

Dated trail · spot stale claims

Dated trail of when this doc was last touched, what changed, and what to look at if it feels stale.

Date	Round	Change	Touched by
`2026-05-26`	`R586`	Added CHANGELOG · SCHEMA · RUNBOOK · BACKLOG sections — wiki became best-in-class operating documentation.	Mike + Claude
`2026-05-25`	`R584/R585`	Wiki originally shipped — 8-section structure (hero / what / when / steps / outcomes / failure-modes / related / for-developers).	Mike + Claude

If today is more than 60 days past the latest changelog row, treat live system behavior as the source of truth. The doc may have drifted — verify against the workflow contract in workflow_definitions WHERE workflow_type='anomaly_remediation' before acting on these claims.

Schema · data contract

The machine-readable spec

Canonical fields, table names, endpoint signatures. What code should match, what tests should assert. workflow_type · anomaly_remediation · risk_level · 3.

Inputs (required + optional)

Field	Type	Description
`anomaly_id`	`integer`	anomalies.id — the row triggering this. Required.
`anomaly_type`	`string`	'cost_spike' \| 'margin_drop' \| 'sync_gap' \| 'volume_outlier'.
`severity`	`integer`	1-5; controls escalation tier.

D1 tables written

Table	Operation	Trigger
`proposed_actions`	INSERT	Recommended remediation
`anomalies`	UPDATE (status=remediating)	Lock so it's not picked up twice
`anomaly_remediation_log`	INSERT	What was proposed, when, by whom
`events`	INSERT (anomaly.remediation_proposed)	audit + dashboard

Endpoints called

Method	Path	Purpose
`POST`	`/api/workflow/execute`	Runner entrypoint
`GET`	`/api/anomalies/open`	Open anomalies dashboard feed
`POST`	`/api/anomalies/:id/acknowledge`	Mike marks no-action-needed

Events fired

event_type	When	Subscribers
`anomaly.remediation_proposed`	Post-fan-out	audit + chat notification
`anomaly.resolved`	When status moves to closed	monthly report

Runbook · when it breaks

It broke at 2am — what now

Different from "how do I use this." This is the page Mike pulls up when something is wrong: logs to check, recovery steps, who to escalate to.

Scenario · Anomaly flagged but no proposed_action was created

Likely the remediation kind for that anomaly_type isn't wired (most are stubs).

Check the log: SELECT * FROM anomaly_remediation_log WHERE anomaly_id=<id>
Inspect status: SELECT status, last_error FROM anomalies WHERE id=<id>
Manual proposal: Use the appropriate propose_* chat tool directly.

Scenario · Same anomaly flagged 5x in one hour

anomaly detector not deduping.

Find dupes: SELECT entity_ref, COUNT(*) FROM anomalies WHERE created_at > datetime('now','-1 hour') GROUP BY entity_ref HAVING COUNT(*) > 1
Mark dupes as closed: Keep oldest; close newer with status=duplicate.
Fix detector: Add a 'last_flagged_at' check per entity_ref to suppress re-flagging within 24h.

Scenario · Anomaly closed but actually got worse

Closure was premature; remediation didn't actually fix.

Re-open: UPDATE anomalies SET status='open', reopened_at=datetime('now') WHERE id=<id>
Escalate severity: Bump to severity=5 to force Mike attention.
Postmortem: Log a reflexion note on why closure was premature.

Logs to check

workflow_run_log · top-level run audit
workflow_step_log · per-step trace
workflow_verify_results · post-window verify outcomes
cron_locks · stuck cron lock detection
events · workflow.completed / workflow.failed event trail
reflexion_log · per-run narrative (if reflexion_enabled)
npx wrangler tail · live Worker logs

Kill switch · emergency stop

If this workflow is misbehaving in a high-impact way (creating bad proposed_actions in volume, pushing wrong things to NS), flip a kill switch:

kill:ns_writes · stops every NS push platform-wide
kill:proposed_apply · stops HITL approvals from executing fan-out
kill:high_risk_ops · stops risk_level >= 4 fan-out

See kill-switches-state-machine.html for the full state machine + recovery procedure.

Escalation

Primary: Mike Levine (single-admin) · mikelevine@globalfoodsolutions.co. For prolonged outage during business hours, notify warehouse lead + accounting lead so they can defer dependent work.

Backlog · open questions

What's not done · what's uncertain

What's not done, what's uncertain, what we punted. Captured so it survives context switches and doesn't die in someone's head.

STUB
Most fan-out kinds for anomaly types are unimplemented
Each anomaly_type maps to a remediation strategy, most of which are stubs. Wire them as concrete workflows.
OPEN
Auto-acknowledge low-severity anomalies?
Severity 1-2 noise often gets ignored. Could auto-close after 24h with no action.
DEFER
Anomaly → root-cause inference
Today we propose a remediation per anomaly. Better: cluster anomalies to a root cause and remediate the root.
DECISION
How long to retain closed anomalies?
Audit value vs DB bloat. Default 1 year retention?