Incidents
Real patterns from production AI failures. Names changed. Dollar amounts not.
Spend Shock
Runaway loops and delegation storms
Audit Delay
Controls existed, evidence did not
Service Risk
Unsafe tool calls and output drift
Cost
A coding agent hit a retry condition and recursively called itself 47,000 times. The team discovered it Monday morning from billing alerts.
What would have caught it
Loop killer + velocity policy + budget block mode.
Tier path
Monitor (free) for detection, Protect+ for blocking
Security
A healthcare assistant included social security numbers and policy IDs in responses. Discovery happened in audit review, not production monitoring.
What would have caught it
PII scanner + response redaction + enforcement receipts.
Tier path
Protect
Security
A tool-enabled assistant interpreted “cleanup” as a destructive SQL action and attempted privileged table deletion.
What would have caught it
Command firewall + tool allowlist + human approval gate.
Tier path
Protect
Compliance
The team had controls and logs but no verifiable evidence that the controls executed for specific trace events.
What would have caught it
Signed evidence chain and framework-mapped exports.
Tier path
Govern
Risk
Underwriters requested independently verifiable governance evidence artifacts. Dashboard screenshots were rejected.
What would have caught it
Underwriting-ready evidence bundle with signatures and timestamps.
Tier path
Govern
Reliability
A legal drafting assistant cited non-existent cases in a filed document. There was no pre-release reliability gate.
What would have caught it
Hallucination and groundedness checks in enforce mode.
Tier path
Enforce
Runtime governance converts unknown risk into measurable controls and verifiable evidence.