Incident Response When Customers Bet Training Runs on You

Customers schedule multi-million-dollar training runs against our 48-hour turnaround. That number sits in our incident-response planning the way an SLA sits in a contract: when something breaks, the first casualty is somebody’s training calendar, and the response has to be built for that stake.

Preparation starts with detection surfaces we actually watch: service health with latency baselines, developer-facing request logs with error filtering, webhook delivery logs, and the audit trail for anything privileged. An incident you detect from a customer email has already cost you twice.

Two platform features double as IR tooling. Maintenance mode flips a single admin setting and surfaces a banner across every application shell — honest degradation beats mysterious errors. And support impersonation lets a responder see exactly what an affected user sees, inside guardrails: a short-lived token that cannot renew, an unremovable banner, billing and destructive actions refused for the entire session, and every second of it in the audit log. IR access without accountability is just a second breach.

The public status page follows the same honesty rule: live health checks, measured uptime when we have it, and incident history — never a hardcoded 99.9% ornament. We removed exactly that ornament from our own page, on principle.

Every incident ends in a blameless postmortem with the same three questions: what did detection miss, what did containment lack, what does the customer need to hear? The answers become tickets, the tickets become controls, and the controls become the next incident’s shorter timeline. Security operations is compound interest — you either accrue it or pay it.