CASE STUDY 03
Operational systems
Systems designed to be observed, understood, and recovered — long after launch.
Context
Operational load had become the tax on every new feature. Incidents were resolved by the few people who held the context, and that context did not scale. The system worked until it didn't, and recovery depended on who was awake.
The aim was durability of understanding: systems that stated their own health, and recovery paths that did not rely on a single person's memory.
Approach
We defined service-level objectives that reflected what users actually experienced, then instrumented against them so alerts meant something. Noise that did not map to user impact was removed.
With trustworthy signal in place, we automated the routine recovery actions and wrote the remaining runbooks as executable steps. On-call shifted from firefighting toward reviewing what the system had already handled.
A resilient system is one that explains itself while it recovers.
Outcome
- 60%
- Reduction in alert noise
- 3×
- Faster mean time to recovery
- 24/7
- Automated first response
Selected decisions
Alert on impact, not on metrics
Objectives were framed around user experience, so every page corresponded to something a person would notice.
Executable runbooks
Recovery steps became automation with a human checkpoint, turning tribal knowledge into a repeatable, reviewable process.
Design for the second incident
Every postmortem fed back into instrumentation, so the same failure was cheaper and clearer the next time.