← The Factory

Runtime Operations, Observability, Incidents & Healing

Telemetry, diagnosis, and repair, with a path to escalate rather than fabricate success.

medium confidence

Confidence: medium. Evidence: case study and production telemetry. Last substantive change: 2026-07.

This subsystem owns what happens after software is running: watching it, noticing when it breaks, diagnosing why, and either fixing it or handing off to a human.

The conclusion

The factory is incomplete if it stops at merge. It needs outcome telemetry, user signals, anomaly detection, diagnosis, repair, incident records, and a path to escalate rather than fabricate success. The most dangerous operational failure is the one where the system reports a confident, plausible, false success instead of surfacing the problem.

How the thinking got here

Continuous-integration logs gave way to production observability, then to healer loops that diagnose and repair, then to taxonomies of silent and fail-plausible failure and to governance that monitors the monitor. The through-line is that an operator has to be able to trust that green means green, which is exactly what fail-plausible failure destroys.

Credible alternatives, and when each is right

Approach Right when
Human SRE high-stakes production, novel incidents
Agent-assisted diagnosis speeding human incident response
Auto-remediation for known classes well-understood, recurring failures
Full closed-loop healing reversible, well-instrumented systems
Chaos-tested combinations resilience must be proven, not assumed

Where it fails and what we still don't know

Failures include false remediation, correlated errors between a monitor and its healer, and fabricated success that hides a real incident. Evidence is moderate; feasibility is shown, but longitudinal independent evidence is scarce. Open questions include incident command, customer communication, service-level ownership, and safe stop behavior.

What would change our mind

Longitudinal evidence that closed-loop healing improves real production outcomes over quarters, without accumulating hidden failure, would extend autonomy into operations.

Evidence and further reading