← The Factory

Runtime Operations, Observability, Incidents & Healing

The factory is incomplete if it stops at merge: it needs telemetry, diagnosis, repair, and a path to escalate rather than fabricate success.

medium confidence

Confidence: medium. Evidence: case study and production telemetry. Last substantive change: 2026-07.

This subsystem owns what happens after software is running: watching it, noticing when it breaks, diagnosing why, and either fixing it or handing off to a human.

The conclusion

The factory is incomplete if it stops at merge. It needs outcome telemetry, user signals, anomaly detection, diagnosis, repair, incident records, and a path to escalate rather than fabricate success. The most dangerous operational failure is the one where the system reports a confident, plausible, false success instead of surfacing the problem.

How the thinking got here

Continuous-integration logs gave way to production observability, then to healer loops that diagnose and repair, then to taxonomies of silent and fail-plausible failure and to governance that monitors the monitor. Every step strengthens the operator's ability to trust that green means green, which is exactly what fail-plausible failure destroys.

Agents make execution evidence more important because source code no longer predicts a single path through the system. Sydney Runkle puts it plainly: "the story is actually in traces for agents". A trace becomes useful evidence only when the system connects it to outcomes, feedback, and the version of the harness that produced it.

Credible alternatives, and when each is right

Approach Right when
Human SRE high-stakes production, novel incidents
Agent-assisted diagnosis speeding human incident response
Auto-remediation for known classes well-understood, recurring failures
Full closed-loop healing reversible, well-instrumented systems
Chaos-tested combinations resilience must be proven, not assumed

Where it fails and what we still don't know

Failures include false remediation, correlated errors between a monitor and its healer, and fabricated success that hides a real incident. Evidence is moderate; feasibility is shown, but longitudinal independent evidence is scarce. Open questions include incident command, customer communication, service-level ownership, and safe stop behavior.

What would change our mind

Longitudinal evidence that closed-loop healing improves real production outcomes over quarters, without accumulating hidden failure, would extend autonomy into operations.

Evidence and further reading