← The Factory

Runtime operations, observability, incidents & healing

The factory is incomplete if it stops at merge: it needs telemetry, diagnosis, repair, and a path to escalate rather than fabricate success.

medium confidence
Machine-readable Download Markdown

Confidence: medium. Evidence: case studies and production telemetry. Last substantive change: 2026-08.

Once software is live, the factory has to detect failures, diagnose them, and either repair the system or hand the incident to a human.

The conclusion

The factory is incomplete if it stops at merge. It needs outcome telemetry, user signals, anomaly detection, diagnosis, repair, incident records, and a path to escalate rather than fabricate success. The most dangerous operational failure is the one where the system reports a confident, plausible, false success instead of surfacing the problem.

How the thinking got here

Continuous-integration logs gave way to production observability, then to healer loops that diagnose and repair, then to taxonomies of silent and fail-plausible failure and to governance that monitors the monitor. Every step strengthens the operator's ability to trust that green means green, which is exactly what fail-plausible failure destroys.

Agents make execution evidence more important because source code no longer predicts a single path through the system. Sydney Runkle puts it plainly: "the story is actually in traces for agents". A trace becomes useful evidence only when the system connects it to outcomes, feedback, and the version of the harness that produced it.

The Astro maintenance-factory case shows a deliberately simple form of durable operations: labels are the workflow state machine, the issue thread is the event log, and reports are the handoff artifacts between isolated agents. It also treats repeated agent failure as a possible observability signal for opaque architecture, missing documentation, or insufficient tests. That interpretation is promising but remains a single, first-party case.

Credible alternatives, and when each is right

Approach Right when
Human SRE high-stakes production, novel incidents
Agent-assisted diagnosis speeding human incident response
Auto-remediation for known classes well-understood, recurring failures
Full closed-loop healing reversible, well-instrumented systems
Chaos-tested combinations resilience must be proven, not assumed

Where it fails and what we still don't know

Failures include false remediation, correlated errors between a monitor and its healer, and fabricated success that hides a real incident. Evidence is moderate; feasibility is shown, but longitudinal independent evidence is scarce. Open questions include incident command, customer communication, service-level ownership, and safe stop behavior.

What would change our mind

Longitudinal evidence that closed-loop healing improves real production outcomes over quarters, without accumulating hidden failure, would extend autonomy into operations.

Evidence and further reading