Confidence: medium. Evidence: case studies and production telemetry. Last substantive change: 2026-08.
Once software is live, the factory has to detect failures, diagnose them, and either repair the system or hand the incident to a human.
The conclusion
The factory is incomplete if it stops at merge. It needs outcome telemetry, user signals, anomaly detection, diagnosis, repair, incident records, and a path to escalate rather than fabricate success. The most dangerous operational failure is the one where the system reports a confident, plausible, false success instead of surfacing the problem.
How the thinking got here
Continuous-integration logs gave way to production observability, then to healer loops that diagnose and repair, then to taxonomies of silent and fail-plausible failure and to governance that monitors the monitor. Every step strengthens the operator's ability to trust that green means green, which is exactly what fail-plausible failure destroys.
Agents make execution evidence more important because source code no longer predicts a single path through the system. Sydney Runkle puts it plainly: "the story is actually in traces for agents". A trace becomes useful evidence only when the system connects it to outcomes, feedback, and the version of the harness that produced it.
The Astro maintenance-factory case shows a deliberately simple form of durable operations: labels are the workflow state machine, the issue thread is the event log, and reports are the handoff artifacts between isolated agents. It also treats repeated agent failure as a possible observability signal for opaque architecture, missing documentation, or insufficient tests. That interpretation is promising but remains a single, first-party case.
Credible alternatives, and when each is right
| Approach | Right when |
|---|---|
| Human SRE | high-stakes production, novel incidents |
| Agent-assisted diagnosis | speeding human incident response |
| Auto-remediation for known classes | well-understood, recurring failures |
| Full closed-loop healing | reversible, well-instrumented systems |
| Chaos-tested combinations | resilience must be proven, not assumed |
Where it fails and what we still don't know
Failures include false remediation, correlated errors between a monitor and its healer, and fabricated success that hides a real incident. Evidence is moderate; feasibility is shown, but longitudinal independent evidence is scarce. Open questions include incident command, customer communication, service-level ownership, and safe stop behavior.
What would change our mind
Longitudinal evidence that closed-loop healing improves real production outcomes over quarters, without accumulating hidden failure, would extend autonomy into operations.
Evidence and further reading
- The Art of Loop Engineering: How to Build Agents That Improve Over Time
- How Coding Agents Fail (20,574 real sessions)
- When Errors Become Narratives: a taxonomy of silent failures
- LLMs Corrupt Your Documents When You Delegate
- Coding Agents Do Not Know When to Act
- StrongDM: Software Factories and the Agentic Moment
- How we built a software factory to drive Astro's GitHub issue count to zero