---
title: 'Runtime operations, observability, incidents & healing'
description: 'The factory is incomplete if it stops at merge: it needs telemetry, diagnosis, repair, and a path to escalate rather than fabricate success.'
canonical_url: 'https://darkfactory.dev/factory/runtime-operations'
markdown_url: 'https://darkfactory.dev/factory/runtime-operations.md'
collection: factory
date_published: '2026-07-16T00:00:00-04:00'
date_modified: '2026-08-09T00:00:00-04:00'
---

# Runtime operations, observability, incidents & healing


**Confidence: medium.** *Evidence: case studies and production telemetry.* *Last substantive change: 2026-08.*

Once software is live, the factory has to detect failures, diagnose them, and either repair the system or hand the incident to a human.

## The conclusion

**The factory is incomplete if it stops at merge.** It needs outcome telemetry, user signals, anomaly detection, diagnosis, repair, incident records, and a path to escalate rather than fabricate success. The most dangerous operational failure is the one where the system reports a confident, plausible, false success instead of surfacing the problem.

## How the thinking got here

Continuous-integration logs gave way to production observability, then to [healer loops that diagnose and repair](https://www.danshapiro.com/blog/2026/02/you-dont-write-the-code/), then to taxonomies of silent and fail-plausible failure and to governance that monitors the monitor. Every step strengthens the operator's ability to trust that green means green, which is exactly what fail-plausible failure destroys.

Agents make execution evidence more important because source code no longer predicts a single path through the system. Sydney Runkle puts it plainly: ["the story is actually in traces for agents"](https://www.youtube.com/watch?v=jPPiZ22DY3g&t=871s). A trace becomes useful evidence only when the system connects it to outcomes, feedback, and the version of the harness that produced it.

The Astro [maintenance-factory case](https://blog.cloudflare.com/astro-issue-triage/) shows a deliberately simple form of durable operations: labels are the workflow state machine, the issue thread is the event log, and reports are the handoff artifacts between isolated agents. It also treats repeated agent failure as a possible observability signal for opaque architecture, missing documentation, or insufficient tests. That interpretation is promising but remains a single, first-party case.

## Credible alternatives, and when each is right

| Approach | Right when |
|---|---|
| Human SRE | high-stakes production, novel incidents |
| Agent-assisted diagnosis | speeding human incident response |
| Auto-remediation for known classes | well-understood, recurring failures |
| Full closed-loop healing | reversible, well-instrumented systems |
| Chaos-tested combinations | resilience must be proven, not assumed |

## Where it fails and what we still don't know

Failures include false remediation, correlated errors between a monitor and its healer, and fabricated success that hides a real incident. Evidence is moderate; feasibility is shown, but longitudinal independent evidence is scarce. Open questions include incident command, customer communication, service-level ownership, and safe stop behavior.

## What would change our mind

Longitudinal evidence that closed-loop healing improves real production outcomes over quarters, without accumulating hidden failure, would extend autonomy into operations.

## Evidence and further reading

- [The Art of Loop Engineering: How to Build Agents That Improve Over Time](https://www.youtube.com/watch?v=jPPiZ22DY3g)
- [How Coding Agents Fail (20,574 real sessions)](https://arxiv.org/abs/2605.29442)
- [When Errors Become Narratives: a taxonomy of silent failures](https://arxiv.org/abs/2606.14589)
- [LLMs Corrupt Your Documents When You Delegate](https://arxiv.org/abs/2604.15597)
- [Coding Agents Do Not Know When to Act](https://arxiv.org/abs/2605.07769)
- [StrongDM: Software Factories and the Agentic Moment](https://factory.strongdm.ai/)
- [How we built a software factory to drive Astro's GitHub issue count to zero](https://blog.cloudflare.com/astro-issue-triage/)