A dark software factory is a domain-bounded production system. Humans specify intent, risk, and policy; a model, harness, and environment plan, build, verify, ship, observe, and repair software with little routine human intervention.
This synthesis changes when new evidence enters the corpus.
The manufacturing image, lights off and nobody on the floor, is useful rhetoric but a poor literal design goal. The evidence does not support removing humans from accountability, product judgment, or exception handling. It supports removing them from routine code production and routine inspection, and only where independent evidence, constrained execution, rollback, and production feedback make that safe.
The shortest version is:
Intent in; independently verified, policy-compliant outcomes out; evidence and accountability preserved throughout.
Judge the whole production loop, not the model or its latest patch. The factory's product is an accepted outcome plus the evidence that makes it safe to accept. The lights may go off only after one domain has earned that level of trust.
The ten top-line conclusions
-
Factory performance is a system property, not a model score. Task specification, context, tools, state, permissions, verification, and intervention policy jointly shape outcomes. In one ecosystem-scale study of 64,380 runs, the framework explained far more behavioral variance than the choice of model. But more harness is not automatically better: a first-party account of removing 80% of one coding agent's system prompt argues for re-testing and deleting instructions as model capability changes. Evaluate and tune the whole system.
-
Independent verification is the load-bearing kernel. Agent self-validation is not a trustworthy promotion gate, because agents can game, weaken, delete, or overfit to checks they can see. Reliability needs external evidence selected for the risk and domain, for example deterministic checks, held-out scenarios, context-separated review, production canaries, and rollback. These layers reduce correlated failure; none is universally required or individually infallible.
-
Autonomy should follow verifiability and blast radius, not task labels. The same task is disposable in one environment and high-risk in another. Set autonomy separately for each risk domain and give every domain its own run contract.
-
The human bottleneck moves; it does not disappear. Code generation becomes abundant while product judgment, specification quality, review capacity, security triage, and incident ownership become scarce. Accelerating generation without redesigning those downstream queues builds a faster failure system.
-
Production truth outranks build truth. Passing a benchmark or a test suite is local task completion only. Longitudinal maintenance, escaped defects, rollbacks, and incident behavior decide whether the factory actually works, and the evidence here is still thin.
-
Containment beats repeated permission prompts. Human approval degrades under volume: in a controlled sabotage study, 94% of developers missed deliberate data exfiltration and 56% merged it even after a correct alert. Filesystem and network isolation, scoped credentials, and constrained promotion paths cap the damage while permitting more useful autonomy.
-
Parallel agents are a throughput technique, not an intelligence guarantee. Parallelism pays only when work is isolatable and validation is strong. Runtime branching and worktree isolation are better supported than fixed agent org charts or theatrical fleets.
-
Self-improvement is plausible at the harness layer, but it must stay experimentally governed. The near-term target is not recursive self-improvement of the model. It is versioned improvement of prompts, context, tools, workflows, and harness code under frozen evaluations, shadow runs, limited rollout, and automatic reversion.
-
Machine-legible repositories can still be hard to govern. Agent-readable repositories improve performance, but architectural drift, cognitive debt, and loss of a shared domain language accumulate without an immediate test failure. These need explicit observability; better documentation prompts alone will not catch them.
-
The real economic metric is cost per accepted, durable outcome, not tokens, lines, patches, or raw task-success rate. The honest denominator includes retries, validation, review, corrective maintenance, incidents, infrastructure, and human attention. Technical deflation can coexist with rising verification and coordination cost.
How the thesis got here
The idea has moved through stages, and each corrected the previous one's central mistake.
| Stage | Dominant idea | What the next stage corrected |
|---|---|---|
| Autocomplete and chat | The model writes code faster | Local speed says little about end-to-end delivery |
| Coding agent | The model can plan, edit, and test | A capable loop still fails without good context, tools, and completion criteria |
| Spec-driven development | Humans specify; agents implement | Visible specs and tests can be incomplete or gamed |
| Dark factory | Humans need not write or read code | Literal human absence confuses production labor with accountability and intent |
| Harness engineering | Shape the environment and feedback loops | A harness is not static; it needs observability, versioning, and evaluation |
| Governed factory | Risk-scoped autonomy plus independent evidence | Build success is not steady-state production success |
| Adaptive factory | The harness improves itself under gates | The evaluator, the policy boundary, and long-term system health remain unsolved control problems |
The factory breaks into subsystems, each with its own current conclusion, credible alternatives, and open questions. The published subsystems are below, and more arrive as the evidence is synthesized.