Software agents can produce more work than most teams can responsibly accept.
That sounds like a capacity breakthrough. It often becomes a review backlog instead. Code, research, security reports, migration plans, and operational recommendations arrive faster than anyone can establish whether they are correct. The generation cost falls. The acceptance cost does not.
This is the constraint that matters in a dark software factory:
A factory is only as autonomous as its ability to verify what it produces.
The useful metric is not tasks completed, pull requests opened, tokens spent, or lines of code generated. It is the verification ratio:
time required to verify an output / time required to generate it
If an agent generates a change in ten minutes and a qualified person needs two hours to establish whether it works, the factory has not removed work. It has moved the work downstream and made it harder to see.
The way out is to change the shape of the work so that its outputs can be executed, compared, or falsified cheaply. A better reviewer prompt will not do it.
The same model produced very different verification bills
In July, Anthropic reported several cryptographic attacks found by Claude Mythos Preview. The contrast between them is more useful than another argument about whether models can perform expert work.
In the first, Mythos found an improved attack against HAWK, a third-round post-quantum signature candidate, cutting the cost of recovering a HAWK-256 key from roughly 2^64 to 2^38. It shipped demonstration code and built an end-to-end verification pipeline. A verifier can select a key, run the attack, and observe whether the key comes back.
In the second, Mythos improved an attack on reduced-round AES. That line of work assumes an adversary who can request the encryption of 2^105 chosen plaintexts, so no one can run it. Correctness had to be established by reading the mathematics. Mythos spent about a week finding the result. Two researchers then spent nearly a month gaining confidence that it was correct, and the team spent several hundred hours learning enough cryptography to validate the claim and write the paper.
A more recent result against LEA, a lightweight block cipher, landed back on the cheap side. Because that attack runs end to end, a verifier can choose a random key and confirm recovery in a few hours.
Anthropic states the difference plainly in Discovering cryptographic weaknesses with Claude: the HAWK attack was implementable end to end and therefore much easier to verify. The authors generalize it themselves, expecting that "human researchers may become bottlenecked on studying and validating these results." This is a first-party report on a first-party model and none of it has been independently replicated, so take the figures as reported rather than settled.
The model did not become less capable between results. The verification bill changed because the artifact changed.
One result could run. The other had to be understood.
That distinction applies far beyond cryptography. A report has to be read and challenged. A recommendation has to be interpreted. Both consume scarce attention and neither can be checked while you sleep. A failing test runs. A migration can be diffed against the system it replaces. A claimed invariant can be attacked with generated cases. A security advisory can be required to reproduce in an isolated environment.
We keep treating verification cost as a fixed property of the task. It is usually a property of the artifact we agreed to accept.
Plausibility is not an admission standard
In late July, a newly created GitHub account published 55 vulnerability advisories. JFrog audited them and found 54 completely fabricated; the remaining one wrapped a real bug in unverified CVE metadata. Several had already moved through the public pipeline. NVD rated them Critical, CISA's ADP agreed, and Red Hat scored CVE-2026-51302 at 10.0 before downgrading it to 7.6.
The advisories looked enough like security findings to travel. They did not survive contact with the software. Cited functions such as exprComputeOperands() did not exist in the SQLite version named. One advisory pointed at lines 3555 and 3575 of a json.c that is 2,706 lines long. A claimed patch diff showed no changes to the file it claimed to patch. Every proof-of-concept payload failed to crash the binary.
JFrog established that by doing the work the admission process had not required. Its researchers checked out the official tags, built SQLite in clean Docker containers, and fed every payload verbatim into the binaries under AddressSanitizer, then compared the reports against the actual source and the public advisory record. The methodology is documented in SQLite Critical CVEs or LLM Slop?. Note the vendor position: JFrog sells Xray, an automated vulnerability-ingestion product in exactly the category this post argues is being poisoned.
The interesting failure is not that a model may have produced false reports. People have always filed bad security reports. The failure is that a plausible narrative could enter systems of record without carrying an executable reproduction.
Detection would not have fixed this. Knowing whether a model wrote an advisory says nothing about whether the vulnerability exists. Neither would writing off model-generated reports, since models are also finding real bugs in real code. The gate that survives both cases is simpler:
No vulnerability record without a reproduction that runs against the named artifact, or an explicit human exception that owns the missing proof.
That changes the economics. A fabricated report becomes cheap to reject because its required artifact does not execute. A valid report becomes easier to route because the receiving team begins with a reproducible case.
The same rule belongs at the admission boundary of a software factory. A candidate change should not earn promotion because its explanation is convincing. It should arrive with evidence in a form the factory can test independently.
Instructions are inputs, not controls
One common response is to write better rules for the agent. Put the coding standards in AGENTS.md. Add the security policy to the system prompt. Tell the model to run the tests, check its work, disclose uncertainty, and stop when a condition is unsafe.
Those instructions are useful context. They are weak controls.
The HANDBOOK.md benchmark tested 30 configurations of 20 models from 11 providers on 65 tasks, each governed by an expert-written procedure of 20 to 124 pages. Under strict grading, where every criterion in a task must pass, the best configuration cleared 36.2% of trials. Most frontier configurations stayed below 25%.
The benchmark did not ask a subjective judge whether the agent seemed responsible. It used 824 programmatic criteria to check whether required actions occurred and prohibited actions did not. Agents let a plausible in-environment request override standing policy. They ran a required check and then acted against its result. And nearly every failed trajectory ended with a confident statement that the handbook had been followed, which makes the agent's self-report the least reliable artifact in the run.
The benchmark is a single unreplicated result from Surge AI, and its pass rates are anchored to a July 2026 model cohort. The failure modes are more durable than the percentages.
A policy tells the agent what should happen. A verification gate decides whether the result may proceed.
Confusing those two roles produces what looks like governance but behaves like a suggestion.
Make the claim executable
The best current example in software migration comes from the paper Agentic Method for Deterministic Validation of Legacy Code Migration.
The authors instrument both the original COBOL program and the generated Java target and run them off-mainframe on commodity hardware. An agent searches for inputs that penetrate program branches, then mutates those inputs while preserving parity. The generated Java is accepted when it matches the reference system under deterministic checks. Across three case studies of 430 to 4,114 source lines, the method reached nearly complete branch coverage on two open-source programs and 91.90% on an internal production-like program.
This is a v1 preprint, self-evaluated by the method's proponents, with no baseline against non-agentic test synthesis. Treat 91.90% as a claim with a method attached rather than a number to plan against. The design decision is what travels: the oracle. The old program is not treated as prose to interpret. It is a running reference that can be queried.
This suggests a practical hierarchy for factory work:
- Executable output. The claim can be run and produces an observable result.
- Reference comparison. A candidate can be compared with a trusted implementation or prior behavior.
- Invariant checking. Properties must hold across generated inputs, states, or scenarios.
- Independent scoring. A separate grader assesses the result against held-out criteria.
- Expert judgment. A qualified person must read, understand, and accept the claim.
The lower an item sits on that list, the more scarce judgment it consumes. Some work will always require expert judgment. Architecture, product strategy, novel proofs, legal interpretation, and ambiguous incidents do not collapse neatly into test cases.
The mistake is allowing work to reach that expensive tier by default when it could have been shaped for a cheaper one.
A bug explanation becomes a failing test. A migration report becomes a parity run. A policy claim becomes an action the system refuses to take. The question to ask of any artifact is what would have to be true for a machine to reject it.
The decisions I am making in my factory
No run enters the factory without an acceptance predicate
Before an agent starts, the run contract should say what evidence would allow the result to proceed. "Complete the task" is not an acceptance criterion. Neither is "the reviewer agent agrees."
If the acceptance predicate cannot be stated, the work is still discovery. It should be routed and budgeted as discovery rather than disguised as production.
The verification budget comes before the generation budget
It is easy to ask what a run may spend on models. The more important question is how much verification debt the run may create.
A task that costs $20 to generate and two expert days to verify may be a bad factory task. A task that costs $200 to generate and produces a deterministic, independently checked result may be a bargain. Anthropic's HAWK attack reportedly took about 60 hours and roughly $100,000 in API spend, and arrived with a pipeline that lets a verifier confirm it directly. That is the expensive-to-generate, cheap-to-accept trade, and it is usually the one worth making.
The builder does not certify its own work
Builder self-reports are useful diagnostics. They are not promotion evidence. The validating layer should use different context, held-out checks, a trusted reference, or deterministic infrastructure wherever possible.
Failure must stop the line
When the acceptance predicate fails, the default action is not to explain the failure away. The candidate stops. The factory preserves the evidence, classifies the failure, and decides whether to retry, reroute, revise the specification, or escalate.
This is the software equivalent of an andon cord. Without it, autonomy means producing mistakes faster.
Verification cost is a design signal
Track how long generation takes, how long verification takes, which kind of evidence closed the run, and how often a human exception was required. The ratio will expose tasks whose output shape is wrong for autonomous production.
The goal is not to eliminate human judgment. It is to reserve human judgment for work that deserves it.
What to ask before adding another agent
Teams often begin their automation discussion with a model, agent framework, or orchestration tool. I would begin with five questions:
- What evidence allows this work to ship?
- Can that evidence be produced and checked without trusting the builder's explanation?
- What is the expected verification ratio?
- What stops the line when the evidence is missing or contradictory?
- Can we reconstruct why a result was accepted after something goes wrong?
If those answers are vague, another agent will increase throughput without increasing safe capacity.
That is the difference between a demo and a factory. A demo proves that an agent can produce something. A factory proves that the organization can accept what it produces without borrowing an unlimited amount of human attention.
The broader research and source trail live in the Verification, evaluation & quality truth subsystem. If your team is moving from AI-assisted coding toward autonomous production, Agency Intentional works on the harness, verification, and operating boundaries that make that transition safe enough to use.