---
title: 'Verification, evaluation & quality truth'
description: 'How factory output earns the right to ship when humans cannot inspect every generated line.'
canonical_url: 'https://darkfactory.dev/factory/verification'
markdown_url: 'https://darkfactory.dev/factory/verification.md'
collection: factory
date_published: '2026-07-16T00:00:00-04:00'
date_modified: '2026-08-09T00:00:00-04:00'
---

# Verification, evaluation & quality truth


**Confidence: high.** *Evidence: benchmark · controlled study · production telemetry · case study.* *Last substantive change: 2026-08.*

When humans cannot inspect every generated line, "is this safe to ship?" cannot be answered by a person's judgment on a diff alone. Verification is the factory's **quality kernel**: the oracles, gates, and preserved evidence that turn a candidate change into an *accepted outcome*.

## The conclusion

**Promotion to production must rest on independent, layered, reconstructable evidence, never on the builder's own report that it passed.** No single oracle suffices; each is gameable or fallible in its own way. Trust comes from stacking independent checks and keeping the evidence that justified acceptance. Evaluate and report the whole *model + harness + environment + budget* system, not an undifferentiated "agent score."

This rests on four load-bearing findings.

**1. Self-validation is not a promotion gate.** An agent's own "it passes" is the single most common place the pipeline breaks. *T2J-Bench* shows more compute does not buy correctness and that self-validation is the dominant failure mode; *SpecBench* shows the reward-hacking gap *widens* as codebases grow; the silent-failure and 20,574-session in-the-wild studies show agents narrating success they did not achieve. The agent that wrote a change must never be the authority that certifies it.

**2. Promotion requires risk-calibrated, independent evidence.** The strongest production cases converged on *multiple layers chosen for the change's failure modes and blast radius*. Bun's Zig-to-Rust migration used a language-independent **conformance suite as oracle** (1.38M assertions, zero tests skipped or deleted) *and* an **implementer ≠ reviewer** split-context review where reviewers see only the diff. A separate practitioner account describes [conformance-driven development](https://www.youtube.com/watch?v=owmJyKVu5f8) as a general regime for implementing against a behavioral suite. As the Bun team put it, "the Claude that wrote the code wants the code to get accepted." StrongDM replaced binary pass/fail with a probabilistic **satisfaction** metric measured against a **Digital Twin Universe** of cloned enterprise apps. Different layers catch different failures: in one production runtime, a human reading the product caught roughly 70% of silent failures while the test suite caught roughly 0%. These are patterns, not a universal checklist; nominally separate checks can still share models, evidence, or infrastructure and fail together.

**3. Evaluate the system, not the model.** A single score conflates variables that move independently. Anthropic found that [**infrastructure noise alone can shift eval scores by more than the gaps between leaderboard models**](https://www.anthropic.com/engineering/infrastructure-noise), so a number without its run conditions is nearly meaningless. Version evaluations against the exact model, harness, environment, and budget; re-baseline when any of them change. Agent-operable tools such as [smevals](https://primeradiant.com/blog/2026/smevals.html) make this practical by separating tasks, configurations, preserved runs, and replayable graders, though their graders still need calibration and audit.

**4. Human review is a weak primary detector, not a disposable layer.** In a controlled study, **94% of developers missed deliberate data-exfiltration sabotage, and 56% merged it even after a correct alert fired.** Human review is essential for accountability, semantic judgment, and retained system understanding on consequential changes, but the detection load must sit on independent automated layers plus containment. Two recent practitioner accounts explicitly reject the literal no-read position; neither shows that manual review alone scales as a defect detector. Humans own consequences, and automation must supply evidence they can meaningfully challenge.

**5. A graph can organize evidence, but it cannot create truth.** A control graph can require an evaluator to run, and an evidence graph can preserve which result justified a promotion. Neither makes the evaluator correct. As the [critique of graph-engineering claims](https://www.louisbouchard.ai/graph-engineering-explained/) usefully stresses, a system can arrange weak or circular evidence into a convincing structure. Important truth signals must originate outside the builder and outside the graph being judged.

## How the thinking got here

Unit tests, then integration and regression suites, then held-out scenarios and probabilistic satisfaction, then observational equivalence and digital twins, then independent judges and context-separated adversarial review, then reconstructable episode evidence. Each stage corrected the last. Tests are sufficient, until agents game the visible tests. Held-out scenarios give independent truth, until graders and infrastructure themselves fail. Promotion needs layered, reconstructable evidence.

## Credible alternatives, and when each is right

| Oracle | Right when |
|---|---|
| Visible test suites | fast gate for low-blast-radius changes; never the promotion oracle |
| Held-out / hidden scenarios | the builder could otherwise overfit to visible checks |
| Observational equivalence / digital twins | a trusted reference or prior behavior exists to diff against |
| Conformance suite as oracle | migrations or rewrites with a language-independent behavioral spec |
| Context-separated adversarial review | you can afford a reviewer that sees only the diff |
| Model-as-judge graders | the grader is independent of the builder and spot-audited |
| Formal verification | small, high-criticality cores with a formally stateable spec |
| Human code review | accountable or critical changes, for ownership rather than detection |
| Production canaries + rollback | reversible changes where staged exposure is safe |

## When it fails

Two failures show the stakes. In the **fail-plausible** case, an encoding bug was laundered into a confident, fabricated "platform crisis" digest that passed every automated check and was caught only by a human reading the output. In the [**9-second production-volume deletion**](https://neuraltrust.ai/blog/pocketos-railway-agent), an irreversible action completed faster than any human could intervene, because "can execute" was never gated by "may execute." The lesson repeats: containment must be architectural, and the detecting layer must be independent of the builder. Adding more agents or graph nodes does not create independence when they share a model, prompt assumptions, evidence, or evaluator.

## What we still don't know

**How independent must a grader be from the builder?** Layered verification collapses into one correlated check if the "independent" layers share the builder's failure modes. The corpus shows that *some* independence helps but has no controlled measurement of the independence-reliability curve. *This would change if a controlled study measured covert-defect catch rate against grader independence: same model, then same harness, then same vendor, then fully independent.*

## What would change our mind

A single oracle, formal or model-based, that reliably gates promotion across diverse domains without being gamed, at acceptable cost, would collapse the layered-evidence requirement toward one strong check. Conversely, evidence that independent layers correlate in failure would widen the band of changes that require accountable human ownership. Neither exists today.

## How we approach it *(operator note)*

In ContextStack, builder self-reports inform but never gate. An independent quality signal (held-out scenarios plus independent judges) is trusted above the builder, the builder-versus-holdout gap is a tracked metric, and rollback is rehearsed before auto-approval expands. *This is one operator's current choice, not a universal prescription; it would be revisited if a single trustworthy grader emerged.*

---

*Sources: T2J-Bench; SpecBench; Anthropic (eval noise); "When Errors Become Narratives" (arXiv 2606.14589); "How Coding Agents Fail" (20,574 sessions); "Rewriting Bun in Rust" (bun.com); "Can Human Developers Detect AI Agent Sabotage?"; StrongDM, "Software Factories and the Agentic Moment."*

## Evidence and further reading

- [T2J-Bench: compute does not buy correctness](https://arxiv.org/abs/2605.29054)
- [SpecBench: the reward-hacking gap grows with codebase size](https://arxiv.org/abs/2605.21384)
- [Infrastructure noise moves eval scores more than model margins](https://www.anthropic.com/engineering/infrastructure-noise)
- [When Errors Become Narratives: a taxonomy of silent failures](https://arxiv.org/abs/2606.14589)
- [How Coding Agents Fail (20,574 real sessions)](https://arxiv.org/abs/2605.29442)
- [Rewriting Bun in Rust](https://bun.com/blog/bun-in-rust)
- [Can Human Developers Detect AI Agent Sabotage?](https://arxiv.org/abs/2606.05647)
- [StrongDM: Software Factories and the Agentic Moment](https://factory.strongdm.ai/)
- [NeuralTrust: post-mortem of the 9-second AI database deletion](https://neuraltrust.ai/blog/pocketos-railway-agent)
- [Bouchard: Graph Engineering Explained](https://www.louisbouchard.ai/graph-engineering-explained/)
- [Engineering Practices That Make Coding Agents Work](https://www.youtube.com/watch?v=owmJyKVu5f8)
- [smevals: a small eval suite for evaluating models, prompts, and harnesses](https://primeradiant.com/blog/2026/smevals.html)