---
title: 'Selection evals and acceptance evals are different jobs'
description: 'Selection evals answer which model, prompt, or harness to use. Acceptance evals answer whether this system ships. They want opposite tooling, and conflating them is why eval suites rot.'
canonical_url: 'https://darkfactory.dev/blog/selection-evals-and-acceptance-evals-are-different-jobs'
markdown_url: 'https://darkfactory.dev/blog/selection-evals-and-acceptance-evals-are-different-jobs.md'
collection: blog
---

# Selection evals and acceptance evals are different jobs

Two people shipped eval tooling within seventy two hours of each other and made opposite architectural bets.

Simon Willison released [`smevals`](https://simonwillison.net/2026/Jul/31/smevals/), a standalone eval runner built with Jesse Vincent's Prime Radiant lab. He is direct about where the work went:

> The most time-consuming part of this project was figuring out the vocabulary for it!

The vocabulary he settled on has nine terms: eval, task, config, run, runner, grader, grade, check, and checker. An eval directory holds YAML. You run it from the command line against multiple models, grade the runs separately, and browse the results in a local web server or a static report.

Two days earlier, at Laracon US in Boston, Nuno Maduro shipped the same capability inside [Pest 5](https://laravel-news.com/pest-5) and invented no vocabulary at all. An eval is a test. The assertion API is `expect()`, which every Laravel developer already uses for everything else.

```php
it('answers capital city questions correctly', function (): void {
    expect(CapitalCityAgent::class)
        ->prompt('What is the capital of France?')
        ->toContain('Paris')
        ->toBeRelevant()
        ->toSatisfy('The response names exactly one city.');
});
```

I was in the room for that talk. The reaction told me something the changelog does not: nobody treated this as an AI feature. It landed as a testing feature, next to test impact analysis and a PHPStan plugin, and the audience received it the way you receive a good new matcher.

It is tempting to read this as a competition. Both bets are right, because they are answering different questions, and naming the split is more useful than declaring a winner.

## The split

> **Selection evals** answer: which model, prompt, or harness should I use?
> **Acceptance evals** answer: does the system I built work well enough to ship?

Those look similar. They are almost opposites.

A selection eval needs to hold the task fixed and vary the configuration. Its value comes from comparison across models, across prompts, across harnesses. It must live outside any one application, because the moment it depends on your fixtures it stops being portable and stops being comparable. `smevals` makes configuration a first class dimension for exactly this reason: a config specifies the model, and may also specify system prompts, parameters, or agent harnesses.

An acceptance eval needs the opposite. It holds the configuration fixed, because you have already chosen, and varies the input against *your* system. Its value comes from fixture proximity. An eval that cannot reach your factories, your `RefreshDatabase` state, and your fakes is grading a model. One that can is grading your application.

That is what Pest's `--agent` flag is for. It runs a snippet inside a full Pest test, with your factories and fakes available exactly as they would be in a feature test:

```bash
./vendor/bin/pest --agent='$user = \App\Models\User::factory()->create();
    $this->actingAs($user)->get("/dashboard")->assertOk();'
```

A standalone eval runner structurally cannot do that. Not because of an implementation gap, but because being outside your application is the entire source of its portability.

Most teams I talk to have one eval suite and are asking it to do both jobs. That is why their suites rot. A selection eval that drifts toward using production fixtures stops being comparable. An acceptance eval that stays generic stops being informative.

## Laravel just made the case against its own scoreboard

The most useful evidence for this split came from Laravel itself, and it is unusually honest for a vendor writing about its own benchmark.

[Boost Benchmarks](https://laravel.com/blog/idiomatic-laravel-ai-coding-agents) is Laravel's suite for answering whether AI coding agents can write correct Laravel. Seventeen evals, real Pest tests, fresh app each run. The team reports that frontier models now clear those seventeen tasks at or near 100%, up from sixteen of seventeen at 99.4% test accuracy in their previous writeup.

Then they said the interesting thing:

> a green test suite was always a proxy for "this is good code." It is a useful proxy, one I would much rather have than not have, but it has limits.

And named the gap:

> That little gap between "works" and "belongs in this codebase" is where things start getting interesting.

Their examples of how pass rates mislead are specific. OpenAI stopped reporting SWE-bench Verified after an audit found a large share of problems had flawed test cases. Cursor demonstrated that many "successful" fixes came from models locating the known answer in GitHub or `.git` history rather than working it out, and that scores dropped sharply once that history was sealed.

Note the vendor position: this is Laravel writing about a benchmark Laravel maintains, in a post that concludes you should use more Laravel Boost. The self criticism is real, and so is the marketing frame around it.

Their pivot is to two new axes: correct code per token spent, and idiomatic rather than merely correct. On the first, they report that one model can spend an order of magnitude or two more than another on the same task for nearly identical results. On the second, they now think reference free LLM-as-judge scoring is good enough to ask "is this idiomatic?" without a golden answer to diff against, scored against nineteen concrete convention areas from Boost's best practices skill, whose first rule is "Consistency First."

Boost Benchmarks is a **selection** eval. It exists to compare models. It saturated, which is what selection evals do when the field catches up, and Laravel's response was to add new discriminating dimensions rather than to make it more specific to any one application. That is the correct move for a selection eval and would be the wrong move for an acceptance eval, which should get *more* specific to your system over time, not less.

## What Pest actually shipped, and the part I would be careful with

The [Evals plugin](https://pestphp.com/docs/evals) splits its checks along a line that matches the argument above.

**Deterministic checks need no model at all.** `toContain()`, `toMatch()`, `toBe()`, `toBeJson()`, `toHaveToolCalls()`, `toFollowTrajectory()`. These inspect output directly.

**Scored checks call a judge or an embeddings model.** `toBeRelevant()`, `toBeSafe()`, `toBeCorrect(expected: ...)`, `toBeSimilar()`, `toSatisfy()`. Each takes a threshold between 0.0 and 1.0, defaulting to 0.7.

Two design choices deserve attention.

First, `toBeCorrect()` does not ask a judge to produce a number. It asks the judge to *classify* the relationship between the response and a reference, then maps each category to a fixed score: `equal` 1.0, `approximately_equal` 0.9, `superset` 0.8, `subset` 0.6, `disagreement` 0.0. Classification is a task language models are decent at. Calibrated numeric scoring is a task they are bad at. Converting the second into the first is the kind of small decision that determines whether a [grader](/glossary/grader) is trustworthy.

Second, the scorers do not call a provider directly. They go through two single method drivers, `JudgeDriver` and `EmbeddingsDriver`, and a custom scorer declares `RequiresJudge` or `RequiresEmbeddings` if it needs one. Unmarked scorers are treated as deterministic and always run. So the expensive path is opt in by construction, and `laravel/ai` is a default rather than a dependency. You can hand it a closure that returns a canned score and exercise the whole scoring path without spending money.

The cost objection that kept evals out of CI gets a boring answer: evals skip on a normal run and only execute under `--evals` or `PEST_EVALS=1`. There is also `repeat(3)`, which samples the same prompt several times and requires every sample to pass, which is the minimum honest response to [nondeterminism](/glossary/nondeterminism).

Now the part I would be careful with.

`toFollowTrajectory()` asserts that an agent invoked a sequence of tools in the expected order. It is deterministic, it is useful, and it is grading the route rather than the result.

```php
expect(SupportAgent::class)
    ->prompt('I want to return my order and get a refund.')
    ->toFollowTrajectory(['lookup_order', 'create_return', 'issue_refund']);
```

Route checks are **regression** tools, not **quality** tools. They tell you that behavior changed. They do not tell you that behavior got worse. A model that gets better and reaches the same outcome by a shorter path fails this assertion, and the failure is correct as a signal and wrong as a judgment. If you treat a red trajectory assertion as "the agent broke," you will spend the next year pinning your system to the route a July 2026 model happened to take.

I would encode trajectories only where the sequence is itself the requirement, which is to say where an ordering constraint is a real business rule: check inventory before charging the card, verify identity before disclosing an account, take a backup before a destructive migration. Those are not routes. Those are invariants that happen to look like routes.

## What I am doing in my factory

**Two suites, two purposes, never merged.** Selection evals live outside the application and are allowed to be generic, because comparability is their whole value. Acceptance evals live in `tests/Evals` and are allowed to be parochial, because fixture proximity is theirs. When I catch myself wanting a fixture in a selection eval, that is the signal it has become an acceptance eval and should move.

**Acceptance evals gate promotion. Selection evals never do.** A selection eval informs a decision I make. An acceptance eval is a [verification gate](/glossary/verification-gate) that fails closed. Confusing the two produces a suite that blocks deploys for reasons nobody can act on.

**Deterministic first, scored only where deterministic cannot reach.** Every scored check costs money, adds variance, and introduces a second model whose failure modes I now own. `toHaveToolCalls()` before `toSatisfy()`, every time.

**Ordering constraints, not trajectories.** I encode a sequence only when the sequence is a business rule. Otherwise I assert the outcome and let the route move.

**Re-baseline on any config change.** A score without its run conditions is close to meaningless. Model, prompt, harness, and budget are all part of the configuration under test, which is the one place I think `smevals` has the better vocabulary: making config an explicit dimension is a good idea that Pest's ergonomics quietly hide.

## What to ask about your eval suite

1. For each eval you have, is it answering "which configuration" or "does this ship"? If you cannot tell, it is probably doing neither well.
2. Which of your evals block a deploy, and could the person they block act on the failure?
3. How many of your checks need a model to run? What would it take to move one of them down to deterministic?
4. When your provider ships a new model next month, which suite tells you whether to switch, and which tells you whether you are still safe?
5. Are you asserting outcomes, or are you asserting the path a model took in July 2026?

The eval layer spent two years organizing itself as a product category. What it looks like from here is a split: one half becomes a comparison instrument that lives outside your system, and the other half becomes assertions in the suite you already run. The teams that get hurt are the ones still running a single suite and wondering why it keeps telling them nothing.

---

*Research and source trail for this area live in the [Verification, evaluation & quality truth](/factory/verification) subsystem, and the related [Feedback, learning & controlled self-improvement](/factory/feedback-self-improvement) subsystem. If your team is moving from AI-assisted coding toward autonomous production, [Agency Intentional](https://agencyintentional.com/) works on the harness, verification, and operating boundaries that make that transition safe enough to use.*