← Blog

Selection evals and acceptance evals are different jobs

Selection evals answer which model, prompt, or harness to use. Acceptance evals answer whether this system ships. They want opposite tooling, and conflating them is why eval suites rot.

Machine-readable Download Markdown

Two people shipped eval tooling within seventy two hours of each other and made opposite architectural bets.

Simon Willison released smevals, a standalone eval runner built with Jesse Vincent's Prime Radiant lab. He is direct about where the work went:

The most time-consuming part of this project was figuring out the vocabulary for it!

The vocabulary he settled on has nine terms: eval, task, config, run, runner, grader, grade, check, and checker. An eval directory holds YAML. You run it from the command line against multiple models, grade the runs separately, and browse the results in a local web server or a static report.

Two days earlier, at Laracon US in Boston, Nuno Maduro shipped the same capability inside Pest 5 and invented no vocabulary at all. An eval is a test. The assertion API is expect(), which every Laravel developer already uses for everything else.

it('answers capital city questions correctly', function (): void {
    expect(CapitalCityAgent::class)
        ->prompt('What is the capital of France?')
        ->toContain('Paris')
        ->toBeRelevant()
        ->toSatisfy('The response names exactly one city.');
});

I was in the room for that talk. The reaction told me something the changelog does not: nobody treated this as an AI feature. It landed as a testing feature, next to test impact analysis and a PHPStan plugin, and the audience received it the way you receive a good new matcher.

It is tempting to read this as a competition. Both bets are right, because they are answering different questions, and naming the split is more useful than declaring a winner.

The split

Selection evals answer: which model, prompt, or harness should I use? Acceptance evals answer: does the system I built work well enough to ship?

Those look similar. They are almost opposites.

A selection eval needs to hold the task fixed and vary the configuration. Its value comes from comparison across models, across prompts, across harnesses. It must live outside any one application, because the moment it depends on your fixtures it stops being portable and stops being comparable. smevals makes configuration a first class dimension for exactly this reason: a config specifies the model, and may also specify system prompts, parameters, or agent harnesses.

An acceptance eval needs the opposite. It holds the configuration fixed, because you have already chosen, and varies the input against your system. Its value comes from fixture proximity. An eval that cannot reach your factories, your RefreshDatabase state, and your fakes is grading a model. One that can is grading your application.

That is what Pest's --agent flag is for. It runs a snippet inside a full Pest test, with your factories and fakes available exactly as they would be in a feature test:

./vendor/bin/pest --agent='$user = \App\Models\User::factory()->create();
    $this->actingAs($user)->get("/dashboard")->assertOk();'

A standalone eval runner structurally cannot do that. Not because of an implementation gap, but because being outside your application is the entire source of its portability.

Most teams I talk to have one eval suite and are asking it to do both jobs. That is why their suites rot. A selection eval that drifts toward using production fixtures stops being comparable. An acceptance eval that stays generic stops being informative.

Laravel just made the case against its own scoreboard

The most useful evidence for this split came from Laravel itself, and it is unusually honest for a vendor writing about its own benchmark.

Boost Benchmarks is Laravel's suite for answering whether AI coding agents can write correct Laravel. Seventeen evals, real Pest tests, fresh app each run. The team reports that frontier models now clear those seventeen tasks at or near 100%, up from sixteen of seventeen at 99.4% test accuracy in their previous writeup.

Then they said the interesting thing:

a green test suite was always a proxy for "this is good code." It is a useful proxy, one I would much rather have than not have, but it has limits.

And named the gap:

That little gap between "works" and "belongs in this codebase" is where things start getting interesting.

Their examples of how pass rates mislead are specific. OpenAI stopped reporting SWE-bench Verified after an audit found a large share of problems had flawed test cases. Cursor demonstrated that many "successful" fixes came from models locating the known answer in GitHub or .git history rather than working it out, and that scores dropped sharply once that history was sealed.

Note the vendor position: this is Laravel writing about a benchmark Laravel maintains, in a post that concludes you should use more Laravel Boost. The self criticism is real, and so is the marketing frame around it.

Their pivot is to two new axes: correct code per token spent, and idiomatic rather than merely correct. On the first, they report that one model can spend an order of magnitude or two more than another on the same task for nearly identical results. On the second, they now think reference free LLM-as-judge scoring is good enough to ask "is this idiomatic?" without a golden answer to diff against, scored against nineteen concrete convention areas from Boost's best practices skill, whose first rule is "Consistency First."

Boost Benchmarks is a selection eval. It exists to compare models. It saturated, which is what selection evals do when the field catches up, and Laravel's response was to add new discriminating dimensions rather than to make it more specific to any one application. That is the correct move for a selection eval and would be the wrong move for an acceptance eval, which should get more specific to your system over time, not less.

What Pest actually shipped, and the part I would be careful with

The Evals plugin splits its checks along a line that matches the argument above.

Deterministic checks need no model at all. toContain(), toMatch(), toBe(), toBeJson(), toHaveToolCalls(), toFollowTrajectory(). These inspect output directly.

Scored checks call a judge or an embeddings model. toBeRelevant(), toBeSafe(), toBeCorrect(expected: ...), toBeSimilar(), toSatisfy(). Each takes a threshold between 0.0 and 1.0, defaulting to 0.7.

Two design choices deserve attention.

First, toBeCorrect() does not ask a judge to produce a number. It asks the judge to classify the relationship between the response and a reference, then maps each category to a fixed score: equal 1.0, approximately_equal 0.9, superset 0.8, subset 0.6, disagreement 0.0. Classification is a task language models are decent at. Calibrated numeric scoring is a task they are bad at. Converting the second into the first is the kind of small decision that determines whether a grader is trustworthy.

Second, the scorers do not call a provider directly. They go through two single method drivers, JudgeDriver and EmbeddingsDriver, and a custom scorer declares RequiresJudge or RequiresEmbeddings if it needs one. Unmarked scorers are treated as deterministic and always run. So the expensive path is opt in by construction, and laravel/ai is a default rather than a dependency. You can hand it a closure that returns a canned score and exercise the whole scoring path without spending money.

The cost objection that kept evals out of CI gets a boring answer: evals skip on a normal run and only execute under --evals or PEST_EVALS=1. There is also repeat(3), which samples the same prompt several times and requires every sample to pass, which is the minimum honest response to nondeterminism.

Now the part I would be careful with.

toFollowTrajectory() asserts that an agent invoked a sequence of tools in the expected order. It is deterministic, it is useful, and it is grading the route rather than the result.

expect(SupportAgent::class)
    ->prompt('I want to return my order and get a refund.')
    ->toFollowTrajectory(['lookup_order', 'create_return', 'issue_refund']);

Route checks are regression tools, not quality tools. They tell you that behavior changed. They do not tell you that behavior got worse. A model that gets better and reaches the same outcome by a shorter path fails this assertion, and the failure is correct as a signal and wrong as a judgment. If you treat a red trajectory assertion as "the agent broke," you will spend the next year pinning your system to the route a July 2026 model happened to take.

I would encode trajectories only where the sequence is itself the requirement, which is to say where an ordering constraint is a real business rule: check inventory before charging the card, verify identity before disclosing an account, take a backup before a destructive migration. Those are not routes. Those are invariants that happen to look like routes.

What I am doing in my factory

Two suites, two purposes, never merged. Selection evals live outside the application and are allowed to be generic, because comparability is their whole value. Acceptance evals live in tests/Evals and are allowed to be parochial, because fixture proximity is theirs. When I catch myself wanting a fixture in a selection eval, that is the signal it has become an acceptance eval and should move.

Acceptance evals gate promotion. Selection evals never do. A selection eval informs a decision I make. An acceptance eval is a verification gate that fails closed. Confusing the two produces a suite that blocks deploys for reasons nobody can act on.

Deterministic first, scored only where deterministic cannot reach. Every scored check costs money, adds variance, and introduces a second model whose failure modes I now own. toHaveToolCalls() before toSatisfy(), every time.

Ordering constraints, not trajectories. I encode a sequence only when the sequence is a business rule. Otherwise I assert the outcome and let the route move.

Re-baseline on any config change. A score without its run conditions is close to meaningless. Model, prompt, harness, and budget are all part of the configuration under test, which is the one place I think smevals has the better vocabulary: making config an explicit dimension is a good idea that Pest's ergonomics quietly hide.

What to ask about your eval suite

  1. For each eval you have, is it answering "which configuration" or "does this ship"? If you cannot tell, it is probably doing neither well.
  2. Which of your evals block a deploy, and could the person they block act on the failure?
  3. How many of your checks need a model to run? What would it take to move one of them down to deterministic?
  4. When your provider ships a new model next month, which suite tells you whether to switch, and which tells you whether you are still safe?
  5. Are you asserting outcomes, or are you asserting the path a model took in July 2026?

The eval layer spent two years organizing itself as a product category. What it looks like from here is a split: one half becomes a comparison instrument that lives outside your system, and the other half becomes assertions in the suite you already run. The teams that get hurt are the ones still running a single suite and wondering why it keeps telling them nothing.

Research and source trail for this area live in the Verification, evaluation & quality truth subsystem, and the related Feedback, learning & controlled self-improvement subsystem. If your team is moving from AI-assisted coding toward autonomous production, Agency Intentional works on the harness, verification, and operating boundaries that make that transition safe enough to use.