A small eval framework lets a coding agent build its own tests

Prime Radiant built smevals to answer a narrower question than most benchmarks ask: not "which model is best," but "which cheap model is good enough for this specific task." The post opens with the pricing case for asking that question at all: GPT-5.5 and GPT-5.6 Sol cost twice what GPT-5.4 did, Claude Fable 5 costs twice Claude Opus 4.8, and even Gemini's cheap Flash-Lite tier has gone up in price since 3.1, while open-weight small models keep improving.

smevals is a Python CLI whose documentation targets agents and people: point a coding agent at uvx smevals docs and it has enough to construct an eval on its own. An eval is a directory of YAML and small scripts, built from tasks (a specific challenge, such as "write a haiku about a pelican"), configs (which model and parameters to test), and graders (checks that score the output, from a simple string match up to a script that asks another model to judge quality). Running uvx smevals run . -g executes the tasks against one or more model configs and grades the results; a bundled web app, or a static build via uvx smevals build ., lets you compare configs side by side. The framework separates running from grading on purpose, so a grader can be rewritten and re-applied with smevals grade . --regrade without re-running the model calls.

Why it matters: Most "just use a cheaper model" arguments are made with someone else's benchmark. smevals is a first-party, reproducible answer to that: an agent can read one README and build a task-specific eval in the time it takes to describe the task, which turns "is this small model good enough" from a guess into something you can actually run.