---
title: 'Model selection, routing & budgets'
description: 'Route by task and evidence need, and treat budget as a control variable rather than only a cost cap.'
canonical_url: 'https://darkfactory.dev/factory/model-routing-budgets'
markdown_url: 'https://darkfactory.dev/factory/model-routing-budgets.md'
collection: factory
date_published: '2026-07-16T00:00:00-04:00'
date_modified: '2026-08-09T00:00:00-04:00'
---

# Model selection, routing & budgets


**Confidence: medium.** *Evidence: controlled studies and production self-report; fast-decaying.* *Last substantive change: 2026-08.*

Model rankings change quickly. The durable work is matching tasks to models, deciding what evidence each task needs, and enforcing spend limits outside the prompt.

## The conclusion

**Route by task and by the evidence a task requires, and do not bind the factory's identity to a single frontier model.** Put model access, routing, budget policy, configuration, and trace logging behind a common control plane. Budget is a safety and control variable, not merely a cost cap: use visibility, progressive friction, and model downshifting before a hard cutoff, while retaining a stop for runaway loops. Model rankings perish quickly; routing and measurement principles are durable.

## How the thinking got here

One-model agents gave way to model specialization by role, then to dynamic routing, and then to the recognition that the harness and the infrastructure can move outcomes as much as a model upgrade does. A factory tied to one model inherits that model's failure modes and its pricing. Databricks' first-party [AI Gateway account](https://www.databricks.com/blog/managing-ai-coding-costs-scale) turns that principle into an operating pattern: centralize the model menu and cost telemetry, then route at the request or task level. Prime Radiant's [smevals](https://primeradiant.com/blog/2026/smevals.html) demonstrates a smaller, agent-operable mechanism for comparing models, prompts, and harnesses on local tasks rather than relying on a global leaderboard.

## Credible alternatives, and when each is right

| Approach | Right when |
|---|---|
| One general model | simple, uniform work |
| Fixed role-based models | roles have stable, distinct needs |
| Cascades | escalate only when a cheap model fails |
| Competitive ensembles | quality matters more than cost |
| Learned routing | enough history to train a router |
| Local or open models | bounded tasks, privacy or cost constraints |
| Progressive spend gates and downshifting | broad access matters but accidental spend must be constrained |

## Where it fails and what we still don't know

Failures include provider-correlated outages, ungraceful degradation when a model changes underneath the factory, routing against unrepresentative evals, and budget overruns that become incidents. Databricks reports more than 30% lower average task cost from its router at roughly matched quality, but that figure is internal and not independently replicated. Evidence is moderate and decays fast. Open questions include routing on accepted-outcome rather than benchmark score, uncertainty calibration, model retirement, and preventing budget overrun before it happens.

## What would change our mind

Durable, provider-independent evidence that a specific routing policy maximizes accepted outcomes per unit cost would turn routing from art into a measured discipline.

## Evidence and further reading

- [Same Signal, Different Semantics](https://arxiv.org/abs/2605.18332)
- [Infrastructure noise moves eval scores more than model margins](https://www.anthropic.com/engineering/infrastructure-noise)
- [Scale the Harness](https://arxiv.org/abs/2605.26112)
- [Token Budgets](https://arxiv.org/abs/2606.04056)
- [Managing AI Coding Costs at Scale](https://www.databricks.com/blog/managing-ai-coding-costs-scale)
- [smevals: a small eval suite for evaluating models, prompts, and harnesses](https://primeradiant.com/blog/2026/smevals.html)