Google publishes its own playbook for evaluating coding-agent harnesses

Google argues that end-to-end benchmarks such as Terminal-Bench and DeepSWE can show that a composite score changed without explaining why. The post proposes behavioral evaluations that assert on a specific tool call or file edit instead of relying only on a final pass or fail.

Its evaluation loop starts with one observed failure mode. The team then writes an assertion suited to the task's complexity, using a strict check when there is one correct path or an LLM-as-judge check when several paths can work. Finally, the team runs batches over time and watches the aggregate trend instead of blocking a change on one noisy run. A worked example for Google's Antigravity SDK checks that an agent calls a web-search tool rather than answering a weather question from memory.

Why it matters: Teams often adopt behavioral checks informally after a harness starts drifting. Google's runnable example turns that practice into a pattern they can apply directly to their own harnesses.