Schema-validated planning sharply improves reproducibility in a deterministic harness

Dhage wrapped two open models, Qwen-2.5-7B and Gemma-3-27B, in a harness of finite-state execution, forced tool selection, output validation and bounded retry, then ran each against two synthetic pipelines 100 times per condition. Against a plain, unconstrained baseline, the harness significantly improved exact-match reproducibility in one of the four model-task combinations, significantly degraded it in two, and had no measurable effect on the fourth, even though task success held steady at 97 to 100 percent throughout. Tracing the result layer by layer showed why: tool order, state sequence and output were already close to fully consistent, so the model's free-text planning step, left unconstrained, became the only place variance could still hide. Adding a schema-validated planning gate before any tool call, which Dhage calls Structured Planning, closed the gap: three of the four cells reached a perfect Reproducibility Rate of 1.000 at 100 runs, the fourth reached 0.980 on account of a small number of genuine tool-calling failures unrelated to plan wording, task success rose to 100 percent in three of the four, and token cost fell 15 to 17 percent versus baseline across the board. The fix was not free for every model. Structured Planning made Qwen 12 to 21 percent faster than the plain harness, and Gemma 15 to 24 percent slower, a split that grew rather than shrank when the sample size was doubled to check it wasn't noise.

Why it matters: Finite-state execution and forced tool selection left free-text planning as the main place variance could hide. That made the harness score worse than the unconstrained baseline in two tests. Schema validation cost little in tokens, but its effect on speed depended on which model sat underneath it, so teams should measure that tradeoff before shipping.