Definition
Generalization is performance beyond the examples used to fit a model or tune a system. The narrowest test uses held-out examples drawn from the same process as the training data. Harder tests change time period, population, task wording, tools, budgets, or environment. A model can generalize well under the first test and fail under the second.
Training loss alone cannot demonstrate generalization. Evaluation needs data that did not influence parameter updates, prompt selection, hyperparameter tuning, or benchmark-specific engineering. Leakage and repeated benchmark optimization weaken that separation even when the examples were not literally included in a training file.
For agent systems, the unit being tested is larger than the model. Prompts, retrieval, tool schemas, retry policies, and verification gates can improve or damage behavior on new tasks. Dark Factory therefore treats generalization as a property of the evaluated model-and-harness configuration under a stated operating regime, not as a permanent score attached to one checkpoint.
SpecBench provides a concrete software-factory warning: apparent task satisfaction can diverge from real specification compliance as codebases grow. Passing small or familiar repositories does not establish generalization to larger systems with more opportunities for reward hacking.
Distinguish it from nearby terms
Generalization describes transfer beyond fitted cases. Memorization reproduces fitted patterns. Robustness concerns performance under specified perturbations or failures. A benchmark samples one evaluation regime and cannot by itself establish broad transfer.
Check your understanding
An agent passes new issues from repositories used during harness design but fails on unfamiliar build systems and larger codebases. Which result is in-distribution performance, and what new evaluation would support a stronger generalization claim?