---
title: 'Held-out set'
description: 'Examples kept separate from training and development decisions so evaluation measures generalization to data the workflow has not optimized against.'
canonical_url: 'https://darkfactory.dev/glossary/held-out-set'
markdown_url: 'https://darkfactory.dev/glossary/held-out-set.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Held-out set


## Definition

A held-out set contains examples kept separate from training and from the decisions used to improve a system. Its purpose is to estimate performance on data the model, prompt, workflow, and development team have not optimized against.

Separation is broader than leaving rows out of model training. If developers repeatedly inspect failures, tune prompts to the score, copy answers into retrieval, or let an agent read reference labels through a tool, the set is no longer a clean holdout for those decisions. Google notes that even test and validation sets "wear out" through repeated use. A fresh or private set is then needed for an unbiased final check.

A holdout must also represent the intended use. Perfect isolation does not rescue a set with label errors, duplicates, stale policy, or a distribution unrelated to production. Document how examples were split, deduplicated, accessed, and refreshed.

## Distinguish it from nearby terms

- A validation set is used during model or workflow selection. A test set is reserved for final evaluation. Both should be held out from training, but only the test set remains clean from tuning if the distinction is maintained.
- A hidden set is not visible to participants. A visible set can still be held out from model training, but repeated optimization against it compromises the estimate.
- A golden set is curated for trusted reference outcomes. It may or may not be held out from development.

## Check your understanding

An agent never receives the held-out inputs, but its developer sees the score after every prompt revision. What has leaked, and what evaluation design would restore a clean final estimate?

## Also called

holdout set

## Related factory areas

- [Verification, evaluation & quality truth](https://darkfactory.dev/factory/verification)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [SpecBench: the reward-hacking gap grows with codebase size](https://arxiv.org/abs/2605.21384)
- [GPT-4 Technical Report](https://cdn.openai.com/papers/gpt-4.pdf)
- [Where Does Agent Reliability Come From?](https://arxiv.org/abs/2607.17044)