---
title: 'In the News: November 16, 2026 (Extra)'
description: "Anthropic's guide to safety evals doubles as a checklist for LLM judges, and reports a 34-point score swing from configuration alone."
canonical_url: 'https://darkfactory.dev/news/2026-11-16-extra'
markdown_url: 'https://darkfactory.dev/news/2026-11-16-extra.md'
collection: news
date_published: '2026-11-16T10:05:00-05:00'
date_modified: '2026-11-16T10:05:00-05:00'
---

# In the News: November 16, 2026 (Extra)


Anthropic has written down how it builds the safety evaluations behind its deployment decisions. The subject is model safety, not code, but most of the advice is about graders, sample sizes and test configuration, and it applies to anyone who lets a model judge another model's work.

## 1. Anthropic says LLM graders fail silently, and lists how to check them

**[Building strong safety evaluations](https://www.anthropic.com/research/building-strong-safety-evaluations)** · Ruth Appel, Evan Frondorf, Jeremy Hadfield and five co-authors, Anthropic · November 13, 2026

The post borrows from psychology's replication reforms and turns them into recommendations, illustrated with examples from Anthropic's system cards. Its central warning is about LLM graders, which it says "can fail without anyone noticing": they keep tallying scores after they break, and their verdicts drift when the underlying model changes. The fixes it lists are specific. Use atomic yes/no criteria instead of one overall score. Hide the eval's design metadata from the grader. Use a different model as grader than the one under test, because graders favour their own outputs, longer answers, and whichever option they see first. Flag and exclude items the grader breaks on instead of scoring them silently. Before trusting any number, check the grader against human labels, corrected for chance agreement. In the post's words: "If the conclusions change when the grader does, you've likely measured the grader, not the model."

Anthropic's own numbers show how noisy and saturated the results can be. It says its single-turn suites on the API now score between 94% and 98% for every recent model, so they work mainly as regression checks. Confidence intervals on its multi-turn API evals can exceed ±10 percentage points. On one multi-turn scams eval on claude.ai, Claude Opus 5 and Opus 5.5 differed by 98% versus 94%, and the gap was not statistically significant. In testing Claude Fable 5.1, the appropriate-response rate on multi-turn safety evals differed by as much as 34 percentage points between the raw API and claude.ai with its system prompt applied. The post also says models can probe the boundaries of their test environments, and that "in response to our own incidents" Anthropic hardened its eval environments and paused many evaluations that give models internet access. It does not describe those incidents.

Two limits apply. The examples come from safety evals of conversations, not coding tasks, so the transfer to software verification is ours. The figures are Anthropic's own, and the post does not release the underlying eval data.

**Why it matters:** If an LLM judge sits in your verification loop, this is a working checklist for whether to trust it: binary criteria, a grader that is a different model from the one under test, loud failures, and agreement with human labels measured before the gate goes live. The 34-point configuration swing also means a score from the raw model may say little about the same model inside your harness.
