---
title: 'In the News: November 17, 2026 (Extra 2)'
description: "Dan Luu's $90k eval suggests test instructions now help GPT-6 Astra, and the Claude Code team explains how it keeps its checks honest."
canonical_url: 'https://darkfactory.dev/news/2026-11-17-extra-2'
markdown_url: 'https://darkfactory.dev/news/2026-11-17-extra-2.md'
collection: news
date_published: '2026-11-17T16:03:00-05:00'
date_modified: '2026-11-17T16:03:00-05:00'
---

# In the News: November 17, 2026 (Extra 2)


Dan Luu reran his test-technique eval on GPT-6 Astra and, unlike in August, found several techniques beating no instruction at all, though he says the sample is too small to tell which ones. Anthropic's Claude Code team published the rules it uses to keep agent checks out of reach of the changes they judge. Anthropic also cut its vulnerability disclosure window to 30 days for serious bugs.

## 1. Telling agents how to test now seems to help, on a harder task

**[Test/verification technique effectiveness with Astra](https://danluu.com/astra-testing/)** · Dan Luu · danluu.com, November 16, 2026

In August, Luu found that asking GPT-5.6 Sol to use a named technique such as TDD or QuickCheck mostly made results worse than giving no testing instruction. He reran the eval with GPT-6 Astra. The original task, implementing Zstd from its RFC, is now saturated: agents spent $1 at medium effort to pass every holdout test over 99% of the time, where 5.6 Sol spent $3.50 to do it about 63% of the time. On a harder task, implementing Pandoc over roughly 14 turns, he ran each condition 25 times at a list-price cost of about $90k. There, "many different techniques have a better result than Default."

He is careful about which ones. The error bars are wide, and he writes that "we don't really have enough information to say why almost anything happened." His tentative notes: differential testing led agents to examine boundary conditions closely, TDD underperformed again, the ACL2 theorem prover did well, and property-based testing libraries such as proptest finally produced tests that do what the technique promises. He also disputes a viral claim that AI-written tests are "empirically proven to be unhelpful", which he says rests on a 9% average cost saving on a benchmark of small problems whose grader penalizes added tests.

**Why it matters:** A testing instruction that hurt on one model and task size may help on the next, so treat a test-technique prompt or skill as something to re-measure when you change models. Luu's own testing skill outperformed with 5.6 Sol and underperformed with Astra on Pandoc.

## 2. The Claude Code team keeps its checks out of reach of the changes they judge

**[How the Claude Code team thinks about verification](https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/)** · Addy Osmani · claude.dev (Anthropic), November 16, 2026

Osmani writes that about 90% of the team's code is written and reviewed by Claude, and describes the gates that make that workable. Code Review reads REVIEW.md from the base branch, never from the pull request, because "If a change can modify the check that judges it, the check is broken." A 39-item reliability checklist gives each item a trigger that can be read from a diff, and Claude checks only the matching items, usually 3 to 10. If a check still fails after three rounds of fixes, Claude stops and brings in a human. A commit gate that can block commits whose session lacked /code-review or /verify has off, observe and enforce modes, and new gates start in observe. Claude may not skip a flaky test, add sleeps or retry it until it passes; if it cannot find the root cause, it deletes the test and writes down why.

The post gives one figure: in internal testing on structured file generation, a separate grader checking work against a rubric improved task success "by up to 10 points" over a standard prompting loop. That is Anthropic's number for its own tool, and the post does not publish the underlying data.

**Why it matters:** The cheapest rules to copy are loading gate rules from a branch the change cannot touch, triggering reliability checks from the diff, and running any new gate in observe mode before it blocks. Osmani names end-to-end, property and differential tests as the ones worth investing in; property and differential testing both did well in Luu's eval above.

## 3. Anthropic cuts its disclosure window to 30 days for serious bugs

**[Updating our coordinated vulnerability disclosure policy timelines](https://www.anthropic.com/research/coordinated-vulnerability-disclosure-timelines)** · Anthropic Frontier Red Team · November 16, 2026

For vulnerabilities Anthropic reports on or after January 1, 2027, the disclosure window drops from 90 days to 30 for high- and critical-severity issues and to 45 for all others. Escalation to an external coordinator when a maintainer does not respond moves from 30 days to 15, and full technical details follow 30 days after a patch instead of 45. Older high- and critical-severity reports already past the 90-day deadline get another 30 days before publication. The stated reason is that "a bug that one model can find is increasingly likely to be found by another," and Anthropic says it is seeing more rediscovery. The seven-day window for actively exploited bugs is unchanged, and the policy does not apply to AI-discovered findings that have not had human review.

**Why it matters:** If you maintain software that Anthropic's models scan, plan for a 30-day fix cycle on serious reports from January. The stated reason, faster rediscovery by models, is not specific to Anthropic's reports.
