---
title: 'In the News: November 19, 2026 (Morning)'
description: 'A Genentech study finds agents answer from results they were warned were wrong, and only a harness that withholds them fixes it.'
canonical_url: 'https://darkfactory.dev/news/2026-11-19-morning'
markdown_url: 'https://darkfactory.dev/news/2026-11-19-morning.md'
collection: news
date_published: '2026-11-19T07:05:00-05:00'
date_modified: '2026-11-19T07:05:00-05:00'
---

# In the News: November 19, 2026 (Morning)


If your harness flags a bad tool result and leaves the decision to the model, a new preprint from Genentech and Virginia Tech says the model will usually use the bad result anyway. Kiro explains how a new model kept a normal benchmark pass rate while 40% of its tool calls failed. A Singapore lab proposes recording whether each agent action can be undone, not only whether it is allowed.

## 1. Agents see the warning and report the wrong number anyway

**[Invariant-Aware Harness Enforcement to Protect LLM Agents from Silent API Failures](https://arxiv.org/abs/2611.20595)** · Sajib Acharjee Dip (Virginia Tech, Genentech), Brian Ouyang and Lala Motlhabi (Genentech) · arXiv, November 16, 2026

The paper studies APIs that return HTTP 200 with a plausible but wrong result, for example by ignoring a misnamed filter. The authors found this in 11 of 15 public research APIs, with counts inflated up to 691 times. On their count benchmark, an unguarded agent reported a wrong number in 74.0% of faulted runs across seven open models. Checklist prompts, a reviewer model and Outcome Monitors barely moved that (72.3%, 73.8% and 72.2%). The authors' explanation: "models often see the problem and answer anyway." A judge shown the check's results flagged 82.4% of faulted runs, and agents still reported the flagged count in 77.4% of those runs.

Their fix, SilentGuard, runs invariant checks in the harness. One example: the count with a condition plus the count with its negation must equal the count without it. When an exact check fails twice, the harness withholds the value. That cut wrong answers to 1.7%, with no loss on fault-free tasks, and from 70.2% to 0.5% on three frontier models. "A warning from the same check leaves 70.1% wrong, so enforcement is the key." The authors are clear about the limits. The harness mostly abstains rather than recovering. On fault types written after the method was frozen, wrong answers fell only from 61.1% to 24.5%. The frontier-model logs were not released under company policy. Code and tasks are [on GitHub](https://github.com/Sajib-006/silentguard).

**Why it matters:** Telling a model that a result looks wrong did almost nothing in these tests; taking the value away did. If your harness surfaces failed checks as warnings for the agent to weigh, this is evidence for making it block instead, and for giving the agent an explicit "cannot be determined" answer.

## 2. Kiro's benchmark passed while 40% of tool calls failed

**[Lessons learned from one million evaluations](https://kiro.dev/blog/1m-evals-lessons/)** · Mac Merritt, Kiro · kiro.dev, November 17, 2026

When Kiro added a new model, its Terminal Bench 2.1 pass rate matched expectations. Internal users still reported a worse experience. Merritt writes that "there were tools failing on 40% of their calls because the tool description that had worked for other models caused the new model to call the tools with invalid inputs." The model kept retrying until it got the call right, so the cost appeared only in tokens and latency. Kiro now scores every benchmark run in three layers: pass rate, efficiency signals such as latency, tokens and tool-failure rate, and LLM-as-judge ratings for code quality and tone. It matches the suite to aggregated internal usage across 20+ task categories and 25+ domains, and checks it against surveys and user anecdotes.

The post also describes a scenario Kiro nearly added. An internal pull request had been merged and rated 5 stars, but the next commit fixed bugs it introduced, so the benchmark would have scored buggy code as correct. The post gives no figures for the improvements it attributes to the new suite.

**Why it matters:** A saturated pass rate can hide a model and harness mismatch that users notice. If you swap models behind a fixed set of tool descriptions, tool-failure rate per model is a cheap number to start tracking.

## 3. A proposal to label agent tool calls by whether they can be undone

**[Can You Undo It? Reversibility Classes for AI Agent Actions](https://arxiv.org/abs/2611.22075)** · Deepak Waghmare and Shaad Sufi, Two Much Labs · arXiv, November 17, 2026

The paper starts from the observation that "permission to call a tool does not establish whether a mistake can be undone." It defines five classes, R0 to R4, with argument-sensitive rules. Unrecognised calls default to R4, and intended delivery of content to outsiders is R4 even when the message can be deleted. The authors publish a registry covering 536 tools across 20 MCP servers and a classifier whose default hook asks for approval on R2 to R4. Replaying 2,158 recorded agent runs, its session guard gated the action in all 29 attack-positive runs it could attribute, against 18 for a read/write proxy. It added nine approval points across 1,209 benign runs. Mapping native MCP annotations disagreed with the draft registry labels for 118 of 481 comparable tools. The authors say this "does not establish registry-wide accuracy, reduced production harm or general recovery reliability." The registry entries are still drafts, and the one review was AI-assisted. The specification and classifier are [on GitHub](https://github.com/TwoMuchLabs/reversibility-classes).

**Why it matters:** Most permission setups ask whether an agent may call a tool. This gives you a second question to encode, whether the result can be reversed, and an open registry to start from. By the authors' own account it is a vocabulary to test, not a proven safeguard.
