A compaction benchmark says what you keep matters more than how well you summarize

FutureOS, whose agent runtime is on GitHub, ran three compaction strategies through one 178-question exam: grow a session until the context is full, force a compaction, then ask for values that can only be answered from memory, with eight decoy questions to catch guessing. Its own default retained 147 of 178 (83%). Its reimplementation of OpenCode's summary-plus-recent-tail approach retained 83 (47%). Its reimplementation of Codex's approach, which keeps user messages plus a whole-history summary and drops assistant prose and tool output outright, retained 68 (38%). None fell for a decoy. The more transferable number is the composition tally behind the design: across five real session chains, tool output was 96.1% of characters, while 90% of the "what happened earlier" follow-up questions pointed at something the user or the assistant had said. The post states its own limits. The Codex and OpenCode arms are single-point reimplementations pinned to specific commits, not those products running; the real-session data cannot be published, so outside reproduction means substituting your own sessions; and there is one draw per cell, so differences of a few points are not resolvable.

Why it matters: Tool output fills the window and almost nobody asks about it later; the prose is tiny and is what gets asked about. A compaction that compresses both the same way is spending its budget on the wrong half.