---
title: 'In the News: November 21, 2026 (Extra)'
description: 'An Ant Group code-search study finds much of the embedding lead over keyword search comes from how benchmarks are set up.'
canonical_url: 'https://darkfactory.dev/news/2026-11-21-extra'
markdown_url: 'https://darkfactory.dev/news/2026-11-21-extra.md'
collection: news
date_published: '2026-11-21T10:01:00-05:00'
date_modified: '2026-11-21T10:01:00-05:00'
---

# In the News: November 21, 2026 (Extra)


Code-search benchmarks show dense encoders beating keyword search by a wide margin, and teams build embedding indexes for coding agents on the strength of that gap. A new paper from Ant Group reruns that comparison against a fairly tuned keyword baseline and finds much of the margin goes away. In an agent, the model mattered far more than the search tool. LaunchDarkly also describes how one team is handling nearly twice the pull request reviews it did in early summer.

## 1. Much of the embedding lead in code search comes from how benchmarks are set up

**[Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search](https://arxiv.org/abs/2611.20455)** · Siqiao Xue, Ziyin Zhang, Zihan Liao, Jin Qin, Yixiang Mu and Hang Yu, Ant Group · arXiv, submitted November 16, 2026

The authors built RepoPremium, a file-ranking benchmark whose core has 1,265 queries over ten SWE-bench Verified repositories. It pairs the cleaned-up issue statements SWE-bench uses with raw pull request text from 2022, 2023 and 2026, and adds a preregistered held-out pool of 539 queries over 91 repositories. They compared thirteen dense encoders against BM25. Two setup choices inflated the dense lead. Giving BM25 whole files instead of the encoder's 1,000-character chunks cut the lead on the SWE-bench issue statements from +0.121 to +0.034 nDCG@10. On raw pull request text, the lead after the same change was +0.059 to +0.125. Counting edited test files as relevant raised that lead from +0.034 to +0.165.

They then had an LLM write likely identifiers, APIs and error terms into the query before a BM25 search, which they call generate-then-grep. That added 0.11 to 0.19 nDCG@10 to BM25 and left it ahead of the canonical encoder in every core Python pool by point estimate, significantly so only on the SWE-bench statements (by 0.081). BM25 beat dense retrieval outright on 24% of the SWE-bench statements and 21% of the 2026 pull request queries, but the authors write that "No method we tried predicts which retriever will win on a given query." Of about 30 routers they built, the one they froze and scored on the held-out pool missed its preregistered criterion. A fixed fusion of the two retrievers beat dense retrieval on the held-out pool. In a SWE-agent on 96 held-out tasks, swapping between a shell with grep, a lexical index, a dense index or a router moved file-localization exact-match by about 0.04. Across four setups using three LLMs on a 30-task subset, it ranged from 0.03 to 0.30. No tests were run, so the agent result measures whether it found the right files, not whether it fixed the bug. Most of the work is in Python, the cross-language pools have one repository per language, and BM25's settings were tuned on the evaluation queries.

**Why it matters:** If your case for an embedding index rests on a published code-search gap, check whether that benchmark gave BM25 whole files and left test files out of the answer key. The authors' advice for agents is direct: "spend on the model before the search endpoint." Where a dense index is already running, fusing it with whole-file BM25 was the most robust choice they tested. The queries, labels and harnesses are released, so you can rerun the comparison on your own repositories.

---

## Also this cycle

- **[An expedition for expeditious PR reviews](https://launchdarkly.com/blog/an-expedition-for-expeditious-pr-reviews/)** · David Brackbill, LaunchDarkly, November 6, 2026 · LaunchDarkly's Agent Platform team submitted nearly twice as many PR reviews from late September to late October as from late May to mid-June, which Brackbill links to coding agents writing more code. Auditing two weeks of PRs in the company's main repository, he counted an average of 13 notifications per approved PR, mostly CODEOWNERS pings for cross-cutting changes. The team now has agents stage review comments in a pending GitHub review, asks them to keep comments within 300 characters, and uses a round-robin for CODEOWNER duty so only one or two people get each ping.
