Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search · Siqiao Xue, Ziyin Zhang, Zihan Liao, Jin Qin, Yixiang Mu and Hang Yu, Ant Group · arXiv, submitted November 16, 2026
Much of the embedding lead in code search comes from how benchmarks are set up
The authors built RepoPremium, a file-ranking benchmark whose core has 1,265 queries over ten SWE-bench Verified repositories. It pairs the cleaned-up issue statements SWE-bench uses with raw pull request text from 2022, 2023 and 2026, and adds a preregistered held-out pool of 539 queries over 91 repositories. They compared thirteen dense encoders against BM25. Two setup choices inflated the dense lead. Giving BM25 whole files instead of the encoder's 1,000-character chunks cut the lead on the SWE-bench issue statements from +0.121 to +0.034 nDCG@10. On raw pull request text, the lead after the same change was +0.059 to +0.125. Counting edited test files as relevant raised that lead from +0.034 to +0.165.
They then had an LLM write likely identifiers, APIs and error terms into the query before a BM25 search, which they call generate-then-grep. That added 0.11 to 0.19 nDCG@10 to BM25 and left it ahead of the canonical encoder in every core Python pool by point estimate, significantly so only on the SWE-bench statements (by 0.081). BM25 beat dense retrieval outright on 24% of the SWE-bench statements and 21% of the 2026 pull request queries, but the authors write that "No method we tried predicts which retriever will win on a given query." Of about 30 routers they built, the one they froze and scored on the held-out pool missed its preregistered criterion. A fixed fusion of the two retrievers beat dense retrieval on the held-out pool. In a SWE-agent on 96 held-out tasks, swapping between a shell with grep, a lexical index, a dense index or a router moved file-localization exact-match by about 0.04. Across four setups using three LLMs on a 30-task subset, it ranged from 0.03 to 0.30. No tests were run, so the agent result measures whether it found the right files, not whether it fixed the bug. Most of the work is in Python, the cross-language pools have one repository per language, and BM25's settings were tuned on the evaluation queries.
Why it matters: If your case for an embedding index rests on a published code-search gap, check whether that benchmark gave BM25 whole files and left test files out of the answer key. The authors' advice for agents is direct: "spend on the model before the search endpoint." Where a dense index is already running, fusing it with whole-file BM25 was the most robust choice they tested. The queries, labels and harnesses are released, so you can rerun the comparison on your own repositories.