Richer LSP output reversed coding agents' preference for grep

Xu tested three Claude models against grep and LSP-backed code navigation, across Python and TypeScript repositories. On simple code-location tasks the models chose the semantic tool only 0 to 6 percent of the time when both were available, and forcing them onto the LSP path first dropped task success from 100 to 89 percent. The result flipped for reference-completeness tasks, where models reached for LSP 45 to 57 percent of the time on their own and its precision beat grep's, 1.00 against 0.76. The sharpest finding came from a small change to the tool itself: when Xu made the LSP tool return a line of source code with each result instead of a bare file and line number, pass rate on multi-file rename tasks rose from 0.67 to 0.83, and follow-up file reads fell from 15.2 to 3.2 per episode. Several Hacker News commenters flagged the post's prose as apparently AI-polished; Xu replied in the thread that he is a non-native English speaker who used AI to edit a manually researched draft. The full experimental data is public on GitHub.

Why it matters: A tool's output shape can matter as much as its underlying capability. Before swapping grep for something more sophisticated in a harness, check whether the replacement returns enough context for the model to act without a follow-up read.