Every grep burns context tokens. LSP returns the answer with zero tokens burned.

That’s the whole argument. The rest is just evidence.

When an AI agent searches for a function definition by grepping across the codebase, it reads dozens of files, pipes back hundreds of lines, and stuffs all of it into context — most of which is noise. The token cost is real. The context degradation is real. And the answer is often wrong anyway, because grep matches strings, not code.

LSP and ast-grep solve different halves of this problem. LSP eliminates the token waste. ast-grep eliminates the false positives. Together they give AI agents a navigation stack that’s faster, cheaper, and more accurate than text search.

Why large context hurts more than it helps

The intuition that “more context = better answers” is wrong.

Liu et al. (2023) documented the Lost-in-the-Middle effect: when relevant information is buried in the middle of a long context window, accuracy drops by 20+ percentage points compared to when it appears at the start or end. The model attends to the edges.

More recent benchmarks make it worse. NoLiMa found that 11 of 12 tested models fell below 50% baseline accuracy at just 32K tokens when irrelevant context was present — even when the relevant information was technically retrievable. Context length itself degraded performance 13.9–85%, not just the model’s ability to find things.

NoisyBench reported GPT-4o dropping from 99.3% to 69.7% accuracy under distractor conditions. The distractors in a code navigation context are the grep matches that aren’t the symbol you’re looking for.

The fix isn’t a bigger context window. Selective context — keeping only 4.5–8.7% of tokens — maintains precision performance. The goal is to send the model the right tokens, not more tokens.

Agentic workflows are especially exposed. Increased reasoning steps make things worse in noisy contexts. Every tool call that dumps unneeded file content into the context compounds the problem across the entire session.

The token math

Consider finding all references to a function called Close in a real codebase.

On Consul (319K lines of code), grep -r "Close" returns 1,156 matches. Twelve of them are actual references to the symbol you care about. That’s a 99% false-positive rate. Every one of those 1,156 matches gets read and shoved into context.

LSP findReferences on the same symbol: 12 results, 50ms, zero false positives. It runs in the language server, which has already parsed and type-checked the code. It returns semantic references, not string matches.

The measured token savings are 5–34x depending on project size. For symbol renaming — which requires finding every reference before changing anything — the savings reach 92–1,441x. A 500-line file read costs every token in those 500 lines. A goToDefinition call costs the single location it returns.

Speed matters too. Grep on a large codebase runs 30–60 seconds. LSP runs 50ms. That’s not a marginal improvement — it’s a different order of magnitude.

Claude Code already has this

You don’t need to configure anything. Claude Code has native LSP support as of v2.0.74, exposing:

  • goToDefinition — jump to where a symbol is declared
  • findReferences — all call sites and usages
  • hover — type signatures and docs without reading the file
  • documentSymbol — all symbols in a file
  • getDiagnostics — type errors reported by the language server

The Language Server Protocol spec is what makes this work across languages — the same protocol that powers VS Code intelligence powers Claude Code’s navigation. When you ask Claude Code to find a function, it can call goToDefinition directly rather than reading files to search for text.

The practical implication: any task that starts with “find where X is defined” or “find all callers of Y” should use LSP, not grep. Grep is for text in configs, comments, and docs — content that doesn’t have an LSP server.

Structural search: what grep cannot express

grep matches text. Some queries can’t be expressed as text patterns at all.

Consider: “find all await expressions that aren’t inside a try/catch block.” There is no regex for this. The syntactic relationship between the await and its enclosing context is structural, not textual.

ast-grep searches code as a syntax tree. It uses tree-sitter grammars to parse source into an AST, then matches patterns against the tree’s structure. The wildcard syntax is simple: $VAR matches a single AST node, $$$ARGS matches zero or more.

The await without error handling rule is a valid, runnable ast-grep rule file:

id: await-without-try-catch
language: TypeScript
rule:
  pattern: await $PROMISE
  not:
    inside:
      kind: try_statement

Run it with sg scan --rule await-without-try-catch.yaml. Every match is an actual structural violation — not a text coincidence.

Relational rules extend this further. inside, has, follows, and precedes let you express queries like “a function call that follows a null check” or “an if statement that has no else branch.” grep cannot express either of these.

The precision difference is measurable. On a fetchUserData() search, grep returns 6 matches; ast-grep returns 1 — the actual call site, not variable declarations, comments, or function definitions that happen to contain the string. That’s roughly 45% false-positive rate for grep vs 0% for ast-grep on the same query.

ast-grep is slower than grep — 1.2s vs 0.5s on 100K LOC — because parsing costs more than string matching. The tradeoff is precision, not speed. When you need structural queries, there’s no text-based alternative.

The accuracy case

Token economics is the lead, but accuracy compounds the benefit.

Gorilla (Patil et al., NeurIPS 2024) showed that tool use produces an 83% improvement over base LLaMA on API-heavy tasks and mitigates hallucination endemic to GPT-4 operating from memory alone. ReAct (Yao et al., 2022) established that tool use overcomes “hallucination and error propagation prevalent in chain-of-thought reasoning alone.”

The mechanism is straightforward. When an agent reads a stale grep match and reasons from it, it reasons from wrong data. When it calls goToDefinition, it gets the current location — whatever the source file says right now. The tool grounds the reasoning.

SWEzze measured a 51.8–71.3% token reduction while improving issue resolution 5–9.2% on SWE-bench by tightening the tool context agents receive. Precise, line-level code context produces F1 > 99%; cluttered context drops that substantially. AST-guided code generation produced +40% accuracy over text-based baselines.

The pattern across all of it: smaller, precise context outperforms large, approximate context. Tools that return precise results — LSP for symbol navigation, ast-grep for structural patterns — are the mechanism for achieving that precision without asking the model to sift through noise.

When to use which tool

Task Use
Find where a function is defined LSP goToDefinition
Find all callers of a function LSP findReferences
Find all usages of a variable LSP findReferences
Get a type signature without reading the file LSP hover
Find await without try/catch ast-grep structural rule
Find function calls with a specific argument shape ast-grep pattern
Detect deprecated API patterns with wildcards ast-grep
Search for a string in comments or config grep
Find which files mention a term grep

grep remains useful for non-code content — markdown, YAML, config files — where there’s no LSP server and no AST grammar. For anything that’s source code, start with the tool that understands code structure.

The session-level effect accumulates. Every LSP call instead of a grep-and-read keeps context tighter. Tighter context means the model attends to what matters. Smaller, cleaner context windows produce better answers, not just cheaper ones.