In this guide
Retrieval-augmented generation was built for documents: split text into chunks, embed each chunk as a vector, find the chunks closest to a question, and give them to a model. It works well for knowledge bases, support articles, and policy documents. Applied to source code, the same recipe produces answers that are often close and sometimes confidently wrong, and the reasons are structural rather than a matter of tuning.
This post looks at exactly where chunk-and-embed retrieval breaks on code, why, and what to add so retrieval becomes reliable enough for engineering work. The short version: code has explicit structure that similarity search ignores, and the fix is to retrieve over that structure as well as over text.
Key takeaways
- Fixed-size chunking cuts through functions and separates code from the context that gives it meaning.
- Embedding similarity finds code that looks related, not code that is related. Callers, implementations, and configuration rarely look like what they affect.
- Identifiers carry precise meaning that embeddings blur. Exact symbol lookup is a different and necessary operation.
- Retrieval quality on code depends on knowing relationships: definitions, calls, imports, inheritance. A code graph captures those directly.
- A practical system combines exact search, structural queries, and semantic search, and returns results with source anchors and freshness information.
How chunk-and-embed works on code
A typical code RAG pipeline:
- Walk the repository and read each file.
- Split files into chunks, by line count, token count, or roughly by blank lines.
- Embed each chunk with a general or code-tuned embedding model.
- Store vectors in an index.
- At query time, embed the question, retrieve the top matching chunks, and include them in the prompt.
For "how do we validate email addresses", this often works. Somewhere there is a chunk with validateEmail and a regex, and it is semantically close to the question. The trouble starts with the questions engineers actually ask during real work.
Where it breaks
1. Chunks cut through meaning
A function split across two chunks loses its signature in one and its return logic in the other. A method separated from its class loses the fields it reads. A chunk of a 600-line file may contain the end of one function and the start of another, matching a query because of the wrong one.
Syntax-aware chunking (splitting at function and class boundaries using a parser) improves this, and if you build a code RAG pipeline at all, you should do it. But even perfect chunks are isolated: a function chunk does not contain the types it uses, the configuration it reads, or the functions it calls.
2. Similar is not related
Embedding search ranks by resemblance. Code relationships rarely resemble each other.
| Question | What is related | Does it look similar? |
|---|---|---|
Who calls chargeCard? | A checkout handler, a retry job, an admin refund tool | Not really; each is about its own concern |
What implements PaymentProvider? | StripeProvider, AdyenProvider | Somewhat, but so does every payment-related file |
What breaks if OrderStatus changes? | Serializers, state machine, reports, UI badges | Only loosely |
| Where is this feature flag read? | One line in a config helper, one in a component | No |
The answers are defined by edges in the code (calls, imports, implements), not by textual similarity. A similarity search might return the top five "payment-ish" chunks and miss the retry job entirely because it is mostly about queues.
3. Identifiers get blurred
To a developer, getUser, getUsers, fetchUser, and getUserById are four different functions with different contracts. To an embedding model, they are near neighbors. Ask about getUserById and you may get back getUser with high confidence. The model then writes code against the wrong signature, which is one root cause of agents hallucinating APIs.
Exact identifier lookup is not a weaker form of semantic search. It is a different operation, and code needs both.
4. Top-k is not completeness
Many engineering questions need every result, not the best few. "Find all callers before I change this signature" is useless at 80 percent recall. Similarity search returns a ranked top-k with no notion of "that is all of them". A graph query returns the full set of known edges, and can say when the set is incomplete.
5. Indirection hides relationships
Dependency injection containers, event buses, route tables, string-keyed registries, and generated clients all connect code without a direct reference. Neither embeddings nor plain text search follow these. A structural index can model some of them explicitly (route tables and registries are often parseable), and at least flag the rest.
6. Staleness is invisible
Embeddings computed last week do not know about this morning's rename. Re-embedding a large repo is not free, so indexes lag. Unless results carry the revision they came from, the model cannot tell that a chunk describes code that no longer exists.
7. Context is spent on near misses
Each retrieved chunk costs context. If half the top-ten chunks are lookalikes, half the budget goes to text that does not influence the answer, or worse, misleads it.
What to add
None of this means embeddings are useless for code. They are good at fuzzy, intent-level queries: "where do we handle rate limiting", "code that formats currency". The fix is to add the retrieval modes code needs, and route each question to the right one.
Exact and lexical search
For identifiers, error messages, config keys, and string literals, use exact or regex search over the source, ideally symbol-aware so it can distinguish definitions from mentions. This is the baseline every code tool needs.
A code graph
Parse the repository into a graph of symbols and relationships: definitions, calls, imports, inheritance, implementations, and where possible routes, jobs, and registry entries. Queries then become graph traversals:
find_symbol("chargeCard") -> definition at payments/charge.ts:31
trace_calls("chargeCard", inbound) -> checkout/handler.ts:88, jobs/retry.ts:14, admin/refunds.ts:52
get_dependents("payments/charge.ts") -> 6 modules, 4 test files
These answers are complete for statically visible relationships, precise, and cheap. Our complete guide to code knowledge graphs covers how they are built and queried.
Semantic search over well-formed units
Keep embeddings, but embed meaningful units: whole functions, classes, and modules, each with its signature, docstring, and file path. Use them for intent queries and as an entry point into the graph.
Hybrid routing
The strongest pattern is to use semantic or lexical search to find a starting point, then expand structurally:
- "Where do we charge cards?" -> semantic search finds
chargeCard. - Expand: its callers, its callees, and the types it uses.
- Return a bounded neighborhood: the function, its signature, its direct callers, and relevant tests, with anchors.
This gives the model connected context instead of a scattering of lookalikes.
Anchors and metadata
Every result should include file path, line range, symbol name, and the revision indexed. Say when results were truncated. This lets the model, and the human checking its work, verify claims and notice staleness.
Freshness
Re-index incrementally on file change or commit, and expose the indexed revision. A graph built from a parser is usually cheaper to update incrementally than re-embedding, because only changed files need re-parsing.
Comparing approaches
| Need | Chunk + embed | Lexical search | Code graph | Hybrid |
|---|---|---|---|---|
| Intent query ("where do we...") | Good | Weak | Weak alone | Good |
| Exact identifier | Weak | Good | Good | Good |
| All callers / dependents | Poor | Partial | Good | Good |
| Implementations of an interface | Partial | Partial | Good | Good |
| Config keys, strings | Partial | Good | Not modeled | Good |
| Bounded, connected context | Poor | Poor | Good | Good |
| Source anchors | Possible | Good | Good | Good |
The deeper comparison is in code knowledge graph vs embeddings.
How to evaluate your retrieval
Before trusting a code retrieval system in an agent workflow, test it on questions with known answers from your own repository.
1. Find the definition of <ambiguous identifier>. Expect: exactly one, correct.
2. List all callers of <widely used function>. Expect: complete vs IDE "find references".
3. What implements <interface>? Expect: all implementations.
4. Where is <config key> read? Expect: every read site.
5. How does <feature> work end to end? Expect: entry, core logic, storage, tests.
6. After renaming a function, query the old name. Expect: no stale results.
Score recall (did it find all of them), precision (how many results were noise), and grounding (are anchors correct). Question 2 is the one most chunk-and-embed systems fail, and it is the one that matters most before changes.
What this means for agents
Coding agents are retrieval systems with a loop. When their only retrieval is similarity search, they compensate by reading many files, which costs tokens and still misses indirect relationships. Giving them structural queries through MCP changes the loop: find the symbol, trace its callers, read only the anchored ranges, and check dependents before finishing. The token side of this is covered in reducing AI agent token costs.
Where OpenVisio fits
OpenVisio takes the graph-plus-search approach. It parses repositories with tree-sitter, locally in the browser or on your machine, and exposes the result to agents over MCP with find_symbol (by name, pattern, or description), search_code for literal and regex search, and trace_calls, get_dependents, and get_neighborhood for structure, all returning anchored results.
Chunk-and-embed retrieval is a reasonable starting point for code search. It is not enough for work where missing one caller means a broken release. Add exact search for identifiers, a graph for relationships, and anchors on everything, and retrieval stops being the weak link.
