In this guide
Most comparisons of AI coding agents are run on small projects: a todo app, a fresh API, a single algorithm. Those tests say little about how an agent will behave in a five-year-old monorepo with three languages, generated clients, and a build that takes twenty minutes. The behaviors that matter at scale, such as finding the right module on the first try, respecting boundaries, and noticing the caller you forgot, only show up when the repository is large and real.
This post gives you a vendor-neutral way to choose. It lists the criteria that matter for large codebases, explains how to test each one, and ends with a two-week evaluation plan you can run on your own repository. There are no benchmark numbers here, on purpose. The only results that should drive your decision are the ones you measure on your own code.
Key takeaways
- Evaluate agents on your own repository with tasks taken from your real backlog. Public benchmarks do not reflect your layout, conventions, or build.
- The criteria that separate agents on large repos are context gathering, instruction following, verification habits, extensibility (especially MCP), and controls (permissions, sandboxing, auditability).
- Hold the setup constant: same instruction file content, same tools, same tasks, same reviewers. Otherwise you are comparing configurations, not agents.
- Score outcomes a reviewer cares about: correct files touched, tests passing, review rounds needed, and scope respected.
- Plan to re-evaluate. Agents change quickly, so keep the test set and run it again in a few months.
Why large repositories change the evaluation
On a small project, almost any capable agent can read everything relevant. The differences show up in code quality and speed. On a large project, the bottleneck moves to context: the agent cannot read everything, so its results depend on how it decides what to read. Three things become decisive:
- Orientation. Does the agent find the right part of the codebase quickly, or does it latch onto the first plausible match?
- Restraint. Does it stay inside the area you named, or does it "improve" neighboring code?
- Verification. Does it run the right tests and check dependents, or does it declare victory after one green test?
A good evaluation is designed to expose these. Our post on how AI coding agents understand a codebase explains the mechanics behind orientation in more detail.
The criteria
1. Context gathering
How does the agent build its picture of the code? Watch the first few minutes of a session. Strong behavior looks like: reading the instruction file, running a targeted search, opening only the relevant line ranges, and stating what it found before planning. Weak behavior looks like: opening many whole files, repeating searches, or planning before looking.
What to check:
- Does it read and use your instruction file (CLAUDE.md, AGENTS.md, GEMINI.md, or equivalent)?
- Can it use external tools for structure, such as an MCP code graph server?
- Does it report file and line anchors for its claims?
2. Instruction following
Large repos run on conventions: "never import from internal", "money is integer cents", "do not edit generated files". Put three or four of your real rules in the instruction file and check whether the agent's changes respect them without being reminded in the prompt.
3. Planning quality
Ask for a plan before edits. A useful plan names files, orders steps, and calls out risks such as changed behavior for other callers. A weak plan restates the task.
4. Verification habits
Does the agent run the relevant tests on its own? Does it pick a targeted test command rather than the whole suite? When a test fails, does it fix the cause or weaken the test? Does it check dependents of what it changed?
5. Change quality
Diff size relative to the task, adherence to local style, absence of unrelated edits, and whether the code would pass your normal review. Use your existing reviewers and your existing bar.
6. Extensibility
Can you add tools? MCP support matters on large repos, because it lets you give any agent the same structural context (definitions, callers, dependents) instead of relying on each agent's built-in search. Also check for hooks, custom commands, and non-interactive modes for automation.
7. Controls and safety
What can the agent do without asking? Look for configurable approval levels, sandboxing of file and network access, and a clear record of the commands it ran. On a shared codebase, you want the narrowest permissions that still let the agent finish tasks. See human-in-the-loop for AI coding agents for patterns.
8. Cost and latency
Measure per completed task, not per hour or per token. An agent that takes longer but needs no review rounds can be cheaper overall than a fast one whose changes get rewritten.
9. Team fit
Does it work from your developers' terminals, editors, and CI? Can work be handed off and reviewed in your existing pull request flow? Is the data handling acceptable to your security team?
Build the test set from your backlog
Pick 10 to 15 tasks from real tickets, ideally ones that were already completed by humans so you have a reference solution. Cover a range:
| Task type | Example | What it exposes |
|---|---|---|
| Localized bug fix | Off-by-one in date range filter | Orientation, test discipline |
| Cross-module change | Add a field from database to API response | Finding all layers, dependents |
| Refactor | Extract a duplicated validation into one function | Restraint, behavior preservation |
| Convention-heavy change | New endpoint following house patterns | Instruction following |
| Investigation only | "Explain how invoices are generated" | Orientation, grounding, accuracy |
| Test writing | Cover an untested module | Understanding behavior, not just code |
| Ambiguous request | "Make the export faster" | Asking questions vs guessing |
For each task, write the prompt once and reuse it verbatim for every agent. Include scope and a definition of done, as you would for a teammate.
Hold the setup constant
The easiest way to get a meaningless result is to give one agent better inputs than another. Before running anything:
- Instruction files. Write one set of repo notes and place identical content in each agent's expected file (CLAUDE.md for Claude Code, AGENTS.md for Codex CLI, GEMINI.md for Gemini CLI, and so on).
- Tools. If you are testing with an MCP code graph server, connect the same server to every agent. If not, connect none.
- Starting state. Same commit, clean worktree, same dependencies installed.
- Permissions. Comparable approval and sandbox settings.
- Reviewers. The same people review every agent's output, ideally without knowing which agent produced it.
Score what reviewers care about
Use a simple sheet per task per agent:
Task: add-tax-exempt-field
Agent: A
Correct files touched (y/n, list misses):
Tests pass (y/n):
Instruction rules respected (count of violations):
Scope respected (unrelated files changed):
Review rounds to mergeable (0, 1, 2, rejected):
Dependents checked (y/n):
Human time spent (minutes, prompting + review):
Notes:
"Review rounds to mergeable" and "human time spent" are the two most honest numbers. They capture whether the agent saved anyone time. Our playbook for reviewing AI agent pull requests has a checklist reviewers can apply consistently.
For investigation tasks, score accuracy against what an expert on that area knows, and count claims without file anchors as unverified.
A two-week evaluation plan
Days 1 to 2: prepare. Choose tasks, write prompts, write the shared instruction notes, decide on tools, and set up worktrees. Pick two or three agents to compare. More than that and you will not have time to review properly.
Days 3 to 7: run. Each agent attempts every task. Use a separate branch or worktree per agent per task. Do not coach agents mid-task beyond what the prompt allows, or note it when you do.
Days 8 to 10: review. Reviewers score each result blind where practical. Collect notes on recurring behaviors: "kept editing generated files", "never ran tests unprompted", "asked a good clarifying question".
Days 11 to 12: vary one thing. Re-run the hardest three tasks with one change, such as adding an MCP code graph server for all agents. This shows how much each agent benefits from better context, which is often more informative than the baseline.
Days 13 to 14: decide and document. Summarize scores by task type. Write down the decision, the reasons, and the configuration you will standardize on. Keep the task set; you will want to run it again.
Interpreting results
Expect no agent to win everything. Patterns you might see:
- One agent is strongest at investigation and explanation, another at mechanical refactors.
- Differences narrow when all agents get the same structural tools, which suggests context was the bottleneck rather than the model.
- Instruction-following failures cluster on a particular rule, which usually means the rule is badly worded or buried.
It is reasonable to standardize on one agent for most work and allow another for specific task types, as long as the instruction files and tools are shared. Keeping your instructions in one source and generating each agent's file from it avoids drift; see writing AGENTS.md and CLAUDE.md.
Red flags during evaluation
- Edits to files outside the stated scope without explanation.
- Tests modified to pass rather than code fixed.
- Claims about code with no file reference, especially functions that do not exist.
- No tests run before declaring done.
- Repeated full-file reads of the same large files in one session.
- Inability to explain its own change when asked.
Where OpenVisio fits
One way to make the comparison fairer is to give every agent the same structural context. OpenVisio exposes a locally built code graph over MCP to agents such as Claude Code, Codex, and Gemini CLI, with tools like resolve_context, find_symbol, and trace_calls. Running the evaluation with and without it on the hardest tasks shows how much of each agent's performance comes from context gathering.
Whatever you choose, the evaluation itself is the durable asset. A dozen real tasks, a fixed setup, and a scoring sheet will tell you more about your options than any leaderboard, and they will still be useful when the next version ships.
