How to Measure Developer Onboarding Without Fooling Yourself

Most teams believe their onboarding is fine, or at least not their biggest problem. Then a new engineer spends the first week waiting for access, the second fighting a local setup that only works on the machines of people who joined years ago, and the third afraid to touch anything because nobody can explain which tests matter. None of this shows up anywhere unless someone measures it.

Measuring onboarding is tricky, though. The obvious metrics are easy to game and easy to misread. A team that sets a target of "first PR in two days" will get first PRs in two days, mostly typo fixes, and learn nothing. This guide proposes a small set of metrics that track real progress toward autonomy, explains how to collect them from systems you already have, and shows how to turn the numbers into improvements instead of a dashboard nobody reads. For the onboarding plan itself, see how to onboard developers to a large codebase; this post is about knowing whether it works.

Key takeaways

• Measure milestones on the path to autonomy, not activity. Time to first PR is a setup metric, not an onboarding metric.
• Use four layers: setup, first change, independent change, and confidence. Each catches different problems.
• Collect timestamps from systems you already use (identity provider, git host, CI, ticket tracker) rather than asking people to log time.
• Pair numbers with a friction log and a short survey. Quantitative data shows where; qualitative data shows why.
• Small samples are normal. Look at each cohort individually and at trends over several cohorts, not averages across three people.
• Never use onboarding metrics to evaluate the new hire. They measure the system.

What onboarding is actually for

Onboarding succeeds when a new engineer can pick up an ordinary ticket in their team's area, figure out what to change, make the change safely, and get it reviewed without anyone holding their hand. Everything before that point is progress toward it. So the metrics should mark that progress, and the question for each one should be "what in our system made this take longer than it needed to?"

A four-layer metric set

Layer 1: Setup

Metric	Definition	Source
Time to access	Start date to having every account needed for a normal ticket	Identity provider logs, access request tickets
Time to local build	Start date to first successful local build and test run	Ask, or log the first run of a setup script
Time to first CI green	Start date to first CI run that passes on their branch	CI API

These are the cheapest to improve and often the most embarrassing. If access takes several days, that is a process fix, not an engineering one. If local setup takes days, the fix is usually a scripted setup, a dev container, or a documented happy path.

A setup script that records its own success is a cheap source of data:

#!/usr/bin/env bash
# scripts/bootstrap.sh
set -euo pipefail
start=$(date +%s)
./scripts/install-deps.sh
./scripts/seed-local-db.sh
npm test -- --run smoke
echo "bootstrap ok in $(( $(date +%s) - start ))s on $(date -u +%F)" >> .bootstrap.log

Ask the new engineer to paste the log into the onboarding ticket. You learn both whether it worked and how long it took.

Layer 2: First change

Metric	Definition	Source
Time to first merged PR	Start date to first merged PR with a non-trivial change	Git host API
First PR review cycles	Number of review rounds before merge	Git host API
First PR cycle time	PR opened to merged	Git host API

Define "non-trivial" before you start counting, for example "changes behavior and includes a test". Otherwise typo fixes dominate. Review cycles on the first PR are informative: many rounds often mean conventions are undocumented, and reviewers are teaching them one comment at a time.

Pull these with the gh CLI:

gh pr list --author new-engineer --state merged --limit 5 \
  --json number,title,createdAt,mergedAt,reviews \
  --jq '.[] | [.number, .createdAt, .mergedAt, (.reviews | length), .title] | @tsv'

Layer 3: Independent change

This is the milestone that matters most.

Metric	Definition	Source
Time to independent ticket	Start date to first ticket picked from the normal backlog and shipped without pairing	Ticket tracker plus a short confirmation from the engineer
Areas touched	Count of distinct modules changed in the first 90 days	git log --author with path grouping
Questions per ticket (trend)	How often the engineer needs synchronous help per ticket, over time	Self-reported in a weekly check-in

The independent ticket is the honest version of "productive". It requires access, setup, understanding of the area, knowledge of conventions, and confidence. When it takes a long time, one of those is missing, and the friction log will tell you which.

A rough "areas touched" query:

git log --author="new.engineer@company.com" --since="90 days ago" --format= --name-only \
  | grep -v '^$' | cut -d/ -f1-2 | sort | uniq -c | sort -rn

Layer 4: Confidence

Some of the most important outcomes are not visible in any system. Measure them with a short survey at 30, 60 and 90 days. Keep it to five questions on a 1 to 5 scale, plus one open question.

1. I can explain how a typical request flows through the system I work on.
2. I know who to ask when I am stuck in an area I do not know.
3. When I change code, I can predict what else might be affected.
4. I trust the tests to tell me if I broke something.
5. I can find the reason a piece of code exists when it looks strange.
6. What was the most confusing thing you ran into this month?

Questions 1, 3 and 5 target exactly the knowledge that knowledge silos hide. Low scores there point at missing maps, missing tests or missing written intent, not at the new hire.

The friction log

Numbers tell you that week two took too long. A friction log tells you why. Ask each new engineer to keep a simple running list during their first month, in a shared doc or a channel.

2026-10-06  Setup  README says Node 18, repo needs Node 20. Lost ~1h.
2026-10-07  Access  Waiting on staging DB credentials. Blocked half a day.
2026-10-09  Code  Could not find where order emails are sent. Asked in #orders.
2026-10-10  Tests  Integration tests fail locally without a VPN. Not documented.

Each entry should be small and specific. Review the log with them weekly and file a ticket for anything that would affect the next person. This is often where the most valuable fixes come from, and it gives the new engineer a concrete way to contribute from day one.

Collecting data without a platform

You do not need a developer experience product to start. A spreadsheet with one row per new engineer and the following columns is enough for the first year:

• Start date
• Access complete date
• First local build date
• First non-trivial PR merged date, with link
• First independent ticket shipped date, with link
• Survey scores at 30, 60 and 90 days
• Top three friction log items

Fill the dates from your systems, confirm them in a quick check-in, and keep the friction log link. After a few cohorts you will see patterns.

Reading the numbers honestly

Onboarding data has small samples and a lot of variation. A few rules keep you from drawing the wrong conclusions.

• Look at individuals, not averages. With three new engineers in a quarter, the average hides the person who took twice as long because their team's area has no tests.
• Normalize for context. Someone joining a team that owns a complex legacy service will take longer than someone joining a greenfield project. Compare like with like, or at least note the difference.
• Watch the trend across cohorts. One quarter's numbers say little. A metric that improves for three cohorts after you scripted setup is real evidence.
• Look for the gap between layers. A fast first PR followed by a very slow independent ticket usually means the first task was too easy, or the area lacks the maps and tests that make independent work possible.
• Keep it out of performance reviews. The moment new hires think these numbers judge them, they will optimize for the metric and stop reporting friction honestly.

Turning measurement into improvements

The point of measuring is to fix the system. A simple loop per cohort:

1. At day 30, review the dates, survey scores and friction log with the new engineer and their manager.
2. Pick the two problems that would most affect the next hire.
3. Assign each as a normal ticket with an owner and a due date before the next start date.
4. At the next cohort's day 30, check whether those problems recurred.

Typical high-leverage fixes, in rough order of cost:

• Scripted access requests or a pre-start access checklist.
• A bootstrap script and a dev container that are tested in CI.
• A short, current first-week checklist.
• A recorded codebase walkthrough for each major area.
• Characterization tests in areas new engineers are afraid to touch.
• A curated list of starter tickets that are real, bounded and close to a core workflow.

Helping engineers build the mental model

Survey question 1, "I can explain how a typical request flows through the system", is often the slowest to improve, because it depends on understanding the architecture rather than on any single fix. Generated maps help here because they stay current without someone maintaining them. With OpenVisio, a new engineer can take a guided Tour of a repository stop by stop, then use Ask to get a sourced explanation of a specific flow they need for their first ticket. Track whether confidence scores move after you introduce something like this; that is the honest test of any onboarding tool.

Onboarding metrics are worth collecting only if they change what the team does. Measure milestones toward autonomy, pair them with a friction log, read small numbers carefully, and fix two things before the next person starts.