Debugging an Unfamiliar Service During an Incident

It is 2 a.m., you are on call, and the alert is for a service you have never opened. The person who wrote it is asleep in another time zone or no longer at the company. The runbook link is a 404. Error rates are climbing and someone in the incident channel is asking for an update.

This happens to almost every on-call engineer eventually, and it is one of the most stressful situations in the job. The good news is that you do not need to understand the service to respond well. You need a calm, ordered method: stabilize first, orient fast using what changed and what the telemetry says, narrow down to the code path that matters, and only then read code. This guide lays out that method step by step, with commands and a checklist, and then covers what to prepare in advance so the next person has an easier night.

Key takeaways

• Mitigate before you understand. Rollback, failover, feature flag off or scale up are often possible without knowing the code.
• Ask "what changed?" first. Recent deploys, config changes, dependency updates and traffic shifts explain most incidents.
• Orient using telemetry before code: which endpoint, which error, which dependency, since when.
• Read code only along the failing path. Start from the error message or the failing endpoint and trace outward.
• Communicate in short, regular updates with what you know, what you are doing and when the next update is.
• Write down what you learn as you go. The timeline is half of the postmortem and the start of the next runbook.

Step 0: Breathe and declare

Before touching anything, take thirty seconds. Confirm the alert is real by checking one dashboard or one request. Then declare the incident according to your process, open a channel or thread, and post a first update:

Investigating elevated 5xx on checkout-api since ~01:52 UTC. Impact: some
checkouts failing. I'm on it, next update 02:15.

This buys you time. People stop asking for status when they know when the next one is coming. If you need help, ask now, not in an hour. Paging the owning team or a more experienced engineer is the right call when customer impact is significant, even if you might solve it alone.

Step 1: Can you mitigate without understanding?

Many incidents can be stopped without knowing why they started. Check these in order, because they are fast and usually reversible:

1. Was there a recent deploy? If the problem started shortly after one, roll it back. You can investigate the bad version afterward.
2. Was there a recent config or flag change? Revert it.
3. Is one instance, zone or region unhealthy? Drain it or fail over.
4. Is it load? Scale up, or enable rate limiting or load shedding if available.
5. Is a dependency down? If the service supports degraded mode or a circuit breaker, turn it on.

Find recent deploys quickly. Depending on your platform:

# Kubernetes: rollout history and recent events
kubectl rollout history deployment/checkout-api -n prod
kubectl get events -n prod --sort-by=.lastTimestamp | tail -30

# Roll back to the previous revision
kubectl rollout undo deployment/checkout-api -n prod

# From git, what merged recently
git log --since="6 hours ago" --merges --oneline origin/main
git log --since="6 hours ago" --oneline origin/main -- services/checkout/

A rollback you can undo is almost always safer than a forward fix written under pressure in code you do not know. The exception is when the deploy included a data migration that the old version cannot handle. Check for migration files in the diff before rolling back.

Step 2: Orient with "what changed?"

If mitigation is not obvious, gather the change context. Systems that worked yesterday and fail today usually changed in one of a few ways:

Change	Where to look
Code	Deploy history, git log on the service path
Config and flags	Flag service audit log, config repo history
Infrastructure	Terraform or cloud console change history, node rotations
Dependencies	Status pages, error rates on calls to downstream services
Data	Recent migrations, backfills, large imports
Traffic	Request volume, new client versions, a large customer onboarding
Time	Certificate expiry, scheduled jobs, month or year boundaries

Line up the start time of the incident with each of these. A precise start time is your best tool; get it from the graph, not from when the alert fired.

Step 3: Orient with telemetry

Before opening the code, use logs, metrics and traces to narrow the problem to a specific path. You want to answer:

• Which endpoints or jobs are failing? All of them, or one?
• What error? Status codes, exception types, error messages.
• Where in the call chain? Is the failure in this service, or in something it calls?
• For whom? All users, one region, one customer, one client version?

Log queries are the fastest route for most teams. Examples in a few common forms:

# Kubernetes logs, recent errors only
kubectl logs deployment/checkout-api -n prod --since=30m | grep -iE 'error|exception|panic' | head -50

# Count error types to find the dominant one
kubectl logs deployment/checkout-api -n prod --since=30m \
  | grep -oE '"error":"[^"]+"' | sort | uniq -c | sort -rn | head

If you have distributed tracing, find a failing trace and read it from the top. It shows you the actual path of a failing request across services and which span errored, which is often more informative than any amount of code reading. Prioritize the most common error, not the most alarming one; a scary stack trace that occurs twice is less important than a boring timeout that occurs thousands of times.

Step 4: Find the code path

Now you have something specific: an endpoint, an error message, a failing dependency call. Use it to enter the code at the right place instead of starting from the top.

# Start from the exact error message
git grep -n "payment authorization failed" -- services/checkout/

# Or from the route
git grep -nE "(post|router\.post|@Post)\(['\"]/checkout" -- services/checkout/

# What changed recently along that path
git log -p --since="7 days ago" -- services/checkout/src/payments/

Read outward from that point, one hop at a time. What called this? What does it call? Where does the failing value come from? You are looking for the specific condition that leads to the error, not for an understanding of the service. The general technique is in tracing a request through a web app; during an incident, you apply it narrowly, along a single path.

Useful questions while reading:

• Is there a timeout, retry or circuit breaker on this call, and what are its settings?
• Is there a cache in front of this, and could it be serving bad data?
• Does this read config or a flag that might have changed?
• Is there a recent commit touching these exact lines?

Step 5: Form a hypothesis and test it cheaply

Write down a hypothesis in the incident channel: "I think the payment client timeout was lowered in yesterday's config change, and the provider has been slower tonight." Then test it in the cheapest way possible: check the config value, check the provider latency graph, compare with the start time. If the evidence fits, apply the smallest mitigation it implies, such as reverting the config. If not, discard it and form the next one.

Avoid making several changes at once. If error rates drop, you want to know which change did it.

Step 6: Communicate as you go

Regular, short updates help everyone, including you, because writing them forces you to summarize what you know.

02:15 update: 5xx on checkout-api is from timeouts calling the payment provider.
Provider latency is up since 01:50. Our client timeout was reduced from 10s to 3s
in a config change yesterday. Reverting that now. Next update 02:30.

Keep a running timeline in the channel, with times, actions and observations. It costs a few seconds per entry and saves an hour when writing the postmortem.

Step 7: Resolve, then hand back

Once impact is mitigated, confirm on dashboards that error rates and latency are back to normal for a sustained period. Post a resolution update. Then hand over the cause investigation and permanent fix to the owning team during working hours, with your timeline and notes. You do not need to fix the root cause at 3 a.m. in a service you do not own.

Prepare before the next incident

The best time to make unfamiliar-service incidents easier is before they happen. A few investments pay off disproportionately:

• A one-page service card for every service: what it does, who owns it, its dependencies, its dashboards, how to roll back, and known failure modes.
• Runbooks for known failure modes with exact commands. A runbook template is in the engineering handoff checklist.
• Consistent telemetry: structured logs with request IDs, standard metrics per endpoint, and tracing across service boundaries.
• Fast, boring rollback that any on-call engineer can run without help.
• Game days: practice incidents in services people do not own, so the first time is not a real outage.
• A map of service dependencies that is generated rather than hand-drawn, so it is current when you need it. The microservices architecture guide covers how to build one.

A service card template:

# checkout-api

Owner: #team-payments, escalation via on-call schedule "payments"
Purpose: creates orders and charges payment methods for web and mobile checkout
Depends on: orders-db (Postgres), payment provider API, inventory-api, events bus
Dashboards: latency/errors (link), provider calls (link)
Rollback: `kubectl rollout undo deployment/checkout-api -n prod`
Flags: checkout.new-tax-engine, checkout.provider-fallback
Known failure modes:
  - Provider latency spikes: enable checkout.provider-fallback
  - inventory-api down: checkout degrades to no stock check (by design)

Orienting in code you have never seen

Even with good preparation, you will sometimes need to understand a code path quickly in a repository you do not know. A structural map helps you find the path without reading everything. OpenVisio indexes a repository in the browser, so during an incident you can use Explore to search for the failing function and see what calls it and what it depends on, or use Ask to get a sourced explanation of a flow such as "how does checkout call the payment provider?" Treat any explanation as a lead to verify against logs and code, not as the answer. If you are working with a coding agent, the same graph is available to it over MCP through tools like trace_calls and search_code.

Unfamiliar-service incidents are survivable with a method: mitigate first, ask what changed, let telemetry point you to the path, read only that path, test one hypothesis at a time, and communicate on a schedule.