Engineering

Debug Performance Problems

Measure before reasoning: find WHERE the time goes, separate latency from memory from throughput, and think at production scale.

Overview

Performance debugging dies of premature theorizing: the team optimizes the suspected hotspot while the real one idles unprofiled. This setup investigates a dashboard that got four times slower over a month: the performance strategy demands measurement before hypothesis (where does the time actually go?), separates latency from throughput from memory — different causes, different fixes — and reasons at production scale, where the checklist lives: N+1 queries, hot-path allocations, blocking calls, cache misses, expensive loops. The month-long degradation timeline is itself evidence: what grew?

Workflow

Locate before explaining
The first demand is a measurement plan: where do the 8 seconds go? Theories come after the profile.
Use growth as evidence
4× slower over a month with no deploy points at data volume — the timeline narrows the suspects.
Validate with a toggle
The gold standard applies: turn the suspected cause off (cache it, limit it) and watch the metric move.

Why This Works

Measurement-first ordering prevents optimizing the wrong thing well
The latency/throughput/memory split routes each symptom to its own cause family
Scale reasoning finds the problems that test environments are too small to show

Best for

Pages and jobs that got slower without a deploy to blame
Systems where "it's the database" is assumed, not measured
Performance work that must justify its engineering time

Not for

Writing performance tests after the fix — that's the Test Case Prompt Generator's performance coverage
Reviewing code for performance smells without a live problem — that's the Code Review Prompt Generator

Use cases

Diagnosing gradual slowdowns that track data growth
Separating database time from render time before optimizing either
Finding the N+1 that was invisible at last year's scale

Customize This Resource

Opens this setup in Debugging Prompt Generator. Generate to get the full investigation contract — then adjust the problem type, mode, and environment.

Open in Debugging Prompt Generator

Prompt Template

DEBUGGING OBJECTIVE
Diagnose why the dashboard became four times slower over the last month.
Problem type: performance — slowness, timeouts, or resource spikes; measure before reasoning.
Determine what is actually happening. Do not jump directly to a solution before identifying the most likely root cause.

SYSTEM CONTEXT
Environment: PRODUCTION — investigate with production discipline:
- Assess customer impact first: scope, severity, and whether it is ongoing.
- Review rollback options before any intervention — the fastest fix may be reverting.
- Review the deployment history for the symptom window.
- Review observability: dashboards, alerts, and logs for the affected window — note what monitoring missed.
- After any fix, verify recovery in monitoring — not just the absence of complaints.
Reported severity: High.
Investigation strategy for this problem type:
- Identify WHERE time or memory goes before asking why — measurement precedes hypothesis.
- Separate latency from throughput from memory: they have different causes and different fixes.
- Reason at production scale: a problem invisible at 100 rows owns the system at a million.

KNOWN FACTS
Symptoms: Dashboard loads in 8–12 seconds; it loaded in about 2 seconds a month ago.
Expected behavior: Dashboard loads in under 2 seconds.
Actual behavior: 8–12 second loads, worse during business hours.
Reproduction steps:
- Open the main dashboard as a user with more than 1,000 projects
Logs / errors: none provided.

INVESTIGATION FRAMEWORK
Structure the investigation in exactly these stages:
1. SYMPTOMS — restate the observable symptoms, separating observations from interpretations.
2. KNOWN FACTS — list what is directly evidenced, each fact with its source.
3. REPRODUCTION STEPS — verify the provided steps reproduce the problem; reduce them to the minimal failing case.
4. POSSIBLE CAUSES — candidate causes, drawn from the checklist below and from the evidence.
5. ROOT CAUSE ANALYSIS — the most likely cause and the alternatives, per the root cause rules.
6. EVIDENCE REVIEW — which evidence is solid, which is weak, which is missing, and what contradicts what.
7. MISSING INFORMATION — what you need that you do not have. Ask for it; do not guess it.
8. VALIDATION PLAN — the tests or observations that will confirm or eliminate the leading cause.
9. FIX RECOMMENDATION — per the fix requirements below.
10. POST-FIX VERIFICATION — per the verification requirements below.

DEBUGGING CHECKLIST
Candidate causes to check for this problem type:
1. N+1 queries and database calls inside loops
2. Excessive allocations or copies in hot paths
3. Blocking calls on async or UI paths
4. Cache misses — or absent caching on repeated work
5. Expensive loops with avoidable work per iteration

ROOT CAUSE RULES
- Identify the MOST LIKELY root cause — and at least two ALTERNATIVE causes.
- For each candidate cause: list the evidence supporting it AND the evidence contradicting it.
- Define a validation step for each candidate: what observation would confirm or eliminate it.
- The symptom site and the cause site may be different places — finding where it hurts is not finding why.

EVIDENCE RULES
Label every statement as exactly one of:
- FACT: directly observed. Example: "The application returned HTTP 500."
- ASSUMPTION: believed without direct evidence. Example: "The database is probably down." Assumptions must be verified or discarded — never silently promoted to facts.
- HYPOTHESIS: a testable causal claim. Example: "A failed database connection may be causing the 500 response." Every hypothesis comes with its test.
- Prefer evidence over inference; where evidence is missing, label the assumption explicitly.
- Carry at least two hypotheses until evidence separates them.

VALIDATION REQUIREMENTS
- Before proposing any fix, state how the diagnosis was validated — or what validation is still pending.
- The gold standard: a reproduction that turns the failure on and off by toggling the suspected cause.
- If the root cause cannot be validated with the available information, say so and list what is needed.

FIX REQUIREMENTS
- Any fix proposal must include: the likely cause, the supporting evidence, the proposed change, and how to verify it worked.
- Frame it as "here is why this is likely the fix" — never an unexplained patch.
- Prefer the smallest fix that addresses the root cause; label symptom-only fixes as temporary mitigations.

POST-FIX VERIFICATION
- Define what to observe after the fix: the original reproduction passing, error rates at baseline, the affected metric recovered.
- Check for regressions in the paths the fix touched.
- State how long to observe before declaring the problem resolved — intermittent problems need longer windows.

Related Resources

Explore more resources that work well with this one.