Engineering

Investigate Intermittent Bugs

It fails once in twenty runs: the occurrence pattern IS the evidence. Hunt the difference between failing and passing runs — forensically.

Overview

Intermittent bugs gaslight engineers: every look at the code says it's fine, and one run in twenty says otherwise. This setup investigates a flaky CI test under forensic mode: occurrence patterns treated as primary evidence (when it fails and when it doesn't is the strongest signal available), the difference hunted between failing and passing runs — timing, load, data, environment — and concurrency suspected first, per the checklist: races on shared state, timing dependencies, retry behavior, stale state, cache inconsistencies. Forensic mode matters here most: with evidence this thin, guesses are expensive.

Workflow

Log the pattern before the code
Failure rate, time of day, parallelism, data — the occurrence pattern is the dataset; collect it first.
Diff the runs
What is different about run 17? CI parallelism, shared state, timing — the difference IS the lead.
Demand a longer verification window
The contract's post-fix rule: intermittent problems need extended observation — one green run proves nothing.

Why This Works

Pattern-as-evidence framing extracts signal from a bug that hides from direct observation
The concurrency-first checklist starts where sometimes-bugs usually live
Forensic rules prevent expensive guesses when evidence is thinnest

Best for

Flaky tests that pass locally and fail in CI
Bugs that vanish when observed (and return after the standup)
Race conditions suspected but never caught

Not for

Deterministic failures — the standard strategies diagnose them faster
Fixing flaky tests by rewriting them — pin the cause first; the Test Case Prompt Generator's discipline prevents the next one

Use cases

Hunting the test that fails 5% of CI runs
Diffing failing runs against passing ones systematically
Catching the race that only fires under CI parallelism

Customize This Resource

Opens this setup in Debugging Prompt Generator. Generate to get the full investigation contract — then adjust the problem type, mode, and environment.

Open in Debugging Prompt Generator

Prompt Template

DEBUGGING OBJECTIVE
Pin down the test that fails roughly once in twenty CI runs.
Problem type: intermittent — it does not reproduce consistently; the occurrence pattern is the evidence.
Determine what is actually happening. Do not jump directly to a solution before identifying the most likely root cause.

SYSTEM CONTEXT
Environment: staging — verify parity with production configuration; any difference from production is itself a suspect.
Reported severity: Medium.
Investigation strategy for this problem type:
- Treat occurrence patterns as primary evidence: when it happens and when it does not is the strongest signal available.
- Hunt the difference between failing and passing runs: timing, load, data, environment.
- Suspect time and concurrency first — races, retries, and timing dependencies are the usual suspects for sometimes-bugs.

KNOWN FACTS
Symptoms: OrderFlowTests.CompletesCheckout fails about 5% of CI runs; it always passes locally.
Expected behavior: The test passes deterministically.
Actual behavior: Sporadic timeout waiting for the confirmation element.
Reproduction steps:
- Re-run the CI suite; the failure is not tied to any code change
Logs / errors: none provided.

INVESTIGATION FRAMEWORK
Structure the investigation in exactly these stages:
1. SYMPTOMS — restate the observable symptoms, separating observations from interpretations.
2. KNOWN FACTS — list what is directly evidenced, each fact with its source.
3. REPRODUCTION STEPS — verify the provided steps reproduce the problem; reduce them to the minimal failing case.
4. POSSIBLE CAUSES — candidate causes, drawn from the checklist below and from the evidence.
5. ROOT CAUSE ANALYSIS — the most likely cause and the alternatives, per the root cause rules.
6. EVIDENCE REVIEW — which evidence is solid, which is weak, which is missing, and what contradicts what.
7. MISSING INFORMATION — what you need that you do not have. Ask for it; do not guess it.
8. VALIDATION PLAN — the tests or observations that will confirm or eliminate the leading cause.
9. FIX RECOMMENDATION — per the fix requirements below.
10. POST-FIX VERIFICATION — per the verification requirements below.

DEBUGGING CHECKLIST
Candidate causes to check for this problem type:
1. Race conditions on shared state
2. Timing dependencies: timeouts, debounce, clock skew
3. Retry behavior masking or causing failures
4. Stale state surviving between runs
5. Cache inconsistencies between layers

ROOT CAUSE RULES
- Identify the MOST LIKELY root cause — and at least two ALTERNATIVE causes.
- For each candidate cause: list the evidence supporting it AND the evidence contradicting it.
- Define a validation step for each candidate: what observation would confirm or eliminate it.
- The symptom site and the cause site may be different places — finding where it hurts is not finding why.

EVIDENCE RULES
Label every statement as exactly one of:
- FACT: directly observed. Example: "The application returned HTTP 500."
- ASSUMPTION: believed without direct evidence. Example: "The database is probably down." Assumptions must be verified or discarded — never silently promoted to facts.
- HYPOTHESIS: a testable causal claim. Example: "A failed database connection may be causing the 500 response." Every hypothesis comes with its test.
- Every conclusion must cite the specific evidence that supports it — quote the log line, the stack frame, the reproduction result.
- Unknowns remain unknown: write "insufficient evidence" rather than a plausible guess.
- No unsupported certainty: every claim carries its confidence level and its evidence.
- Treat the absence of expected evidence as a finding in itself.
- Assumptions are forbidden in conclusions — they may appear only in MISSING INFORMATION, as questions to resolve.

VALIDATION REQUIREMENTS
- Before proposing any fix, state how the diagnosis was validated — or what validation is still pending.
- The gold standard: a reproduction that turns the failure on and off by toggling the suspected cause.
- If the root cause cannot be validated with the available information, say so and list what is needed.

FIX REQUIREMENTS
- Any fix proposal must include: the likely cause, the supporting evidence, the proposed change, and how to verify it worked.
- Frame it as "here is why this is likely the fix" — never an unexplained patch.
- Prefer the smallest fix that addresses the root cause; label symptom-only fixes as temporary mitigations.

POST-FIX VERIFICATION
- Define what to observe after the fix: the original reproduction passing, error rates at baseline, the affected metric recovered.
- Check for regressions in the paths the fix touched.
- State how long to observe before declaring the problem resolved — intermittent problems need longer windows.

Related Resources

Explore more resources that work well with this one.