Debugging Techniques

What and Why

Debugging is the process of identifying why a program is not doing what you intended. It goes beyond running a debugger tool — it's a systematic discipline for reasoning about broken systems under uncertainty.

You debug daily: a test fails unexpectedly, a production service is returning 500s, a service is leaking memory, a race condition reproduces once every three days. The engineers who resolve these quickly aren't smarter — they have better mental models and more disciplined processes.

Why debugging skill matters:

• Time: a senior engineer can resolve in 20 minutes what a junior engineer spends 3 hours on, with the same toolset. The difference is methodology.
• Production pressure: bugs in production degrade user experience every second they're active. Fast diagnosis directly reduces impact.
• Understanding: the act of debugging reveals how a system actually works, not how you assumed it works. Every difficult bug you trace teaches you something.

Core Concepts

The Scientific Method Applied to Bugs

Effective debugging follows a loop:

1. Observe: collect symptoms — what is happening that shouldn't be, or what isn't happening that should?
2. Hypothesize: form a specific, falsifiable explanation for the cause.
3. Predict: if your hypothesis is correct, what should you observe?
4. Test: run an experiment that confirms or refutes the hypothesis.
5. Revise: update your hypothesis based on evidence.

The most common debugging mistake is skipping to step 4 without steps 2–3. Making random changes hoping to stumble on a fix wastes time and often introduces new bugs.

Reproduce First

A bug you cannot reproduce is a bug you cannot verify you've fixed. Before touching code:

1. Find the exact conditions that trigger the bug.
2. Write those down — steps, input data, environment, frequency.
3. Create the smallest possible reproduction case.

Reducing a bug to a minimal reproduction often reveals the cause in the process. It also gives you a regression test to prevent recurrence.

Narrow the Search Space

Instead of scanning all your code looking for the bug, use techniques that cut the search space in half with each step:

• Binary search in time: if you know a commit range where the bug was introduced, use git bisect to find the exact commit.
• Binary search in code: comment out half the code path and see if the bug still occurs. Then the other half. Repeat.
• Divide and conquer with inputs: if a large input causes a failure, reduce the input by half. If the smaller input still fails, reduce again. If not, use the other half.

Read Error Messages Carefully

Error messages tell you where and often why — but engineers frequently skim them. Read the full stack trace. The actual error is usually at the top or bottom; the middle is noise. Identify:

• What exception/error type
• What line / what function
• What the actual message says

If you don't understand an error, search it verbatim before doing anything else.

Understand Before Changing

The temptation is to change things and see if the bug goes away. This works occasionally and misleads you constantly. Before changing anything, build a hypothesis. Understand why the change would fix it.

"I'll change this to X and see what happens" → you're guessing. "I believe the issue is Y because of Z. Changing X should fix Y." → you're debugging.

How It Works in Practice

Printf / Logging Debugging

The simplest tool is still effective: add log statements to emit the state of your system at key points.

Copy
func processOrder(order Order) error {
    log.Printf("processOrder: received order id=%s amount=%.2f", order.ID, order.Amount)

    discount := calculateDiscount(order)
    log.Printf("processOrder: calculated discount=%.2f", discount)

    finalAmount := order.Amount - discount
    log.Printf("processOrder: final amount=%.2f", finalAmount)

    if err := chargeCustomer(order.CustomerID, finalAmount); err != nil {
        log.Printf("processOrder: charge failed customer=%s err=%v", order.CustomerID, err)
        return fmt.Errorf("charge failed: %w", err)
    }
    return nil
}

When using logs to debug: log inputs AND outputs at each step. You want to trace data transformations, not just see that a function was called.

Remove debug logs before committing, or use log levels (DEBUG) and ensure debug-level output is suppressed in production.

Using a Debugger (Step-Through Debugging)

Debuggers let you pause execution at any point and inspect state. Every language ecosystem has one:

• Go: dlv (Delve)
• Python: pdb, ipdb, or IDE debuggers
• Java: IntelliJ's built-in debugger
• Node.js: node --inspect, Chrome DevTools, VS Code debugger

Core operations:

Copy
# Delve (Go) — start debugging a test
dlv test ./billing/ -- -run TestProcessOrder

# Set breakpoint at line 42
(dlv) break billing.go:42
(dlv) continue
(dlv) print order.Amount    # inspect variable
(dlv) locals                # print all local variables

Use conditional breakpoints to avoid breaking on every iteration of a loop:

(dlv) break billing.go:42 if order.ID == "ORD-9999"

git bisect — Finding the Regression Commit

When you know behavior was correct at some point and is now broken:

Copy
git bisect start
git bisect bad HEAD          # current state is broken
git bisect good v2.4.0       # this version was working

# Git checks out the midpoint commit
# Test whether the bug exists on this commit
go test ./...

git bisect good              # bug not present here
# or
git bisect bad               # bug present here

# Git keeps narrowing; repeat until:
# "abc123 is the first bad commit"

git bisect reset             # return to HEAD

Git bisect performs binary search. For 1000 commits, you'll find the culprit in ~10 steps.

For automated bisect:

Copy
git bisect run go test ./billing/...

Git will automatically mark commits good/bad based on the exit code of your test command.

Reading Stack Traces

Given a panic or unhandled exception, the stack trace shows the call chain at the moment of failure.

goroutine 1 [running]:
main.processPayment(...)
    /app/payment.go:87 +0x2a3
main.handleCheckout(0xc000123400)
    /app/handlers.go:42 +0x1b8
net/http.HandlerFunc.ServeHTTP(...)
    /usr/local/go/src/net/http/server.go:2136

Read bottom-up to understand the call chain. Read top-down to find where execution died. The line in your code (not library code) near the top is usually where to focus.

Debugging Race Conditions

Race conditions are non-deterministic — they won't reproduce on every run. Approaches:

Copy
# Go race detector — run tests with race detection enabled
go test -race ./...

# Logs with timestamps and goroutine IDs
log.Printf("goroutine %d: acquired lock", goid())

For races that only appear under load:

• Increase concurrency in tests (more goroutines, shorter sleep durations).
• Add delays at strategic points to widen timing windows.
• Review all shared mutable state — if two goroutines touch the same variable without synchronization, it's a potential race.

Debugging Memory Issues

Copy
# Go: generate and view a heap profile
import _ "net/http/pprof"

# Start profiling endpoint
go tool pprof http://localhost:6060/debug/pprof/heap

# Show top memory consumers
(pprof) top
(pprof) list processOrders   # show source-level allocation

Memory leaks in long-running services typically come from:

• Goroutines that never exit (leaked goroutine)
• Caches or maps that grow unboundedly
• Event listeners or callbacks that aren't deregistered

Structured Approach for Production Incidents

When something is broken in production and you're under pressure:

1. Preserve state before touching anything. Capture logs, metrics, traces, heap dumps. You may lose evidence if you restart.
2. Establish a timeline. When did it start? What changed before it started? (Deployments, config changes, traffic spikes.)
3. Check metrics dashboards first. CPU, memory, error rate, latency, request rate. Changes in these narrow the search space.
4. Read recent logs. Filter to errors and warnings in the affected service.
5. Form a hypothesis. "I think the new deployment introduced a slow query that's causing timeouts under load."
6. Verify. Check query performance metrics, deployment diff, slow query logs.
7. Fix forward or rollback. If the fix is clear and safe, apply it. If not, rollback the last deploy.

Common Mistakes

Changing code before understanding the cause. This wastes time and can mask the root cause. The fix looks like it worked, the bug comes back in production, and now you've also made unexplained changes to the codebase.

Debugging the wrong layer. You spend 45 minutes in application code when the bug is in a configuration file or a database schema. Before diving into code, verify that the environment, dependencies, and configuration are what you expect.

Ignoring the obvious. Have you restarted the service? Is this a cached result? Did your build actually compile the new code? Is the environment variable set? Check the trivial things first.

Failing to isolate the reproduction case. "It fails in production but not locally" is an environmental difference, not magic. Identify what's different: database state, environment variables, request payloads, concurrency level, platform. Replicate those conditions locally.

Not writing a regression test. You found and fixed the bug. Without a test, the same conditions can reintroduce it silently. Add a test that exercises the exact path that was broken.

Tradeoffs

Printf vs. debugger. Printf debugging is fast to set up and works everywhere, including production. Debuggers give you interactive inspection without modifying code and are better for complex state or stepping through unfamiliar code. Use printf for targeted, quick investigations; use a debugger when you need to explore.

Local reproduction vs. debugging in production. Debugging in production gives you real state and real traffic, but poses risk (adding log statements in a hot path can affect performance; attaching a debugger pauses threads). Reproducing locally is safer but may not capture all conditions. For memory or concurrency issues, production profiling is often necessary.

Systematic vs. intuitive debugging. With experience, you develop intuitions for where bugs live. Intuition is fast when correct and expensive when wrong. Use the scientific method to validate intuitions rather than act on them blindly.

Quick Reference

Debugging loop:
  1. Observe symptoms exactly
  2. Hypothesize a specific cause
  3. Predict: if true, what would you see?
  4. Run the experiment
  5. Revise the hypothesis

Narrowing tools:
  git bisect         → find the regression commit
  binary search      → cut the suspected code in half
  minimal repro      → reduce inputs until only the bug remains

Debugger operations:
  breakpoint         → pause at a line
  step over (n)      → execute line, stay at same level
  step into (s)      → descend into function
  step out (fin)     → complete function, return to caller
  continue (c)       → run to next breakpoint
  watch expression   → monitor variable value

Production debugging order:
  1. Preserve state (logs, metrics, heap)
  2. Build a timeline (what changed before the issue?)
  3. Check dashboards (CPU, errors, latency, traffic)
  4. Read logs (errors and warnings first)
  5. Hypothesize → verify → act

After every bug fix:
  → Write a regression test
  → Document the cause in the commit message or incident log