OS for Engineers: Performance & Debugging

Strip away the frameworks and the cloud dashboards and a server is a very simple machine. It can compute (CPU), it can remember (memory), it can persist (disk I/O), and it can talk (network). That is the entire list. Every "the app is slow" ticket you will ever receive resolves to one of these four resources being the bottleneck — the point past which adding more work only adds more waiting. The skill is not knowing a hundred tools; it is knowing which resource is guilty and reaching for the one tool that proves it.

To keep yourself honest, use a fixed methodology instead of poking around. The best known is Brendan Gregg’s USE method: for every resource, check three things — Utilization (what fraction of the time it is busy), Saturation (how much work is queued and waiting because it could not be served immediately), and Errors (outright failures). Utilization tells you it is busy; saturation tells you it is overwhelmed; errors tell you it is broken. A resource can be 100% utilized and perfectly healthy, but a resource with a growing saturation queue is always your problem.

Utilization

The percentage of time the resource was busy over an interval. High is not automatically bad — a batch job should peg the CPU.

Saturation

The extra work that could not be serviced immediately and had to queue: run-queue length, I/O wait, swap activity. This is the real pain signal.

Errors

Hard failures: disk EIO, network drops and retransmits, failed allocations. Rare, but they mislead every latency graph when present.

Bottleneck

The one resource whose saturation is limiting the whole system. Fixing anything else first is wasted effort.

One habit underlies everything below: measure a rate over an interval, never a single instant. A tool that prints one snapshot and exits (like plain vmstat with no argument) shows you averages since boot, which are almost useless during an incident. Always give these tools an interval so they report per-second deltas of what is happening right now.

uptime                 # CPU: load average vs core count
vmstat 1 5             # CPU + memory + swap, 5 one-second samples
free -h                # memory: used vs available vs cache
iostat -x 1 3          # disk: per-device %util and await
ss -s                  # network: socket/connection summary

Start with the cheapest signal on the machine. uptime prints the load average — three numbers for the last 1, 5, and 15 minutes. On Linux, load average is the average number of tasks that were either running on a CPU or runnable and waiting in the run queue (and, unlike other Unixes, it also counts tasks blocked in uninterruptible disk sleep). That last detail matters: a Linux load spike can mean "CPU is busy" or "everything is stuck waiting on disk", which is why load alone is a smoke alarm, not a diagnosis.

$ uptime
 02:14:07 up 9 days,  3:41,  2 users,  load average: 7.98, 6.42, 4.11

$ nproc
8

Read those together. This box has 8 cores and a 1-minute load of 7.98 — almost exactly one runnable task per core, which is full but not overloaded. The trend (7.98 → 6.42 → 4.11 from recent to older) says the pressure is rising, not a spike that already passed. The rule of thumb: divide load by nproc. Around 1.0 per core is fully utilized; comfortably above 1.0 per core means tasks are queuing for the CPU and every one of them is now waiting longer than it runs — the run-queue saturation we studied with the scheduler.

Load says how much; it does not say what kind. For that, break the CPU’s time into its states. top’s summary line and vmstat both split CPU time into user (%us — your application code), system (%sy — kernel work on your behalf, i.e. syscalls), iowait (%wa — CPU idle because it is waiting for disk to come back), and idle (%id). This one line often ends the investigation before it starts.

$ vmstat 1 5
procs -----------memory----------  ---swap-- ----cpu----
 r  b   swpd   free   buff  cache   si   so   us sy id wa
 9  0      0 512344  88120 3421556    0    0   74 18  6  2
11  0      0 509880  88120 3421560    0    0   79 15  4  2
10  1      0 508112  88120 3421572    0    0   81 14  3  2

The r column is the number of tasks on the run queue — here 9 to 11 against 8 cores, confirming CPU saturation directly. The CPU is ~80% user, ~15% system, near-zero iowait: this is a genuinely CPU-bound, compute-heavy workload, and the fix lives in your code or in adding cores, not in the disk. Contrast that with the opposite reading — low %us, low %sy, but high %wa and a b (blocked) column that keeps climbing — which is not a CPU problem at all but a disk problem masquerading as high load. Same load number, completely different fix.

Load average

Average count of running + runnable (+ uninterruptible-sleep) tasks over 1/5/15 min. Divide by core count to interpret.

Run queue (r in vmstat)

Tasks ready to run right now but waiting for a CPU. Consistently above core count = CPU saturation.

%us / %sy

Time in your user code vs time in the kernel serving your syscalls. High %sy points at syscall or context-switch overhead.

%wa (iowait)

CPU sitting idle because it is blocked waiting on I/O. High iowait means the bottleneck is the disk, not the CPU.

top is the tool you will open first in almost every incident, so it is worth reading properly instead of just watching the numbers flicker. The top two lines are the summary from the previous section — load average and the %Cpu(s) breakdown — and the table below is one row per process. The art is knowing which four columns actually matter and ignoring the rest.

$ top -b -n1 | head -n 12
top - 02:16:40 up 9 days,  load average: 8.01, 6.90, 4.55
Tasks: 214 total,   9 running, 205 sleeping,   0 stopped,   0 zombie
%Cpu(s): 80.1 us, 14.9 sy,  0.0 ni,  3.0 id,  1.8 wa,  0.0 hi,  0.2 si
MiB Mem :  15884.0 total,    500.3 free,  11210.6 used,   4173.1 buff/cache
MiB Swap:   2048.0 total,   1998.0 free,     50.0 used.   3980.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  30412 app       20   0 12.4g   9.8g   12m R 512.0  63.1  418:22.10 java
   1188 postgres  20   0 1024m  212m  200m S   8.3   1.3   64:12.44 postgres
   9930 app       20   0  742m   88m   14m S   2.0   0.5    2:41.09 node
    884 root      20   0  120m   14m    8m S   0.3   0.1   12:03.77 systemd

The culprit is obvious once you know the columns. That java process shows %CPU of 512.0 — and no, that is not a bug. top reports CPU as a percentage of a single core, so on this 8-core box the maximum is 800%, and 512% means it is actively using about five cores. Its RES is 9.8g and %MEM is 63.1, so it is also holding two-thirds of physical RAM. One process is both the CPU hog and the memory hog; that is where the investigation goes.

The two memory columns confuse people forever, so pin them down now. VIRT is the process’s virtual size (VSZ) — the total address space it has mapped, including memory-mapped files, shared libraries, and pages it reserved but never touched. It is mostly promises. RES is the resident set size (RSS) — the physical RAM actually backing the process right now. When you are hunting a memory problem, RES is the number that matters; a huge VIRT with a modest RES is completely normal and not a leak. This is the same VSZ-versus-RSS distinction from the virtual-memory chapter, now on a live screen.

VIRT (VSZ)

Total virtual address space mapped by the process — includes untouched reservations and shared libraries. Big by design; rarely the problem.

RES (RSS)

Physical RAM the process currently occupies. The real memory-pressure number. Shared pages are counted in each sharer, so RES can over-count across processes.

%CPU

Percent of one core. Can exceed 100% for multithreaded processes; ceiling is 100% × number of cores.

S (state)

R = running/runnable, S = interruptible sleep (normal idle wait), D = uninterruptible sleep (stuck in I/O — watch this), Z = zombie (dead, unreaped).

The state column is the quiet hero. A process parked in S is just waiting politely for work and costs nothing. A process stuck in D — uninterruptible sleep — is blocked inside a kernel I/O call and cannot even be killed with a normal signal; a pile of D-state processes is the classic signature of a saturated or hung disk (or an unresponsive NFS mount). And a Z, a zombie, is a finished child whose parent never called wait to reap it — harmless in ones and twos, but a growing count means a buggy parent, exactly the reaping problem from the process-lifecycle chapter.

Two moves turn top from a dashboard into an investigation. Inside interactive top, press M to sort by memory or P to sort by CPU so the worst offender floats to the top instantly. And press H (or run top -H -p <pid>) to expand a process into its individual threads — indispensable when one thread of a many-threaded server is spinning at 100% while the process as a whole looks merely busy. htop gives you the same data with per-core bars and easier scrolling, but the columns mean exactly the same thing.

More false alarms are raised over memory than any other resource, almost always because someone read the wrong number. A healthy, hard-working Linux box will report almost all of its RAM as "used" and that is exactly how it is supposed to look. The reason is the page cache: the kernel keeps recently read file data in otherwise-free RAM so the next read is instant, as we saw in the file-I/O chapter. That memory is being used, but it is instantly reclaimable — the moment a process needs real memory, the kernel drops cache pages to give it room. Cache is not consumption.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi        10Gi       480Mi       120Mi       4.6Gi       4.9Gi
Swap:          2.0Gi        50Mi       2.0Gi

Walk this line honestly. free shows only 480 MiB truly free, and a panicky reading stops there. But 4.6 GiB is buff/cache — reclaimable file cache — and the column that tells the truth is available: 4.9 GiB. That is the kernel’s own estimate of how much memory a new process could get right now without swapping, because it would evict cache if needed. On any modern kernel, available is the number you alert on. This machine is fine: it has ~5 GiB of headroom dressed up as "used".

So when is memory genuinely a problem? When there is nothing left to reclaim and the kernel is forced to start swapping — pushing anonymous (non-file) pages out to disk to make room, then reading them back when they are needed again. A little swap sitting used is harmless (idle pages parked on disk). The danger is sustained swap traffic, and vmstat’s si (swap-in) and so (swap-out) columns are the cleanest signal on the machine.

$ vmstat 1 5
procs -----------memory----------  ---swap--  ----cpu----
 r  b   swpd   free   buff  cache    si   so   us sy id wa
 1 6  1980312  38112   1024  40220  4820 5120    5  9  2 84
 2 7  1998640  35880   1020  39880  5360 5980    4  8  1 87
 1 8  2011220  33110   1016  38990  6110 6480    6 10  1 83

This is thrashing, and it is unmistakable. swpd (total swap in use) is climbing, si and so are in the thousands of KB per second every second, iowait (wa) is 84–87% because the CPU is doing nothing but waiting for the swap disk, and the run-queue is small while the blocked (b) column grows. The machine is spending its life shuttling pages between RAM and disk instead of doing work — precisely the page-replacement failure mode from the virtual-memory chapter, where the working set no longer fits in RAM. Throughput falls off a cliff and latency goes vertical.

The other end of memory pressure is abrupt rather than gradual: the OOM killer. When memory is exhausted and swap cannot save it, the kernel picks a process (roughly, the one with the largest badness score) and kills it to survive. You will not see it in top — the process is simply gone — so you must look in the kernel log.

$ dmesg -T | grep -i -A1 "killed process"
[Tue Jul  8 02:41:12 2026] Out of memory: Killed process 30412 (java)
  total-vm:13010224kB, anon-rss:10297540kB, file-rss:0kB, oom_score_adj:0

buff/cache

File data and metadata the kernel caches in spare RAM. Reclaimable on demand — counts as "used" but is not a shortage.

available

The kernel’s estimate of memory obtainable now without swapping (free + reclaimable cache). The real free-memory metric.

si / so

Swap-in and swap-out rate in vmstat. Sustained non-zero values are the definitive thrashing signal.

OOM killer

Kernel last resort that terminates a process to reclaim memory when both RAM and swap are exhausted. Logged in dmesg.

When the CPU is idle but the system is still slow, and %wa is high, the disk is the prime suspect. The tool that settles it is iostat -x, which prints extended per-device statistics. Run it with an interval so it reports live rates, and discard the very first sample — that one is the average since boot and will lie to you.

$ iostat -x 1 2
Device   r/s    w/s   rkB/s   wkB/s  r_await w_await  aqu-sz  %util
nvme0n1  12.0   40.0  480.0  2100.0    0.42    0.55    0.03   3.10

# ...second sample, under a heavy query load:
Device   r/s    w/s   rkB/s   wkB/s  r_await w_await  aqu-sz  %util
sdb     980.0   14.0 15680.0   224.0   89.30   12.40   88.20  99.60

The two devices tell opposite stories and teach the whole lesson. nvme0n1 is doing modest I/O at 0.42 ms per read and 3% utilization — effectively idle. sdb is the problem: %util is 99.6%, meaning the device was busy essentially the entire interval, r_await is 89.3 ms (each read takes nearly a tenth of a second to come back), and aqu-sz — the average queue depth — is 88, meaning ~88 requests were stacked up waiting at any moment. That queue is disk saturation in the USE sense, and the 89 ms await is what your users experience as a slow page.

This is the single most important distinction in storage performance: throughput is not latency. Throughput is how many bytes per second the device moves (rkB/s, wkB/s); latency is how long any one request takes to complete (await). A disk can post excellent throughput while every individual request is agonisingly slow, because throughput rewards big sequential transfers while latency is what a single small random read feels. Users and databases live in latency. A dashboard that only graphs MB/s will happily show green while await quietly ruins you.

The other trap is r/s and w/s — reads and writes per second, i.e. IOPS. These are operations, not bytes. A workload doing 980 tiny 16 KB random reads per second can saturate a device that would barely notice the same bytes delivered as a few large sequential streams, because each seek and each request has fixed overhead. When await climbs with high IOPS but low kB/s, you are looking at a random-I/O pattern — the classic profile of an under-indexed database doing scattered lookups.

Once iostat proves the disk is saturated, one question remains: who is doing it? iotop answers directly, ranking processes by their actual read and write rates — the disk-world equivalent of top.

$ sudo iotop -b -o -n1
Total DISK READ: 15.30 M/s | Total DISK WRITE: 0.22 M/s
  TID  PRIO  USER   DISK READ  DISK WRITE  COMMAND
 1188  be/4  postgres 14.90 M/s   0.00 B/s  postgres: parallel worker ...
 1190  be/4  postgres  0.40 M/s   0.20 B/s  postgres: walwriter

r/s, w/s (IOPS)

Read and write operations per second. Counts requests, not bytes — many small random ops hurt far more than a few large ones.

await

Average time each I/O takes to complete, in milliseconds, including queue time. The latency your application and users actually feel.

%util

Fraction of the interval the device was busy. Near 100% means saturation on a single spindle (less definitive on SSDs/arrays that serve in parallel).

aqu-sz

Average number of requests queued at the device. A deep queue with high await is the definition of a disk bottleneck.

Sometimes CPU, memory, and disk all look fine and the system is still sluggish. The hidden tax is often context switching — the cost of the scheduler saving one task’s state and loading another’s, which we met in the scheduling chapter. Each switch is cheap, but at tens of thousands per second the machine spends its time swapping tasks in and out rather than running them, and useful work quietly evaporates into overhead.

$ vmstat 1 5
procs -----------memory----------  ---system---- ----cpu----
 r  b   swpd   free   cache        in      cs    us sy id wa
 6  0      0 480112  3.4g      12010  188540    22 61 15  2
 7  0      0 479980  3.4g      12240  192880    20 64 14  2
 5  0      0 480560  3.4g      11980  190220    23 62 13  2

The tell here is not the CPU being busy but where it is busy. cs is ~190,000 context switches per second, and %sy (system/kernel time) is over 60% while %us (your actual code) is only ~20%. The machine is burning most of its CPU inside the kernel just scheduling, not running your application. High system time with a high context-switch rate is the fingerprint of either lock contention or simply too many threads fighting over too few cores.

To tell those two apart, split the switches by cause with pidstat -w. A context switch is voluntary (cswch/s) when a task gives up the CPU on its own because it must wait — blocked on a lock, a condition variable, or an I/O it cannot complete yet. It is involuntary (nvcswch/s) when the scheduler forcibly preempts it because its time slice expired and other runnable tasks are queued. The ratio between them points straight at the cause.

$ pidstat -w -p 30412 1 3
#      Time   UID     PID   cswch/s  nvcswch/s  Command
   02:31:10  1001   30412  48210.0     120.0    java
   02:31:11  1001   30412  47980.0     138.0    java
   02:31:12  1001   30412  49110.0     115.0    java

This process is doing ~48,000 voluntary switches per second and almost no involuntary ones. That lopsided ratio is the signature of lock contention: threads keep acquiring a lock, doing a sliver of work, blocking, and handing the CPU away — over and over. The threads are not being preempted; they are constantly putting themselves to sleep waiting on each other, exactly the synchronization bottleneck from the concurrency chapters. The opposite reading — high nvcswch/s (involuntary) — means you simply have more runnable threads than cores and the scheduler is time-slicing frantically; the fix is fewer threads, not faster ones. A thread pool sized to 500 on an 8-core box does not run 500 things at once; it just pays 500 threads’ worth of switching overhead.

The in column is the companion signal: hardware and timer interrupts per second. A network-heavy box under a packet flood, or a misbehaving device, will show interrupts spiking, and mpstat -P ALL will reveal the kernel time concentrated in soft-interrupt handling (%soft) on specific cores. It is the same underlying story — the CPU is busy with kernel bookkeeping rather than your code — just driven by devices instead of by threads.

Context switch (cs)

Scheduler saving one task and restoring another. Necessary, but tens of thousands per second is pure overhead eating your throughput.

Voluntary (cswch/s)

Task yields the CPU itself because it must wait — on a lock, condition variable, or I/O. A high rate points at contention or blocking.

Involuntary (nvcswch/s)

Scheduler preempts the task because its slice ended and others are queued. A high rate points at too many runnable threads for the cores.

Interrupts (in)

Hardware/timer signals per second. Spikes point at network or device load; inspect per-core with mpstat -P ALL.

The tools so far told you which resource is saturated. Tracing and profiling tell you why, down to the syscall and the function. They answer two different questions, and reaching for the wrong one wastes an incident. Ask "what is my process asking the kernel to do, and where is it stuck?" and you want strace. Ask "which of my functions is actually burning the CPU?" and you want perf.

strace lists every system call a process makes with its arguments, return value, and errno — the boundary from the syscalls chapter, made visible on a live process. Its killer feature under pressure is -c, which does not print the flood of individual calls but a summary table of where syscall time and count went. Attach it to a running process with -p, let it sample for a few seconds, then Ctrl-C.

$ sudo strace -c -p 9930
strace: Process 9930 attached
^Cstrace: Process 9930 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 74.10    2.104880          51     41200           read
 18.30    0.519900          25     20600           write
  6.90    0.196010          95      2060      2060 futex
  0.70    0.019880           9      2100           epoll_wait
------ ----------- ----------- --------- --------- ----------------
100.00    2.840670                 65960      2060 total

Read the summary as a story. This process made 41,200 read calls in a few seconds — it is I/O-syscall-bound, hammering read, and the fix is almost certainly to read in bigger chunks so it crosses the boundary far fewer times, the exact batching lesson from the syscalls chapter. The errors column also flags 2,060 failing futex calls, a hint of lock churn worth correlating with the context-switch reading. One caveat that matters in production: strace uses ptrace and can dramatically slow the traced process, so trace briefly and never leave it attached to a busy service.

When the problem is CPU-bound — high %us, your own code eating cores — strace is useless because the process is not making syscalls; it is computing. That is perf’s job. perf is a sampling profiler: many times a second it interrupts the CPU and records the current call stack, so the functions that appear most often are, by definition, where the time goes. perf top gives you a live, htop-like view of the hottest functions across the whole system.

$ sudo perf top
Samples: 480K of event "cpu-clock", Event count: 121000000000
Overhead  Shared Object        Symbol
  38.42%  app.jar (java)       [.] com.acme.pricing.recompute
  15.10%  libc-2.31.so         [.] __memmove_avx_unaligned
   9.77%  [kernel]             [k] copy_user_enhanced_fast_string
   4.31%  app.jar (java)       [.] java.util.HashMap.resize

The answer is right there: one application method, recompute, is 38% of all CPU samples on the machine. That is your hotspot. For a durable artifact — and for anything with a deep call stack — record instead of watch, then fold the samples into a flame graph. perf record captures stacks over an interval; the flame-graph tooling turns them into an SVG where the x-axis is the proportion of samples (wider = more CPU) and the y-axis is stack depth. You do not read a flame graph top-to-bottom; you scan for the widest plateaus, because a wide frame means the CPU sat in that function (and its children) for a large share of the samples.

# Sample the whole system at 99 Hz for 30 seconds
sudo perf record -F 99 -a -g -- sleep 30

# Fold and render (Brendan Gregg’s FlameGraph scripts)
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > cpu.svg

There is also ltrace, the same idea as strace but for library calls rather than syscalls — handy when the time is going into a userspace library (say, repeated regex compilation) that never crosses into the kernel. And the modern frontier is eBPF, which safely runs tiny sandboxed programs inside the kernel to measure things with far less overhead than ptrace-based strace — safe enough to run on production. The bcc/bpftrace toolkit gives you targeted one-liners: execsnoop traces every process that starts, opensnoop every file opened, biolatency draws a histogram of disk-I/O latency, and tcpconnect logs outbound connections. When strace is too heavy and perf too broad, an eBPF one-liner answers a precise question cheaply.

strace

Traces system calls (the kernel boundary). Use -c for a time/count summary. High overhead — trace briefly, never leave attached in production.

perf

CPU sampling profiler. perf top for a live hotspot view; perf record + flame graph to find which function burns the CPU.

Flame graph

Stacks rolled up into a chart: width = share of CPU samples, height = stack depth. Hunt the widest frames — that is where to optimise.

eBPF / bcc

Sandboxed in-kernel programs for low-overhead, production-safe tracing. execsnoop, opensnoop, biolatency answer precise questions cheaply.

Put it all together and performance debugging becomes a two-question funnel you can run cold, half-asleep, during an incident. First: which of the four resources is saturated? Walk them in order with the 60-second sweep from the first section and stop at the one with a queue. Second: which layer owns the problem — your application code, the syscalls it makes, the kernel, or the device underneath? The first question picks the resource; the second picks the tool. Everything in this chapter slots into that grid.

The table below is the playbook itself — symptom to tool to cause. It is worth committing to muscle memory, because the difference between a ten-minute incident and a three-hour one is usually just knowing which tool to open first instead of guessing.

App slow, load high, %us high

CPU-bound in your own code. Tool: top (confirm), then perf top / flame graph. Cause: a hot function — the recompute-style hotspot.

Load high, %wa high, CPU idle

I/O-bound. Tool: iostat -x (await, %util), then iotop for the process. Cause: disk saturation, random I/O, or an unindexed query.

Slow, RAM "full", swap busy

Thrashing. Tool: free -h (read available), vmstat si/so. Cause: working set exceeds RAM — add memory or shrink footprint.

Process vanished, no crash log

OOM kill. Tool: dmesg -T | grep -i kill. Cause: memory exhausted; the kernel sacrificed the biggest process to survive.

Latency spikes, %sy + cs high

Scheduling overhead. Tool: pidstat -w. Cause: lock contention (voluntary switches) or too many threads (involuntary switches).

Fails with no message

Unknown syscall error. Tool: strace -p to catch the failing call and its errno. Cause: a bad path, permission, or exhausted resource (e.g. fd leak).

uptime            # 1. load vs cores — is anything saturated at all?
vmstat 1 5        # 2. run queue (r), swap (si/so), switches (cs), cpu split
top -b -n1        # 3. the culprit process: %CPU, RES, state
free -h           # 4. memory: available, not used
iostat -x 1 3     # 5. disk: await + %util if %wa was high
dmesg -T | tail   # 6. errors: OOM kills, device resets, kernel warnings

Notice how every entry in that table is an operating-system concept you already learned, now wearing an incident’s clothes. "High load" is the run queue from the scheduling chapter. "Thrashing" is page replacement failing when the working set outgrows RAM. "%wa" is the I/O-wait state from the process-lifecycle chapter. "Voluntary context switches" are threads blocking on the locks from the concurrency chapters. "OOM" is virtual memory finally running out of physical backing. You are not learning a separate discipline called "performance"; you are applying the OS model to a live machine.

That is the real payoff of this whole course for a working engineer. The invisible layer under your code is now legible: you can look at a struggling server and reason about it — name the resource, name the layer, reach for the one tool that proves it, and fix the actual bottleneck instead of restarting things and hoping. "It just works" has become "I know exactly what the OS is doing, and I can prove it".