High Linux load average: tell CPU saturation from I/O wait and D-state tasks
Load average is not CPU usage alone. It includes runnable work and uninterruptible I/O waits, so pair it with queue, iowait, and process-state evidence.
Maintained by Kevin · Ovalk
Scope and prerequisites
Linux host-level diagnosis; container CPU limits may differ from host CPU count. iostat usually comes from sysstat, and PSI availability depends on kernel configuration.
Record a time window, workload and normal baseline. Sample several intervals rather than relying on the first cumulative report. Diagnostic access does not authorize killing processes or restarting storage clients.
Common symptoms
- Load is much higher than CPU core count.
- Latency rises while CPU utilization remains modest.
- Many processes are in D state.
1. Take a one-minute incident snapshot
Before restarting or scaling, capture uptime, CPU, memory, disk latency, and process states. High load with low CPU is often storage, NFS, or kernel wait rather than a compute shortage.
uptime
vmstat 1 5
iostat -xz 1 5
ps -eo state,pid,ppid,comm,wchan:32 --sort=state | head -n 302. Separate CPU work from I/O congestion
For CPU saturation, identify hot processes and code paths. For iowait, inspect device queues, errors, filesystem behavior, and remote storage. D-state processes need their waited-on resource repaired.
top -b -n1 | head -n 25
dmesg -T | tail -n 803. Recover without amplifying the incident
Rate-limit, pause batch work, or drain traffic before killing many processes. If a node restart is necessary, confirm replicas and connection movement, then validate with the same metrics afterward.
systemctl list-jobs
cat /proc/pressure/ioInterpret the evidence
| Observation | What to check next |
|---|---|
| High runnable queue and busy CPUs | Investigate CPU demand and container throttling. Compare with the CPU allocation available to the workload, not an arbitrary universal threshold. |
| D-state tasks and storage pressure | Correlate wait channels with device or remote-filesystem health. D state is uninterruptible sleep; repeated kill attempts do not repair the waited-on resource. |
| Load remains high after service recovers | Load is averaged over time. Use current latency, queue samples and pressure signals to distinguish lingering averages from an ongoing incident. |
Illustrative diagnosis
A hypothetical example to explain the reasoning, not a reported customer incident or a claim of testing on your stack.
An eight-core host has load 30 but mostly idle CPUs. Several workers are in D state and storage latency rose at the same time. Adding CPU would not unblock those storage waits. Reduce incoming work through the service’s controls, investigate the storage path, and compare the same metrics after the dependency recovers.
Verify recovery
- Compare application latency and error rate with CPU queues, I/O latency and PSI over the same observation window.
- Gradually restore traffic or paused batch work and confirm pressure does not immediately return; a quiet node is not proof of capacity.
Rollback and stopping point
Keep rate-limit and concurrency changes reversible. Before any restart, confirm healthy replicas and storage recovery behavior; abort the restart plan if failover capacity is missing.
Prevention and long-term repair
- Observe CPU, iowait, disk queueing, and PSI together.
- Set concurrency limits and schedule batch jobs away from peak.
- Set availability and latency objectives for storage dependencies.
References and corrections
Use the documentation for your installed version. The references below explain the underlying behavior; commands still need environment-specific validation.
Report a correction to Kevin — include the page URL, version and a redacted reproduction. See our editorial policy.