I/O · XB Field Notes

iotop shows which process is burning the disks

Wed, 15 Mar 2028 00:00:00 +0000

iotop connects task-level kernel accounting with a system-wide storage investigation. It can identify tasks reporting substantial IO or waiting, but those are different observations. A task blocked behind other work may be a victim rather than the source of contention. Kernel counters do not provide perfect attribution for every filesystem, cache, or storage layer.

Use iotop beside device and application measurements. Neither an active writer nor a high wait value alone proves a storage bottleneck. Compare the same time window before deciding which component deserves closer inspection.

Per-process counters

Check the installed implementation and version. In the cited iotop-c, main-process rows already include thread activity; -P hides separate thread rows. Other variants need their own manual. The -o option filters for observed IO activity; it does not classify harmful work. Read/write byte rates and IO-wait percentages have different units and should not be ranked as interchangeable measures.

Linux read_bytes accounts storage-layer reads; cached reads need not produce them. write_bytes is accounted when pages are dirtied, not necessarily when the device completes a write. Cancelled writes and later writeback can separate task totals from device throughput. Delay accounting records task waiting, including synchronous block-IO delay; CPU iowait is a separate aggregate and is not a reliable per-task blame measure.

See whatever sweeps the disk

Choose an explicit sort column and observe long enough to distinguish sustained work from a burst. A high write rate can be an expected backup, index build, or logging workload. Correlate process ownership, application role, and current work before changing scheduling or limits. A high rank identifies a candidate, not the correct remedy.

Resolve the task in the relevant PID namespace and verify that it still refers to the intended process. Command lines, parent relationships, and open files provide context, but an open file is not proof that all observed writes went there. PIDs can be reused between measurements; a stale number can point at unrelated work.

Watch before you act

Observation has costs and access requirements. Additional syscall tracing may perturb the process and expose arguments, so choose a separately authorized, scoped investigation when needed. An append flag belongs to opening a file; it is not a flag on each write syscall. Avoid presenting an imagined combination of outputs as a measured diagnosis.

Record collection conditions for a useful baseline: implementation, interval, process/thread mode, accumulation mode, and comparable workload. With the documented default one-second delay, -b -n 60 is roughly a minute, not five minutes. Device await includes queueing and service time; near-100-percent utilization does not establish saturation on every parallel SSD or RAID device.

When it breaks

Missing permissions, kernel accounting options, or enabled task delay accounting can cause failure or unavailable fields. An empty display does not exonerate either your processes or other users’ processes. Record the actual limitation and required environment; do not silently change capabilities or kernel settings to make the tool work. Some delay-accounting changes affect only tasks started afterward.

Bursty activity can disappear in averages, and short-lived tasks may escape sampling. Varying intervals changes what is visible. Accumulated mode summarizes observed accounting within its applicable lifetime; it does not recover an unrecorded previous hour or guarantee complete coverage of exited tasks. State the observation window with every conclusion.

Compare task bytes, task waiting and device observations over the same interval. Capture and identity checks remain unexecuted; a fresh snapshot cannot prove identity at an earlier sample.

Check it quickly

This unexecuted proposal targets the cited iotop-c interface and sysstat iostat. Confirm the installed implementations first. It captures full reports, stderr and both statuses without selecting a process from text. A fresh configuration directory prevents saved iotop preferences from silently changing the view.

(
 capture=$(mktemp -d) || exit 1
 printf 'Evidence directory: %s\n' "$capture"
 iotop --version || exit $?
 iostat -V || exit $?
 LC_ALL=C iostat -x -y -t 2 5 > "$capture/iostat.txt" 2> "$capture/iostat.err" &
 stat_pid=$!
 if LC_ALL=C XDG_CONFIG_HOME="$capture" iotop -b -P -k -t -d 2 -n 5 \
 --no-only --no-accumulated --no-accum-bw \
 > "$capture/iotop.txt" 2> "$capture/iotop.err"; then
 task_status=0
 else
 task_status=$?
 fi
 if wait "$stat_pid"; then stat_status=0; else stat_status=$?; fi
 printf 'iotop_status=%s iostat_status=%s\n' "$task_status" "$stat_status"
 cat "$capture/iotop.err" "$capture/iostat.err"
 [ "$task_status" -eq 0 ] && [ "$stat_status" -eq 0 ] || exit 1
 cat "$capture/iotop.txt" "$capture/iostat.txt"
)

In iotop-c, -P hides separate thread rows; their activity is already included in process totals. Fixed kilobyte units and non-accumulated mode support comparisons. This batch capture does not promise descending write order. Compare the labelled DISK WRITE values within each report; for an interactive ranking, explicitly select that column and descending direction. Never treat the first printed row as the largest writer.

Compare overlapping timestamped intervals; launch times are not perfectly synchronized. -y excludes iostat’s since-boot report. Missing counters, nonzero status or incomplete windows invalidate the comparison.

Save this helper as inspect_io_process.py. Select a process PID manually from the full report, in the same PID namespace. Python 3.9+, Linux pidfd support and readable proc entries are required:

"""Read-only Linux identity snapshot; requires Python 3.9+ and pidfd support."""
import json
import os
import re
import select
import sys


def snapshot(token, proc_root="/proc", ops=os, poll_factory=select.poll):
 if not re.fullmatch(r"[1-9][0-9]*", token):
 raise ValueError("PID must be a positive decimal process ID")
 pid = int(token)
 pidfd = ops.pidfd_open(pid, 0)
 directory = None
 try:
 directory = ops.open(f"{proc_root}/{pid}", os.O_RDONLY | os.O_DIRECTORY)
 watcher = poll_factory()
 watcher.register(pidfd, select.POLLIN)

 def read(name, limit):
 fd = ops.open(name, os.O_RDONLY, dir_fd=directory)
 with ops.fdopen(fd, "rb") as source:
 data = source.read(limit + 1)
 if len(data) > limit:
 raise ValueError(f"{name} exceeds the snapshot limit")
 return data

 def identity():
 head, tail = read("stat", 65536).rsplit(b")", 1)
 if int(head.split(b" ", 1)[0]) != pid:
 raise ValueError("stat PID mismatch")
 fields = tail.split() # field 3 is index 0; field 22 is index 19
 if fields[0] in (b"Z", b"X", b"x"):
 raise ProcessLookupError("process has exited")
 return int(fields[19])

 before = identity()
 executable = ops.readlink("exe", dir_fd=directory)
 raw_command = read("cmdline", 65536)
 # Remove one final NUL; empty input means no argument data.
 command = raw_command[:-1] if raw_command.endswith(b"\0") else raw_command
 command = command.split(b"\0") if raw_command else []
 after = identity()
 if before != after or watcher.poll(0):
 raise ProcessLookupError("identity changed or process exited")
 return {"pid": pid, "starttime_ticks": before, "executable": executable,
 "argv": [arg.decode("utf-8", "backslashreplace") for arg in command]}
 finally:
 if directory is not None:
 ops.close(directory)
 ops.close(pidfd)


if __name__ == "__main__":
 try:
 if len(sys.argv) != 2:
 raise ValueError("usage: inspect_io_process.py PID")
 print(json.dumps(snapshot(sys.argv[1]), ensure_ascii=True))
 except (OSError, ValueError, IndexError, AttributeError) as error:
 sys.exit(f"Identity snapshot failed: {error}")

python3 inspect_io_process.py "${PID:?Set PID from the reviewed process row}"

A successful snapshot reports an executable, arguments and stable start-time value while checking a pidfd for exit. Permission, missing-process or unavailable-pidfd errors stop the check. PID reuse before acquiring the handle cannot be excluded; this snapshot does not prove identity at the earlier sample. Proc fields can change and are not atomic. Recheck before any later action; nothing is killed or reprioritized here.

Sources

iotop man page, rendered iotop-c manual; verify the installed variant, checked 2026-10-18.
vmstat man page, documents system-wide CPU and IO-wait reporting, checked 2026-09-05.
Linux delay accounting, taskstats delay scope and enablement prerequisites, checked 2026-10-18.
Linux proc filesystem, storage byte accounting, cancelled writes, and CPU iowait limits, checked 2026-10-18.
iostat manual, first report, await, and parallel-device utilization limits, checked 2026-10-18.
iotop-c upstream manual, process aggregation, batch options, configuration and interactive sorting, checked 2026-10-19.
sysstat upstream iostat manual, -y omits the since-boot report and -t timestamps interval reports, checked 2026-10-19.
pidfd_open manual, bounded identity or controlled-operation contract, checked 2026-10-19.
Python os interfaces, bounded identity or controlled-operation contract, checked 2026-10-19.
proc_pid_stat manual, bounded identity or controlled-operation contract, checked 2026-10-19.

Disk schedulers order I/O to move the head efficiently

Tue, 09 Mar 2027 00:00:00 +0000

blk-mq provides CPU- or node-local software staging queues and hardware dispatch queues. Scheduling and merging can occur before dispatch, while driver and device behavior also affect service. Software submission order does not guarantee completion order. A scheduler can influence how competing requests enter the device without controlling every later decision in the storage stack.

On rotating media, request locality can reduce head movement. Flash lacks a mechanical seek penalty, but merging, concurrency, queueing and arbitration still affect performance. Efficient ordering and fair service answer different questions, and neither determines a universal scheduler ranking. Choose from the schedulers actually available for the device and kernel, then measure the relevant workload.

The head is the budget

At 7200 RPM, half a revolution is about 4.17 ms, an idealized average rotational wait. Transfer and seek costs depend on the device and workload. The original text reported 2000 requests, 67,907,411 versus 99,977 tracks, and 91 versus 231 IOPS, but supplied neither the generating fixture nor retained measurements. Those figures remain unverified source claims and must not be presented as measured Linux scheduler results.

Sorting an invented set of track positions can illustrate less head travel, but it cannot establish an IOPS gain without a validated device model or measurement. The previously reported seek percentages and approximately 40 ms NVMe total have the same missing-evidence limitation. A drawing of a shorter path is therefore qualitative, not a latency prediction.

What each scheduler is for

none adds no scheduler reordering; it does not promise a global first-come completion order. mq-deadline combines sector ordering with expiry checks between bounded batches. These policies guide dispatch rather than guarantee an end-to-end completion deadline. kyber throttles requests toward read and synchronous-write latency targets; a target is not proof every request meets it.

bfq provides weight-based service and low-latency heuristics, with configuration-dependent bandwidth or time sharing. Idling can help service guarantees but can reduce throughput on internally queued devices. Kernel support and actual settings matter. Read the active scheduler from the bracketed name in /sys/block//queue/scheduler; changing it is an operational action requiring an authorized device and reversible test plan.

Fairness is a different question from order

A 90/10 issue mix does not establish a guaranteed service share or wait position. The reported 0.099, 1901 and 0.5 values have no retained fixture here. Equal weights in a simplified continuously backlogged model may illustrate sharing, but real outcomes depend on demand, request sizes, priorities and scheduler settings. Compare service received and application latency separately from request arrival counts.

For either media type, test latency, throughput and isolation under representative contention. cgroup io.max delays I/O when configured BPS or IOPS limits are reached and permits temporary bursts; it is not conditional on choosing BFQ. Weight control depends on the active implementation and configuration; BFQ has io.bfq.weight, while io.weight can use the I/O cost controller. A ceiling and a relative share are different controls.

When it breaks

A rotational device using none is a reason to inspect the workload, not proof of a bug. High utilization, low IOPS and high await can have several causes. await includes queueing and service, while near-100% utilization can indicate saturation for a serial device and is less informative for parallel hardware. Confirm topology, request pattern and competing work before attributing delays to seeking.

A bulk job can worsen interactive tail latency even when averages look acceptable. Scheduler policies or cgroup limits may help, but no single name establishes application-level protection. Compare percentiles and useful throughput before and after a controlled change. Idling, queue depth and device-internal scheduling interact; the rotational flag alone does not select a universally correct setting.

Queue ordering, weighted service and io.max ceilings are distinct; the original numerical claims remain unverified.

Check it quickly

Start with this unexecuted read-only inspection of one explicitly selected whole device. Set DEVICE only after reviewing its topology and scope. It rejects path-like names, checks required readable attributes, and reports absent optional tunables. This first block does not change policy.

(
 : "${DEVICE:?Set DEVICE to the reviewed whole-device name}"
 case "$DEVICE" in
 ''|[!A-Za-z]*|*[!A-Za-z0-9_-]*|ALL) printf 'Invalid device name\n' >&2; exit 2 ;;
 esac
 queue="/sys/block/$DEVICE/queue"
 [ -d "$queue" ] || { printf 'No such whole-device queue\n' >&2; exit 1; }
 for field in rotational scheduler; do
 [ -r "$queue/$field" ] || { printf 'Missing: %s\n' "$field" >&2; exit 1; }
 value=$(cat "$queue/$field") || exit 1
 printf '%s=%s\n' "$field" "$value"
 done
 for field in read_expire fifo_batch; do
 if [ -r "$queue/iosched/$field" ]; then
 value=$(cat "$queue/iosched/$field") || exit 1
 printf '%s=%s\n' "$field" "$value"
 else
 printf '%s: unavailable for the active scheduler\n' "$field"
 fi
 done
 LC_ALL=C iostat -x -y -t "$DEVICE" 2 5
)

The bracketed name is current; absent iosched fields can be normal. -y excludes since-boot statistics. Preserve errors and confirm the selected device row.

Save this helper as scheduler_exercise.py. Only on an explicitly authorized, exclusively controlled disposable test device, set EXPECTED_CURRENT and REQUESTED from the observed choices, and ACK_DEVICE to DISPOSABLE: followed by DEVICE. Python 3.9+, writable scheduler access and sysstat are prerequisites. This temporarily changes policy:

"""Unexecuted proposal: temporary policy change on an exclusively controlled test device."""
import json
import re
import signal
import subprocess
import sys
from pathlib import Path


def device_name(value):
 if not re.fullmatch(r"[A-Za-z][A-Za-z0-9_-]*", value) or value == "ALL":
 raise ValueError("use one named whole device, not a path, option, number or ALL")
 return value


def scheduler_state(path):
 tokens = path.read_text().split()
 active = [s[1:-1] for s in tokens if re.fullmatch(r"\[[A-Za-z0-9_-]+\]", s)]
 if len(active) != 1 or not tokens:
 raise ValueError("scheduler state must have exactly one bracketed choice")
 if any(not re.fullmatch(r"(?:[A-Za-z0-9_-]+|\[[A-Za-z0-9_-]+\])", s) for s in tokens):
 raise ValueError("unexpected scheduler state")
 available = [s.strip("[]") for s in tokens]
 if len(set(available)) != len(available):
 raise ValueError("duplicate scheduler choices")
 return active[0], available


def exercise(path, device, expected, requested, run=subprocess.run, emit=print):
 original, available = scheduler_state(path)
 if original != expected or requested not in available or requested == original:
 raise ValueError("current/requested scheduler does not match the reviewed plan")
 emit(json.dumps({"device": device, "original": original, "requested": requested}), flush=True)
 primary_error = restore_error = None
 attempted = False
 try:
 if scheduler_state(path) != (original, available):
 raise ValueError("scheduler state changed before the test")
 attempted = True # even a failed write may have changed state
 path.write_text(requested + "\n")
 if scheduler_state(path)[0] != requested:
 raise RuntimeError("requested scheduler was not activated")
 result = run(["iostat", "-x", "-y", "-t", device, "2", "5"], check=False)
 if result.returncode != 0:
 raise RuntimeError(f"iostat exited with status {result.returncode}")
 except BaseException as error:
 primary_error = f"{type(error).__name__}: {error}"
 finally:
 if attempted:
 try:
 if original not in scheduler_state(path)[1]:
 raise RuntimeError("original scheduler is no longer available")
 path.write_text(original + "\n")
 if scheduler_state(path)[0] != original:
 raise RuntimeError("original scheduler was not restored")
 except BaseException as error:
 restore_error = f"{type(error).__name__}: {error}"
 emit(json.dumps({"primary_error": primary_error, "restore_error": restore_error,
 "restored": attempted and restore_error is None}), flush=True)
 # Report both errors; restoration failure never hides the primary failure.
 return (1 if primary_error else 0) + (2 if restore_error else 0)


def main(args):
 if len(args) != 4:
 raise ValueError("usage: scheduler_exercise.py DEVICE EXPECTED_CURRENT REQUESTED DISPOSABLE:DEVICE")
 device, expected, requested, acknowledgement = args
 device = device_name(device)
 if acknowledgement != "DISPOSABLE:" + device:
 raise ValueError("explicit disposable-device acknowledgement required")
 path = Path("/sys/block") / device / "queue/scheduler"
 return exercise(path, device, expected, requested)


if __name__ == "__main__":
 def interrupted(number, frame):
 raise InterruptedError(f"received signal {number}; attempting restoration")
 signal.signal(signal.SIGINT, interrupted)
 signal.signal(signal.SIGTERM, interrupted)
 try:
 sys.exit(main(sys.argv[1:]))
 except (OSError, ValueError) as error:
 sys.exit(f"Scheduler test refused: {error}")

python3 scheduler_exercise.py "${DEVICE:?}" "${EXPECTED_CURRENT:?}" \
 "${REQUESTED:?}" "${ACK_DEVICE:?}"

The helper records the original, verifies activation, captures intervals, then separately restores and verifies the original. Exit statuses are 0 for success, 1 for primary failure, 2 for restoration failure, and 3 for both; the JSON retains both errors. Do not ignore a restore error. SIGINT/SIGTERM request cleanup, but SIGKILL, repeated interruption or host failure can prevent it. Retain the original for manual recovery. This is a reversible policy exercise, not performance or fairness evidence; numerical claims and actual execution remain held.

Sources

Switching Scheduler, the scheduler names, sysfs paths and how to change them, checked 2026-09-16.
Multi-Queue Block IO Queueing Mechanism (blk-mq), software staging queues versus hardware dispatch queues, checked 2026-09-16.
BFQ (Budget Fair Queueing), proportional-share behaviour, cgroup support and the idling trade-off, checked 2026-09-16.
Kyber I/O scheduler tunables, the read and write latency targets Kyber throttles against, checked 2026-09-16.
Linux deadline tunables, correction reference, checked 2026-10-18.
Linux cgroup v2 I/O controls, correction reference, checked 2026-10-18.
iostat manual, correction reference, checked 2026-10-18.
sysstat upstream iostat manual, -y omits the since-boot report and -t timestamps interval reports, checked 2026-10-19.
Python signals, bounded identity or controlled-operation contract, checked 2026-10-19.
sysstat iostat option parser, bounded identity or controlled-operation contract, checked 2026-10-19.