Memory · XB Field Notes

Postgres uses two cache layers

Thu, 27 Jan 2028 00:00:00 +0000

For ordinary buffered relation I/O, PostgreSQL’s shared buffers sit above the operating system’s file cache. A shared-buffer hit avoids another read through that lower layer; a shared-buffer miss can still be served from memory by the operating system. The same file data can occupy both caches, so this is not a promise that PostgreSQL and the kernel share one physical copy. The mmap shared-memory settings concern shared regions, not a claim that ordinary table files are directly memory-mapped.

When it breaks

An EXPLAIN buffer count labelled shared read means the block was brought into PostgreSQL’s buffer pool. It does not establish that a storage device performed a physical read. Treating every shared read as a disk miss can make a warm operating-system cache look cold and lead to the wrong memory-sizing decision. Likewise, a faster second query does not identify which cache changed by itself.

Cache residency is temporary. Other queries can evict pages, and a scan larger than the available cache cannot keep every page warm indefinitely. A relation may use a scan strategy that changes buffer reuse. Test with a known working set, inspect the access path, and keep timing claims separate from the cache-layer facts actually observed.

Postgres has two cache layers — mechanism diagram.

Check it quickly

Use a disposable PostgreSQL 18 instance as a superuser, with the supplied pg_buffercache and pg_prewarm extensions available but not already installed. A retained earlier run used a PostgreSQL 18.6 standalone backend with 128 MB of shared buffers. This source review did not rerun it; psql transport and concurrency remain untested. The relation below is small enough to fit comfortably. Run blocks in one fresh psql -X -v ON_ERROR_STOP=1 session. The eviction function is for developer testing; restrict it to this isolated fixture.

CREATE SCHEMA cache_layer_demo;
CREATE EXTENSION pg_buffercache WITH SCHEMA cache_layer_demo;
CREATE EXTENSION pg_prewarm WITH SCHEMA cache_layer_demo;
CREATE TABLE cache_layer_demo.items (id integer PRIMARY KEY,payload text)
 WITH (autovacuum_enabled=false);
INSERT INTO cache_layer_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(1,12000) AS g;
VACUUM (ANALYZE) cache_layer_demo.items;
CREATE VIEW cache_layer_demo.residency AS
SELECT count(*) AS heap_buffers FROM cache_layer_demo.pg_buffercache
WHERE reldatabase=(SELECT oid FROM pg_database WHERE datname=current_database())
 AND relfilenode=pg_relation_filenode('cache_layer_demo.items')
 AND relforknumber=0;
SET max_parallel_workers_per_gather=0;
SELECT pg_relation_size('cache_layer_demo.items')/current_setting('block_size')::int AS heap_blocks;
SELECT heap_buffers>0 AS initially_buffered FROM cache_layer_demo.residency;

The heap contains 546 blocks in the tested build, and the initial residency comparison is true. The view filters by database, file identifier, and main fork. It assumes this fresh fixture uses only its default tablespace: general monitoring must also distinguish tablespaces, because a filenode alone is not a database-wide physical identity. Buffer observations are individually consistent, not one atomic cache snapshot. Next, evict only this relation’s shared buffers:

SELECT * FROM cache_layer_demo.pg_buffercache_evict_relation('cache_layer_demo.items'::regclass);
SELECT heap_buffers=0 AS no_shared_heap_buffers FROM cache_layer_demo.residency;

The function reports evicted, flushed, and skipped buffers. This run returned 550, 550, and zero; the total includes auxiliary forks, explaining why it exceeds the heap-only count. The following comparison confirms no heap buffers remain. It says nothing about whether the operating system still caches those file pages.

Read the heap through the operating system without loading PostgreSQL shared buffers:

SELECT cache_layer_demo.pg_prewarm('cache_layer_demo.items','read','main');
SELECT heap_buffers=0 AS still_no_shared_heap_buffers FROM cache_layer_demo.residency;

The prewarm call processes 546 blocks, and the shared-buffer count remains zero. Its read mode performs synchronous reads; buffer mode would instead load the database buffer pool. Now execute a query that needs every payload:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The sequential scan reports 546 shared reads and zero shared hits. Those reads are compatible with an already warm operating-system cache; they are not evidence of 546 device operations. Repeat the same query:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The second scan reports zero shared reads and 546 shared hits. Both scans process twelve thousand rows, so the change in buffer accounting is not a reduction in the logical input. Finally, confirm residency and the aggregate result, then remove the fixture and its extensions:

SELECT heap_buffers>0 AS now_shared FROM cache_layer_demo.residency;
SELECT sum(length(payload)) FROM cache_layer_demo.items;
RESET max_parallel_workers_per_gather;
DROP SCHEMA cache_layer_demo CASCADE;

The result is 3,840,000 payload characters and a true shared-residency comparison. Exact block counts depend on row layout and build parameters. The meaningful observations are the empty shared pool after read-mode prewarming, the subsequent shared reads, and the repeated scan’s hits. They do not measure physical-device latency or guarantee a particular speedup.

Budget memory for the complete server

The PostgreSQL resource guide suggests roughly a quarter of system memory as an initial shared-buffer setting for a dedicated server with sufficient RAM. That is a starting point, not a universal optimum. Connection memory, concurrent sorts and hashes, maintenance, and the operating-system cache all consume the same host budget.

Measure representative working sets and concurrent workloads before changing the allocation. Warm-cache microbenchmarks can hide I/O pressure, while cold-cache tests can exaggerate a workload that normally reuses pages. Keep cache-layer counters, actual device observations, and query results together so a configuration change is credited for the effect it actually produced.

Sources

PostgreSQL 18 resource consumption, shared buffers and operating-system cache budgeting, checked 2026-10-15.
PostgreSQL 18 pg_prewarm, read versus buffer prewarming modes, checked 2026-10-15.
PostgreSQL 18 pg_buffercache, buffer identity and development-only relation eviction, checked 2026-10-15.
PostgreSQL file layout, database, tablespace, and relation-file identity, checked 2026-10-15.

Redis expiry is a deadline, eviction is a capacity policy

Sun, 26 Dec 2027 00:00:00 +0000

Expiration and eviction can both turn a successful lookup into a miss, but they answer different questions. Expiration follows a key’s deadline. Eviction removes eligible keys to respond to memory pressure. Extending a TTL cannot prevent capacity-driven eviction, and increasing memory cannot correct a deadline that the application set incorrectly.

Time and capacity are separate axes

A volatile key has an expiration attached to it. Redis checks expired keys during access and through active expiration work. This does not promise that the key survives until its deadline: explicit deletion, replacement, and an eligible eviction policy can change the outcome earlier. A cache client must handle misses regardless of how far away the original expiration appeared.

Maxmemory selects an accounting limit used by Redis’s memory-management decisions. It is not an operating-system RSS ceiling. Replication and append-only buffers can include memory excluded from eviction accounting; process overhead, allocator behavior, and persistence work also need capacity headroom. A large command can temporarily exceed the configured limit. Size the process environment as well as the cache budget.

Select candidates before debating ranking

Allkeys policies consider keys regardless of whether they have TTLs. Volatile policies restrict candidates to keys with expiration. Within those sets, LRU favors keys that have gone longer without access, LFU uses frequency information, and random policies select randomly. Redis approximates LRU and LFU rather than promising one globally exact victim order. Volatile-ttl favors shorter remaining lifetimes, not keys that have necessarily already expired.

Noeviction declines memory-growing commands when the limit cannot be accommodated instead of discarding keys for capacity. Reads can still succeed. It is not a durability guarantee: ordinary expiration, explicit deletion, process failure, and persistence configuration remain separate. Inspect the actual configured policy instead of assuming a universal deployment or cluster default.

A volatile policy without expiring candidates behaves like noeviction for this purpose. Merely choosing volatile-lru does not cause permanent keys to become disposable. Conversely, putting session or authoritative records beside rebuildable cache entries under an allkeys policy makes those records eligible too. A key prefix is a naming convention, not an eviction boundary.

When it breaks

A sudden miss spike can reflect expiration, eviction, invalidation, or changed key construction. Compare expired_keys and evicted_keys deltas with the request workload and command outcomes. These are cumulative counters, so compare a bounded interval and account for restarts or resets. A rising eviction counter alone does not reveal which application feature lost data.

Watch refused writes as well as reads. A warm cache can keep returning plausible values while its writers receive memory errors. Ignoring those errors turns stale data into an application consistency defect. Increasing TTL may then make existing values older without fixing the inability to write replacements. Retry policy must not turn memory pressure into an unlimited request storm.

Separating authoritative state from a disposable cache is an architectural decision, not a command to migrate data during an incident. Verify recoverability, persistence, and ownership before changing an existing instance’s policy. The exercise below changes configuration only inside processes it creates and tests three candidate policies without changing any production keyspace.

Expiration and eviction act on separate axes. The historical policy exercise shows relationships, not portable key counts or a production capacity limit; maxmemory is not an OS RSS cap.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. Build core Redis with libc and install the pinned client locally. The source and package downloads need the network; no system installation or optional bundled-module build is performed.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Execute the Python block with "$redis_lab/venv/bin/python". Each policy gets its own temporary Unix socket, empty dataset, and process with TCP and persistence disabled. The memory limit is the measured initial allocation plus two mebibytes. Bounded writes use 64-KiB values; TTLs are one hour where enabled. Every process and its runtime directory are cleaned up afterward.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


for policy, expiring in [("allkeys-lru", True), ("noeviction", True),
 ("volatile-lru", False)]:
 with instance(policy) as client:
 baseline = client.info("memory")["used_memory"]
 client.config_set("maxmemory", baseline + 2 * 1024 * 1024)
 payload = "x" * 65536
 written, rejected = 0, False
 for index in range(128):
 try:
 client.set(f"item:{index}", payload, ex=3600 if expiring else None)
 except redis.exceptions.OutOfMemoryError:
 rejected = True
 break
 written += 1
 stats = client.info("stats")
 remaining = client.dbsize()
 assert stats["expired_keys"] == 0
 if policy == "allkeys-lru":
 assert written == 128 and not rejected
 assert stats["evicted_keys"] > 0 and remaining < written
 assert client.ttl("item:127") > 0
 else:
 assert rejected and 0 < written < 128
 assert stats["evicted_keys"] == 0 and remaining == written
 assert client.get("item:0") == payload
 if not expiring:
 assert client.ttl("item:0") == -1
 print(policy, {"written": written, "remaining": remaining,
 "evicted": stats["evicted_keys"],
 "expired": stats["expired_keys"], "rejected": rejected})

In this run, allkeys-lru accepted 128 writes, retained 23 keys, and reported 105 evictions with zero expirations. The newest surviving key still had a positive TTL. This establishes capacity-driven loss before expiration without claiming an exact LRU victim sequence.

Noeviction accepted 23 writes before rejecting the next allocation; existing reads still worked and no keys were evicted. Volatile-lru with no expiring keys produced the same refusal behavior. Exact counts depend on allocation overhead and the build, so assertions check relationships rather than those particular counts. This is not a throughput benchmark or evidence about replication, failover, persistence peaks, or an operating-system memory limit.

Sources

Redis key eviction, maxmemory accounting, candidate policies, and approximate ranking, checked 2026-09-08.
Redis EXPIRE, per-key deadlines and expiration behavior, checked 2026-09-08.
Redis INFO, memory accounting and eviction/expiration counters, checked 2026-09-08.
Redis 8.10.1 build instructions, isolated core build and allocator selection, checked 2026-09-08.

Huge pages trade allocation flexibility for translation reach

Tue, 21 Dec 2027 00:00:00 +0000

Huge pages can increase the amount of memory covered by each cached address translation. They can reduce translation overhead for suitable workloads, but the result depends on access patterns and hardware. A large allocation alone does not prove frequent TLB misses, and fewer pages needed to describe a working set does not mean those translations all fit in the processor’s TLB.

Page size is an input

A 4 KiB base page is common on x86, but Linux page sizes depend on architecture and configuration. Query the running system rather than treating four kilobytes as universal. Supported huge-page sizes also vary; 2 MiB and 1 GiB are familiar examples, not a complete list for every processor.

For an idealized contiguous 8 GiB range, 4 KiB pages require 2,097,152 page-sized mappings, while 2 MiB pages require 4,096. That arithmetic describes coverage. It does not measure hardware TLB capacity, miss rate, page-walk latency, or the number of translations simultaneously resident. Different translation structures can also have different capacities for different page sizes.

Explicit pools and transparent allocation

Hugetlb provides explicitly requested huge pages backed by managed pools. Its persistent pages are unavailable for ordinary allocations and are not swapped out. Pools can be allocated at boot or adjusted later; surplus pages may also be allowed. Reserving at boot often makes contiguous allocation easier, but is not the only supported lifecycle.

Transparent Huge Pages use a different mechanism. Depending on kernel support and policy, larger anonymous-memory allocations can occur on page faults or through later collapse. THP is not merely a background process and does not require the administrator to provision a hugetlb pool. Modern kernels can support multiple THP sizes; do not equate every transparent allocation with one 2 MiB mapping.

The flexibility differs too. THP can fall back to smaller pages and can be split for operations such as reclaim. An explicit hugetlb mapping does not silently become an ordinary-page mapping; an application may separately choose a fallback allocation when its configuration permits. Distinguish the kernel mapping contract from the application’s optional behavior.

Budget capacity and placement

In /proc/meminfo, HugePages_Total describes the default-size pool, HugePages_Free counts currently unallocated pages, and HugePages_Rsvd counts commitments not yet allocated at fault time. Total multiplied by page size is pool capacity, not the amount promised by Rsvd. Reservations can account for part of the apparently free pool. Hugetlb covers all supported pool sizes, while per-size sysfs directories provide finer detail.

A sparse mapping can consume more resident memory than its useful data warrants. Pool capacity also competes with ordinary memory needs. Size from application requirements, verify actual mappings, and leave room for other allocations. Benefits should be demonstrated with representative throughput, latency, and translation measurements rather than inferred from a large configured pool.

When it breaks

Growing a pool can fail because the kernel cannot find suitably contiguous physical memory. Existing free huge pages are already contiguous, so a nonzero free counter is not itself evidence of fragmentation. Wrong page size, reservations, permissions, NUMA placement, or cgroup limits can instead explain why a particular application cannot use apparently available capacity.

NUMA policy can place pages away from the CPUs using them or restrict eligible nodes. Examine per-node allocation and access locality before blaming translation overhead. A translation-cache improvement can coexist with worse memory latency, and neither a global free counter nor a successful startup establishes good placement.

THP allocation, collapse, compaction, and splitting may affect latency differently. thp_collapse_alloc counts successful allocations during collapse; it is not a stall-duration counter. Correlate appropriate counter deltas with latency. Choosing madvise, always, or another supported per-size policy requires workload evidence, not a blanket rule that every database should use the same mode.

The 8 GiB page-count comparison is arithmetic; hugetlb reservations and THP behavior require separate interpretation.

Check it quickly

Save this as hugepage_check.py and run python3 hugepage_check.py. It reads counters and page size without allocating a huge-page workload or modifying kernel settings. Missing data produces an explicitly labeled arithmetic illustration. Parser and boundary assertions use synthetic strings.

import os
import re
from pathlib import Path

WORKING_SET = 8 * 1024**3
FIELDS = ("HugePages_Total", "HugePages_Free", "HugePages_Rsvd",
 "HugePages_Surp", "Hugepagesize")


def parse_meminfo(text):
 values = {}
 for key in FIELDS:
 unit = r"\s+kB" if key == "Hugepagesize" else ""
 match = re.search(r"^" + key + r":\s+(\d+)" + unit + r"\s*$", text, re.M)
 if match:
 values[key] = int(match.group(1))
 return values


def pages_for(size, page):
 if size < 0 or page <= 0:
 raise ValueError("invalid byte count or page size")
 return (size + page - 1) // page


assert pages_for(4097, 4096) == 2
assert parse_meminfo("HugePages_Total: 8\nHugepagesize: 2048 kB\n")["Hugepagesize"] == 2048
assert parse_meminfo("HugePages_Total: bad\nHugepagesize: 2 MB\n") == {}
try:
 pages_for(1, 0)
except ValueError:
 pass
else:
 raise AssertionError("zero page size accepted")

try:
 values = parse_meminfo(Path("/proc/meminfo").read_text())
except OSError as error:
 values = {}
 print("SKIP live hugetlb counters:", type(error).__name__)
try:
 base = os.sysconf("SC_PAGE_SIZE")
 assert base > 0
 print("Observed base page size:", base, "bytes")
except (AttributeError, ValueError, OSError):
 base = 4096
 print("Assumed illustration base page size:", base, "bytes")

if all(key in values for key in FIELDS) and values["Hugepagesize"] > 0:
 huge = values["Hugepagesize"] * 1024
 print("Default-size hugetlb pool counters:", values)
 print("Total pool capacity:", values["HugePages_Total"] * huge, "bytes")
 print("Reserved but not allocated:", values["HugePages_Rsvd"] * huge, "bytes")
else:
 huge = 2 * 1024**2
 print("SKIP complete pool report; assuming 2 MiB for arithmetic only")
print("8 GiB page-count illustration:", pages_for(WORKING_SET, base),
 "base pages versus", pages_for(WORKING_SET, huge), "large pages")
print("No TLB miss rate, THP coverage, allocation success, or latency was measured")

With 4 KiB base pages and a 2 MiB default huge-page size, expect the two page counts above. Zero pool totals are valid observations, not proof that the kernel lacks support. This revised script passed with Python 3.12.14 on Linux. It measures neither THP coverage nor performance; those require a separate, workload-specific experiment.

Sources

Kernel hugepages documentation, defines hugetlb settings and behavior, checked 2026-06-01.
Kernel transparent hugepages documentation, defines THP modes and compaction behavior, checked 2026-06-01.
proc_meminfo: huge-page counters, checks commitment and huge-page counters with units, checked 2026-10-15.

Linux overcommit budgets virtual-memory commitments

Sat, 18 Dec 2027 00:00:00 +0000

Linux overcommit policy controls admission of certain virtual-memory commitments. It is separate from allocating resident physical pages, choosing an OOM victim, and enforcing a container’s memory limit. A successful allocation can precede physical demand, but that does not make all allocations free or guarantee that later accesses will succeed under every resource constraint.

Virtual commitment differs from residence

An allocator such as malloc may obtain memory through brk or mmap and may reuse an existing arena without making a new system call. Anonymous pages are commonly populated lazily. Address-space metadata, page tables, and actual page faults still consume resources; the cost depends on what the application requests and touches.

Committed_AS is an accounting quantity, not the process resident-set sum and not a measurement of current pressure. Mapping type matters: private writable mappings can require commitment even before every page becomes resident, while ordinary shared or read-only file-backed mappings have different accounting. A large virtual-size figure cannot by itself identify the application responsible for a physical-memory shortage.

Three admission policies

Mode 0 is the default heuristic policy. It rejects obvious overcommits while allowing allocations that would exceed a simple resident-memory budget. Its decision is not a promise that every accepted byte has dedicated RAM. Mode 1 skips the overcommit admission check, which can support sparse workloads, but address-space limits, mapping limits, and other allocation failures still apply.

Mode 2 enforces stricter commitment accounting. The nominal CommitLimit is derived from swap plus a configured amount of RAM: ratio-based calculation excludes hugetlb-reserved memory, while a nonzero overcommit_kbytes supplies an absolute allowance instead of the ratio. Kernel reserves and the calling context can further affect admission. The displayed difference from Committed_AS is therefore useful context rather than a universal maximum guaranteed allocation.

A chargeable allocation may fail with ENOMEM when it cannot fit that budget. This can expose capacity problems earlier, including commitments needed by fork and later permission changes. Fork accounting follows applicable mappings and copy-on-write obligations; it is not simply twice the parent’s resident set. Some applications deliberately reserve sparse address space, so changing policy can change their startup behavior.

Strict accounting is not OOM immunity

Mode 2 reduces one class of overcommit risk, but does not override cgroup memory limits, NUMA constraints, other kernel allocations, or all later memory demands. A task can still encounter allocation failure or an OOM event. Do not promise that choosing mode 2 protects a particular worker from being killed.

OOM victim selection is a separate policy using the relevant scope and eligibility rules, not random punishment and not a decision made by the CPU scheduler. The killed process need not be the task that most recently requested memory. Inspect the event’s host or cgroup context before attributing responsibility from the victim’s name alone.

Containers share the host kernel and should not be assumed to own an independent vm.overcommit_memory policy. Their memory controls add another boundary. Virtual machines have their own guest kernel accounting, while host backing and ballooning introduce additional constraints. Record which layer each metric belongs to before comparing budgets.

When it breaks

A workload can successfully reserve memory and then encounter reclaim, swap activity, or OOM when demand grows. Neither mode 0 nor mode 1 guarantees that the failure will be slow, that swap is available, or that a particular process dies. Compare demand and latency with actual reclaim and memory-pressure evidence.

Conversely, mode 2 can reject a request despite apparently free RAM because the commitment budget is already occupied. That is one possible cause of ENOMEM, not a diagnosis from the error alone. Process resource limits, excessive mapping counts, and application configuration deserve separate checks.

Large nominal headroom does not prove the ratio is too conservative, and small headroom does not automatically justify increasing it. Budget changes can admit demand that later competes for real resources. Plan from workload requirements and recovery margin, preserve configuration context, and test changes outside production before rollout.

Overcommit controls commitment admission; accepted requests still face later physical and cgroup constraints.

Check it quickly

Save this as overcommit_probe.py and run python3 overcommit_probe.py. It reads only the named proc files, prints the observed mode and accounting fields, and performs synthetic parser checks. It does not reserve memory, trigger OOM, or suggest automatic tuning.

from pathlib import Path
import re


def read_value(path):
 try:
 return Path(path).read_text(encoding="utf-8").strip()
 except OSError as error:
 print("SKIP", path, type(error).__name__)
 return None


def meminfo_kb(text):
 return {name: int(value) for name, value in re.findall(
 r"^(MemTotal|SwapTotal|CommitLimit|Committed_AS):\s+(\d+)\s+kB\s*$",
 text, re.M)}


def report(mode, info):
 names = {"0": "heuristic", "1": "always overcommit", "2": "strict accounting"}
 lines = ["mode: " + names.get(mode, "unknown or unreadable")]
 limit, committed = info.get("CommitLimit"), info.get("Committed_AS")
 if limit is not None and committed is not None:
 lines.append("nominal commit headroom: %s kB" % (limit - committed))
 else:
 lines.append("commit headroom unavailable")
 if mode == "2":
 lines.append("Commit accounting can reject chargeable requests; other limits still apply")
 else:
 lines.append("CommitLimit is not the enforced mode-2 ceiling in this mode")
 lines.append("This snapshot is not available RAM, a cgroup budget, or OOM immunity")
 return lines


fixture = meminfo_kb("CommitLimit: 100 kB\nCommitted_AS: 0 kB\n")
assert "nominal commit headroom: 100 kB" in report("2", fixture)
assert meminfo_kb("CommitLimit: bad kB\nCommitted_AS: 7 MB\n") == {}
assert "commit headroom unavailable" in report("2", {})
assert report("9", {})[0] == "mode: unknown or unreadable"
for mode in ("0", "1", "2"):
 assert "OOM immunity" in report(mode, fixture)[-1]

mode = read_value("/proc/sys/vm/overcommit_memory")
ratio = read_value("/proc/sys/vm/overcommit_ratio")
kbytes = read_value("/proc/sys/vm/overcommit_kbytes")
text = read_value("/proc/meminfo")
if mode is None or text is None:
 print("SKIP live overcommit report; synthetic checks passed")
else:
 info = meminfo_kb(text)
 print("overcommit_memory=%s overcommit_ratio=%s overcommit_kbytes=%s" % (mode, ratio, kbytes))
 print("Observed kB fields:", info)
 for line in report(mode, info):
 print(line)

Expect a live snapshot or an explicit SKIP message. Zero committed memory remains a valid value. The revised example passed with Python 3.12.14 on Linux; synthetic checks cover malformed input, absent fields, and all three modes. Reading these counters does not test real allocation rejection or recovery.

Sources

Kernel vm sysctl documentation, defines the overcommit_memory modes and defaults, checked 2026-06-01.
proc manual page, documents /proc/meminfo CommitLimit and Committed_AS accounting, checked 2026-06-01.
Kernel overcommit accounting rules, checks policy modes and charged mappings, checked 2026-10-15.
proc_meminfo: CommitLimit and Committed_AS, checks commitment and huge-page counters with units, checked 2026-10-15.
Kernel cgroup v2 memory controls, checks local memory limits and OOM boundaries, checked 2026-10-15.

A page fault is not always a storage read

Sat, 13 Nov 2027 00:00:00 +0000

A page fault means the processor could not complete an access using the current mapping or permissions. It is not synonymous with a storage read, and some faults represent invalid accesses rather than recoverable demand paging. Linux’s minor and major counters distinguish whether handling required page-in I/O. Interpret those counters alongside the mapping type and workload; neither is a direct measurement of all application storage traffic.

Minor does not mean free

A minor fault can map an already resident file page, allocate and zero anonymous memory, or copy a page for a copy-on-write mapping. The kernel may need memory allocation, page-table work, copying, and synchronization. Describing it as only adding a pointer misses those costs. A sufficiently intense minor-fault workload can consume meaningful CPU time and affect latency even without storage reads.

The first application access does not necessarily produce a fresh fault for each base page. Prefaulting, existing mappings, readahead-related mapping behavior, and large pages can change the count. The operating system’s page size is not universally 4096 bytes. An example should obtain the local value and avoid claiming an exact fault count from the number of loop iterations.

Major describes a different path

A major fault indicates handling involving page-in I/O, often for file-backed data or swapped-out memory. Storage can include devices or remote filesystems; the label is not a physical-disk latency benchmark. One fault can initiate readahead that serves later accesses, so a count is not a one-to-one count of pages or disk requests. Conversely, ordinary file reads can cause I/O without producing corresponding major faults in the application.

File-cache residency can allow a new process to incur minor faults even when it has never touched that file. A process restart is not equivalent to dropping the host’s page cache. Anonymous first-write faults commonly allocate zero-filled memory without reading old payloads from storage, but pressure and concurrent activity can still affect measurements. “Anonymous” is not a universal promise that the entire process’s major counter remains unchanged.

When it breaks

A cold-start latency spike can correlate with major faults, but do not prescribe warming before establishing what is missing. Pre-touching unnecessary pages consumes memory and can displace useful cache. Staggered starts may help a measured shared bottleneck; another workload may instead need smaller mappings, a different access pattern, or more memory headroom. Fault counts alone do not choose among these remedies.

Sustained swapping can repeatedly return evicted pages to the working set and increase latency. Adding more workers can worsen that situation by increasing memory demand. Correlate faults with swap activity, reclaim, storage waits, and request latency. A rising major count is evidence to investigate, not a verdict that the storage device is faulty or that every cache miss was avoidable.

Per-process counters also include work outside the line of code being measured. Interpreter activity, startup, and other threads can contribute. A residency probe is only a snapshot, and access immediately afterward can encounter a different state. Avoid dropping global caches or increasing a test allocation until it threatens unrelated work just to obtain a dramatic graph.

Minor faults, major faults and invalid accesses follow different paths; the anonymous-memory example does not exercise every path.

Check it quickly

Save this as fault_probe.py and run python3 fault_probe.py on Linux with Python 3.9 or newer and sufficient spare memory. It writes one byte per native page through a private anonymous mapping of sixteen MiB, then repeats the pass. The mapping is closed automatically, and no file, global cache setting, or memory limit is changed.

import mmap
import resource
import sys
import time

if not sys.platform.startswith("linux"):
 raise SystemExit("This exercise requires Linux getrusage semantics")
PAGE = mmap.PAGESIZE
SIZE = 16 * 1024 * 1024

def counters():
 value = resource.getrusage(resource.RUSAGE_SELF)
 return value.ru_minflt, value.ru_majflt

def touch(region):
 for offset in range(0, len(region), PAGE):
 region[offset] = 1

with mmap.mmap(-1, SIZE, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS,
 prot=mmap.PROT_READ | mmap.PROT_WRITE) as region:
 before = counters()
 start = time.perf_counter()
 touch(region)
 first_seconds = time.perf_counter() - start
 middle = counters()
 start = time.perf_counter()
 touch(region)
 second_seconds = time.perf_counter() - start
 after = counters()
first = tuple(b - a for a, b in zip(before, middle))
second = tuple(b - a for a, b in zip(middle, after))
print(f"page_size={PAGE} mapping_bytes={SIZE}")
print(f"first: minor={first[0]} major={first[1]} seconds={first_seconds:.6f}")
print(f"second: minor={second[0]} major={second[1]} seconds={second_seconds:.6f}")
assert first[0] > 0, "expected demand-fault activity"
assert second[0] < first[0], "repeat pass did not show reduced minor faults"
assert first[1] == second[1] == 0, "observed major faults: this run is not a no-major example"
print("PASS: this run had fewer second-pass minor faults and no measured major faults")

The test expects demand-fault activity on the first pass, fewer minor faults on the second, and no measured majors in this particular run. Those assertions passed for both exact language examples during this review. Counts and durations are observations, not fixed targets. The script does not assert that the second pass is faster; timers and scheduling can vary.

A failed assertion means this run did not establish the expected contrast. Inspect its memory conditions and counters rather than silently deleting the check. The successful result shows one anonymous-memory case where fault activity occurred without measured page-in I/O. It does not demonstrate a file-backed major fault, establish universal fault latency, or replace measurements of the real application under representative pressure.

Sources

Linux getrusage, minor and major fault accounting, checked 2026-10-15.
Linux mmap, private anonymous mappings and population behavior, checked 2026-10-15.
Linux memory concepts, demand paging, page cache, reclaim and huge pages, checked 2026-10-15.
Linux process stat, process fault counters and their scope, checked 2026-10-15.

Redis memory ratios need allocation context

Thu, 03 Jun 2027 00:00:00 +0000

A Redis process can keep a substantial resident footprint after application keys are deleted. That observation deserves explanation, but it does not by itself prove a leak or allocator fragmentation. The useful question is which bytes remain allocated, which pages remain resident, and whether that footprint threatens the actual capacity budget.

Historical libc observations are approximate and distinct from asserted allocation relationships. A higher ratio alone does not establish a leak, fragmentation or effective reclamation.

Separate the measurements

INFO memory exposes used_memory for Redis-accounted allocations and used_memory_rss for the resident process footprint observed by the operating system. Neither is simply the sum of application payload lengths. Keys, objects, metadata, clients, and other overhead consume memory too. MEMORY USAGE provides per-key allocation estimates, including overhead; it does not measure that key’s independent contribution to process RSS.

Mem_fragmentation_ratio broadly compares resident memory with Redis-accounted allocations. Code, shared libraries, stacks, and allocator retention also contribute to the difference. The ratio can rise when the denominator falls after deletion, even if RSS barely changes. Inspect absolute bytes alongside ratios: a small empty instance can have a high ratio without consuming a dangerous amount of RAM.

Used_memory_peak helps explain a footprint left after an earlier workload peak. Used_memory_dataset is another accounting view, not a promise that an empty keyspace leaves zero process allocations. Measurements are not all sampled at exactly the instant INFO runs. Do not require an exact equality between a rounded ratio and independently observed values during active writes.

Check the allocator and version

Allocator-specific metrics can help distinguish fragmentation from retained pages and non-heap overhead, but their meaning depends on the build. Redis 8.10.1’s source computes allocator fragmentation from small-bin counters with a Lua adjustment; blindly reconstructing every version’s metric as active bytes divided by allocated bytes is unsafe. Check the versioned implementation when a general documentation formula disagrees.

This exercise deliberately builds with libc. That build lacks jemalloc’s detailed accounting; Redis substitutes some allocator fields from other measurements, and the fragmentation counter remains zero. An allocator_frag_ratio of one is therefore not proof that this process has no fragmentation. Always record mem_allocator before interpreting the more specialized fields or choosing an intervention.

When it breaks

A rising RSS curve alongside a growing dataset calls for workload and capacity analysis, not an automatic fragmentation diagnosis. A falling dataset with stable RSS suggests retained footprint, but still does not identify every retained page. Likewise, one snapshot cannot rule out a leak. Compare the same workload over time, value-size distribution, peak usage, and relevant process limits.

MEMORY PURGE attempts allocator reclamation where implemented. Redis documents it as a benign no-op for allocators other than jemalloc, so a successful reply is not evidence of recovered memory. It also does not remove application keys. Active defragmentation is a different mechanism and requires supported builds; enabling it can add CPU and latency costs that need measurement.

Restarting merely to lower a ratio can introduce downtime, cold-cache load, or data loss when persistence and recovery are inadequate. Establish a measured capacity problem and a rehearsed recovery path before treating restart as remediation. Maxmemory is not an RSS ceiling, and reducing key count does not guarantee immediate page return to the operating system.

Check it quickly

The original exercise reports macOS, Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0; this review did not rerun it. Use POSIX, a C compiler, make, curl, tar, and Python 3.10+. Setup builds Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It creates a dedicated Redis process and private Unix socket under /tmp, disables TCP and persistence, and writes only its own 320 sample keys. Alternating values contain 16 or 64 KiB. Short pauses permit metric sampling, without assuming exact refresh timing. The process and temporary runtime directory are removed afterward.

from contextlib import contextmanager
import json
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 def snapshot(stage):
 time.sleep(0.2)
 info = client.info("memory")
 fields = ["used_memory", "used_memory_dataset", "used_memory_rss",
 "used_memory_peak", "mem_fragmentation_ratio", "mem_allocator",
 "allocator_frag_ratio", "allocator_frag_bytes"]
 print(stage, json.dumps({"keys": client.dbsize(),
 **{name: info[name] for name in fields}}, sort_keys=True))
 return info

 empty = snapshot("empty")
 for index in range(320):
 client.set(f"sample:{index}", "x" * (16384 if index % 2 == 0 else 65536))
 assert client.dbsize() == 320
 assert client.memory_usage("sample:0") >= 16384
 loaded = snapshot("loaded")
 assert loaded["used_memory"] > empty["used_memory"] + 10 * 1024 * 1024
 for index in range(1, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 160
 half = snapshot("large values deleted")
 assert half["used_memory"] < loaded["used_memory"] - 8 * 1024 * 1024
 for index in range(0, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 0
 assert client.memory_usage("sample:0") is None
 cleared = snapshot("all values deleted")
 assert cleared["used_memory"] < half["used_memory"]
 assert cleared["used_memory_peak"] >= loaded["used_memory"]
 assert client.memory_purge()
 snapshot("after MEMORY PURGE")
 print("PASS: logical allocations fell; no RSS or purge-effect threshold is assumed")

The original run reported about 16.7 MiB of used_memory after loading. Deleting all keys reduced that to about 1.04 MiB, while RSS remained around 20.8 MiB. The overall ratio consequently rose to roughly twenty. MEMORY USAGE confirmed the named key no longer existed; logical allocations fell, but resident memory did not track them proportionally.

The allocator-specific fragmentation ratio stayed at one throughout this libc run, and MEMORY PURGE returned success without a meaningful RSS drop. Counts and logical-allocation relationships are asserted; exact RSS values are intentionally observations, not pass thresholds. This experiment demonstrates accounting differences and the no-op limitation, not jemalloc defragmentation effectiveness, Linux memory-pressure behavior, or a leak diagnosis.

Sources

Redis INFO, process, allocation, peak, and allocator memory fields, checked 2026-09-08.
Redis MEMORY USAGE, per-key allocation estimates and overhead, checked 2026-09-08.
Redis MEMORY PURGE, jemalloc support and other allocators’ no-op behavior, checked 2026-09-08.
Redis 8.10.1 object accounting, version-specific fragmentation calculation, checked 2026-09-08.
Redis 8.10.1 memory sampling, cached measurements and substituted allocator fields, checked 2026-09-08.
Redis 8.10.1 build instructions, core build and allocator selection, checked 2026-09-08.

Swap trades residency for reclaim cost

Wed, 25 Nov 2026 00:00:00 +0000

Swap gives Linux a backing store for eligible memory pages that need not remain resident. It can preserve cold data while RAM serves active work, but it does not make the machine’s working set fit at RAM speed. The useful question is whether reclaim frees space without repeatedly stalling important tasks. Swap occupancy, swap traffic, and application progress answer different parts of that question.

Know which pages can move

Anonymous memory commonly uses swap, but it is not the only case. tmpfs and shared-memory pages can also be swapped when their configuration permits it. Clean pages backed by ordinary files can usually be discarded and reread from those files; dirty file pages need suitable writeback. Not every allocation is reclaimable, and memory locked against paging has another constraint. “Everything from malloc” is not an adequate description of what the kernel can move.

vm.swappiness expresses the relative cost of swapping and filesystem paging on a documented scale from zero to two hundred. It is not the percentage of RAM that triggers swap. Zero does not universally disable swapping. Values above one hundred can make sense when compressed or faster swap is cheaper than filesystem paging, but the appropriate value depends on the workload and storage. Inspect the actual configuration before interpreting a graph.

zram supplies a compressed RAM block device that can be configured as swap. zswap is a compressed cache for swap pages with a backing swap device. Compression consumes RAM and CPU, and its effectiveness depends on the data. Neither offers a universal latency, compression ratio, or safe overcommit percentage. zswap is not a filesystem to mount on /tmp; a tmpfs mount has its own storage and capacity semantics.

When it breaks

Repeatedly reclaiming pages that are immediately needed again can cause severe stalls. Storage latency and throughput may matter, but so can reclaim work, compression cost, cgroup limits, and competing I/O. Thrashing is not guaranteed to end in an OOM kill, and a fixed swap rate is not a universal diagnosis. Compare memory pressure with useful throughput and response times before assigning the cause.

A large used-swap number can reflect cold pages left there after an earlier event. It does not prove active distress. Conversely, little swap traffic does not prove enough memory: swap may be disabled, a workload may hit a cgroup limit, or reclaim may be dominated by file pages. A rising major-fault count includes file-backed faults and cannot be relabeled as a count of swap-ins.

An intended swap file can fail activation because of unsupported layout or filesystem restrictions. Files with holes are unsuitable, but whether copying creates holes depends on the tool, options, and source. Read the active swap inventory rather than assuming a configuration entry succeeded. Do not activate, resize, or disable swap on a live machine merely to reproduce a diagnostic note.

Swap occupancy alone does not establish current pressure; interval traffic and PSI quantify different aspects of activity and stalls.

Check it quickly

Run these read-only commands on Linux with util-linux and procps installed. The check observes the current environment for three intervals; it creates no memory pressure and changes no kernel settings. A host with no active swap is a valid observation, not a failed test:

set -eu
swapon --show
free -h
vmstat -y 1 3
grep -E '^(pswpin|pswpout|pgmajfault) ' /proc/vmstat
if test -r /proc/pressure/memory; then
 cat /proc/pressure/memory
else
 printf '%s\n' 'memory PSI is unavailable in this environment'
fi

swapon --show lists active swap areas. free separates memory and swap totals. With vmstat -y, the initial since-boot report is omitted, so the three rows describe sampling intervals. Read the displayed units and compare swap-in and swap-out activity with blocked tasks and CPU time. The cumulative counters from /proc/vmstat are a separate snapshot, not rates for those exact three rows.

Memory PSI, when available, reports time lost to memory stalls. Its some and full lines distinguish at least some stalled work from simultaneous stalls of all non-idle tasks in the measured scope. A short quiet sample cannot exclude an earlier spike. Container visibility and cgroup boundaries also matter; host-wide counters do not automatically describe one application’s allocation budget.

This check verifies observation and interpretation, not storage performance, safe swap sizing, or OOM recovery. Preserve timestamps and compare a representative incident window with application latency and resource limits. If the evidence shows persistent working-set pressure, reduce demand or provide suitable capacity. Tune reclaim policy only with a reversible plan and measured service outcomes, rather than chasing zero swap usage as an end in itself.

Sources

Kernel VM sysctls, relative I/O cost and zero-value behavior, checked 2026-10-15.
Kernel tmpfs, swap-backed tmpfs and mount limits, checked 2026-10-15.
Kernel zram, compressed block device and storage accounting, checked 2026-10-15.
Kernel zswap, compressed swap cache and backing-store relationship, checked 2026-10-15.
Kernel PSI, memory stall scope and averages, checked 2026-10-15.
vmstat(8), interval reports and swap-rate fields, checked 2026-10-15.
proc_vmstat(5), separate swap and major-fault counters, checked 2026-10-15.
swapon(8), active inventory and swap-file restrictions, checked 2026-10-15.

zram squeezes swap into RAM

Tue, 24 Nov 2026 00:00:00 +0000

When memory tightens, eligible cold pages may be reclaimed to swap. A zram swap device stores compressed pages in RAM, trading CPU work and memory overhead for potential headroom. The benefit depends on the workload; it is not a fixed amount of extra memory.

A compressed block device in memory

zram is a RAM-backed block device that stores compressed data. Configured as swap, it can retain eligible pages without ordinary disk swap I/O. Optional writeback can use a separately configured backing device, so “never touches a drive” is not universal. Compression may reduce latency, but the algorithm, CPU load and data matter. Logical device capacity and physical RAM consumption must be read separately; neither describes free system memory.

Compression is the space win

Compression is the space win, and it is the reason a small box can appear to hold more memory than it physically has. Whether a page pays off depends on its content: text, zeroed pages and structured data compress well, while already-compressed blobs such as media or archives barely shrink. The honest engineering view is that zram turns spare CPU cycles into apparent memory, so the trade is worthwhile when the workload leans compressible and the extra latency of a real disk swap is exactly what you want to avoid.

It can suit small hosts and phones

The fit shines on hosts with little memory and modest I/O, such as a small VPS, a container host, or a compact device like a phone or an embedded box. It is less obviously right for a server that already has plenty of RAM or that swaps into genuinely cold storage anyway, because a real disk bought with the same spend can hold far more. The practical check is to watch the compression ratio and swap usage, decide whether swapping beats killing the process, and pick zram when resident data stays small and compressible enough to make the CPU cost a fair trade.

When it breaks

Poor compression can erase much of the expected gain. Photos, video, encrypted buffers and archives may barely shrink. Compare orig_data_size with compr_data_size, but also inspect mem_used_total, which includes allocator fragmentation and metadata. Pool occupancy, available RAM, workload demand and pressure still matter. A ratio near one does not determine whether or when an OOM event occurs; cold-page residency and reclaim behavior also affect the outcome.

An expensive compressor can increase CPU use and swap latency. Available algorithms and parameters depend on the kernel build. Compare candidates such as lzo-rle, lz4 or zstd under representative pressure rather than assuming one is always suitable. High system CPU or a busy kswapd thread alone does not identify the compressor as the cause. Record application latency and useful progress alongside compression and reclaim activity before changing policy.

Using zswap with zram can add compression, decompression and accounting overhead. On eviction, zswap reconstructs a page before submitting it to the backing swap device; it does not simply hand an already-compressed blob to zram. A zram backing device may then compress that page again. Whether this arrangement helps requires workload evidence. Do not treat the two names as interchangeable mechanisms or claim that every layered configuration has identical costs.

A fixed disksize can become inappropriate after a machine resize, but it sets logical device capacity rather than immediately reserving that many RAM bytes. The sysfs disksize setting initializes the device before mkswap; mkswap does not set the zram device size. mem_limit is a separate limit on compressed-data memory. Capacity policy should reflect demand, overhead and recovery margin, and be reviewed after resizing rather than automatically enlarged to an arbitrary fraction.

A 3:1 fixture payload ratio excludes allocator overhead; the successor is tested only with ordinary files.

Check it quickly

Save this as zram_ratio.py and run python3 zram_ratio.py without optimization. The standard-library self-test creates temporary files with invented mm_stat snapshots, asserts reports and exit decisions, then removes them. It never discovers devices or reads kernel interfaces. An optional filename, as in python3 zram_ratio.py saved-mm-stat.txt, reads a saved ordinary text snapshot. Use ordinary files only.

#!/usr/bin/env python3
"""Offline mm_stat parser/report demo; no device discovery or kernel access."""
from pathlib import Path
import sys
from tempfile import TemporaryDirectory


def parse_mm_stat(text):
 parts = text.split()
 if len(parts) < 3:
 raise ValueError("need orig, compr and mem_used_total")
 if any(not x.isascii() or not x.isdecimal() for x in parts):
 raise ValueError("fields must be unsigned decimal integers")
 values = tuple(int(x) for x in parts)
 if any(x > (1 << 64) - 1 for x in values):
 raise ValueError("field exceeds unsigned 64-bit range")
 orig, compr, mem_used = values[:3]
 if orig == 0 and compr != 0:
 raise ValueError("inconsistent zero-orig snapshot")
 return orig, compr, mem_used


def report(name, orig, compr, mem_used):
 prefix = f"{name}: orig={orig} compr={compr} mem_used_total={mem_used}"
 if orig == 0:
 return prefix + " ratio=n/a (no stored data)"
 if compr == 0:
 return prefix + " ratio=n/a (nonempty; zero compressed payload)"
 ratio = orig / compr
 saved = 100.0 * (1.0 - compr / orig)
 return prefix + f" ratio={ratio:.2f} payload_saved={saved:.1f}%"


def inspect_paths(paths):
 lines = []
 parsed = skipped = 0
 for path in paths:
 try:
 values = parse_mm_stat(path.read_text(encoding="ascii"))
 except (OSError, UnicodeError, ValueError) as exc:
 lines.append(f"SKIP {path.name}: {type(exc).__name__}")
 skipped += 1
 continue
 lines.append(report(path.name, *values))
 parsed += 1
 lines.append(f"SUMMARY: parsed={parsed} skipped={skipped}")
 return (0 if parsed > 0 and skipped == 0 else 1), lines


def self_test():
 if not __debug__:
 raise RuntimeError("run without Python optimization")
 with TemporaryDirectory(prefix="zram-fixture-", dir=".") as directory:
 root = Path(directory)
 cases = {
 "ratio": ("314572800 104857600 125829120 0 125829120 0 0 0 0\n",
 "ratio: orig=314572800 compr=104857600 mem_used_total=125829120 ratio=3.00 payload_saved=66.7%"),
 "same": ("4096 0 0 0 0 1 0 0 0\n",
 "same: orig=4096 compr=0 mem_used_total=0 ratio=n/a (nonempty; zero compressed payload)"),
 "empty": ("0 0 0\n", "empty: orig=0 compr=0 mem_used_total=0 ratio=n/a (no stored data)"),
 "overhead": ("0 0 4096\n", "overhead: orig=0 compr=0 mem_used_total=4096 ratio=n/a (no stored data)"),
 "plain": ("4096 4096 8192\n", "plain: orig=4096 compr=4096 mem_used_total=8192 ratio=1.00 payload_saved=0.0%"),
 }
 for name, (text, expected) in cases.items():
 path = root / name
 path.write_text(text, encoding="ascii")
 assert inspect_paths([path]) == (0, [expected, "SUMMARY: parsed=1 skipped=0"])
 bad = ["", "1", "1 1", "x 1 1", "1 1 x", "-1 0 0",
 "1 -1 0", "1 1 -1", "1 1 1 -1", "0 1 1", "1.0 1 1",
 "+1 1 1", "1_000 1 1", "18446744073709551616 1 1"]
 invalid_paths = []
 for index, text in enumerate(bad):
 path = root / f"bad-{index}"
 path.write_text(text, encoding="ascii")
 invalid_paths.append(path)
 assert inspect_paths([path]) == (1, [f"SKIP {path.name}: ValueError", "SUMMARY: parsed=0 skipped=1"])
 missing = root / "missing"
 assert inspect_paths([missing]) == (1, ["SKIP missing: FileNotFoundError", "SUMMARY: parsed=0 skipped=1"])
 assert inspect_paths([]) == (1, ["SUMMARY: parsed=0 skipped=0"])
 code, lines = inspect_paths(invalid_paths + [missing])
 assert code == 1 and lines[-1] == "SUMMARY: parsed=0 skipped=15"
 assert len(lines) == 16 and all(line.startswith("SKIP ") for line in lines[:-1])
 code, lines = inspect_paths([root / "ratio", missing])
 assert code == 1 and lines == [cases["ratio"][1], "SKIP missing: FileNotFoundError", "SUMMARY: parsed=1 skipped=1"]
 print("PASS: offline file fixtures; ratio, nonempty zero payload, empty, invalid/missing, all-skipped and partial results")


def main():
 if len(sys.argv) == 1:
 self_test()
 return 0
 code, lines = inspect_paths([Path(name) for name in sys.argv[1:]])
 print("\n".join(lines))
 return code


if __name__ == "__main__":
 raise SystemExit(main())

On 2026-10-19, both complete fences passed offline under Python 3.12.14: assertions enabled, exit 0, empty stderr. PASS covers only file-fixture assertions. The invented 300 MiB / 100 MiB snapshot checks ratio 3.00 and payload saving 66.7%. Tests distinguish nonempty zero payload from no stored data, retain allocation, and reject missing, malformed, negative or inconsistent values. Separate mutation tests detect deliberately broken behavior.

The parser requires three fields and unsigned 64-bit decimal tokens; mem_used_total is reported separately. Same-filled pages may have positive original bytes without compressed payload: the ratio is unavailable, not proof of emptiness. No stored data does not mean zero allocation. Optional-file mode never prints PASS and exits nonzero if nothing parsed or anything was skipped. It cannot infer disksize, pool fullness, trends, CPU cost, OOM timing or tuning remedies. No live-device behavior was tested.

Sources

kernel.org: zram block device, the authoritative zram documentation, checked 2026-10-19.
Wikipedia: zram, what zram is and how it works, checked 2026-10-19.
ArchWiki: Zram, retained further reference; access attempted 2026-10-19, but an anti-bot page prevented content verification.
Linux v6.12 zswap source, correction reference, checked 2026-10-18.
Linux zswap documentation, correction reference, checked 2026-10-18.
Linux v6.12 zram source, same-filled page accounting and mm_stat fields, checked 2026-10-19.

OOM investigation starts with the allocation scope

Mon, 26 Oct 2026 00:00:00 +0000

Linux can invoke an out-of-memory killer when an allocation cannot be satisfied within its relevant constraints after appropriate reclaim attempts. It is a last-resort recovery mechanism, not an application priority scheduler. Some allocations fail without a kill, and configuration can instead lead to panic behavior. Start an investigation by identifying the allocation scope and recorded event, rather than declaring that the machine must have exhausted every byte of RAM.

Scope comes before the score

A global memory shortage and a memory-control-group shortage are different situations. A cgroup can hit its own limit while the host still has available memory. Its OOM handling cannot simply solve that shortage by selecting an arbitrary process outside the constrained group. Other constraints, such as allowed memory nodes, can also matter. A dashboard of host-wide free memory is therefore insufficient evidence for or against a particular allocation failure.

Within the applicable selection rules, memory use and an adjustment influence victim choice. The kernel does not know which process is commercially valuable, idle-but-disposable, or essential to the business. A large database can be selected even when smaller workers contributed to pressure. oom_score is an observation exposed through procfs, not a promise that sorting every visible process predicts the next victim in every possible OOM domain.

Interpret adjustments without promising immunity

oom_score_adj ranges from -1000 to 1000. Positive values increase susceptibility and negative values reduce it; -1000 excludes a task from the kernel OOM selection addressed by that control. A modest negative adjustment does not guarantee survival. It is also not protection against explicit signals, a service manager, or a separate userspace OOM policy. Numeric adjustments cannot create missing capacity.

Protecting every task is not a workable capacity plan. If no eligible victim exists, the kernel cannot be assumed to escape a cgroup boundary and kill an unrelated supervisor. Group-kill behavior and panic settings require separate analysis. Adjustments can be inherited by children, while persistent service settings belong to the supervisor’s configuration. Changes require deliberate authorization and testing; the exercise below makes none.

When it breaks

A restart loop can hide an ongoing leak. If the same workload grows until it fails repeatedly, adding memory may merely lengthen the interval. Correlate allocation rate, resident and charged memory, workload size, reclaim, and swap activity. A process’s RSS alone does not describe all cgroup charges or all shared-memory relationships. Lowering a limit can contain growth but may make the application fail sooner.

Exit status 137 is compatible with SIGKILL; it is not exclusive evidence of OOM. Check the runtime’s termination reason, cgroup memory-event counters, and kernel records available to the operator. Distinguish a failed allocation from a killed process, and kernel OOM from userspace pressure-driven termination. A missing kernel log can reflect unavailable visibility rather than proof that no event occurred.

The score itself changes as memory and scope change. Processes can exit between procfs reads, and container PID visibility may hide much of the host. Never treat a successful read as an atomic system snapshot. Preserve the event time and affected cgroup when diagnosing an actual failure, then compare observations from the same scope.

OOM scope and event context come before score ranking; the PID ordering here is an invented presentation fixture.

Check it quickly

Save this as oom_observe.py and run python3 oom_observe.py with Python 3.9 or newer. It reads only its own process’s two procfs score files, allocates no pressure workload, and changes no limits or policy. Two deterministic assertions test a presentation sort on explicitly invented rows.

"""Read-only observations and a sorting test, not an OOM predictor."""
from pathlib import Path
import os

def snapshot(pid):
 base = Path('/proc') / str(pid)
 try:
 return (int(pid), int((base/'oom_score').read_text()),
 int((base/'oom_score_adj').read_text()))
 except (OSError, ValueError):
 return None

def ranked(rows):
 return sorted(rows, key=lambda row: (-row[1], row[0]))

fixtures = [(42, 700, 0), (17, 100, -500), (99, 700, 200)]
assert [row[0] for row in ranked(fixtures)] == [42, 99, 17]
assert ranked([]) == []
row = snapshot(os.getpid())
if row is None:
 print('SKIP: this process has no readable Linux OOM score files')
else:
 pid, score, adjustment = row
 assert -1000 <= adjustment <= 1000
 print(f'pid={pid} oom_score={score} oom_score_adj={adjustment}')
 print('Observed one process; no victim selection or memory pressure tested')
print('PASS: deterministic sorting checks; live observation is reported separately')

The fixture orders equal scores by numeric PID and puts the lower score last. It does not calculate kernel badness from the invented adjustment values. On Linux, the separate live branch prints the current process’s score and adjustment; without readable files it reports SKIP while still running the sorting assertions. A PASS line therefore does not imply live OOM handling was exercised.

Both exact examples passed the sorting assertions and read their own procfs files in this review’s Linux environment. No victim was selected or killed. Use this small observation to learn the interface, then investigate real events using scope-specific evidence. Changing the ranking or giving an important process a negative adjustment is never sufficient proof that the service will remain available under memory pressure.

Sources

Linux memory concepts, allocation, reclaim and OOM boundaries, checked 2026-10-15.
proc_pid_oom_score, reported process score, checked 2026-10-15.
proc_pid_oom_score_adj, adjustment range and constrained allocation context, checked 2026-10-15.
Linux cgroup v2, memory.max, OOM groups and protection scope, checked 2026-10-15.

Memory · XB Field Notes

Postgres uses two cache layers

When it breaks

Check it quickly

Budget memory for the complete server

Sources

Redis expiry is a deadline, eviction is a capacity policy

Time and capacity are separate axes

Select candidates before debating ranking

When it breaks

Check it quickly

Further reading

Sources

Huge pages trade allocation flexibility for translation reach

Page size is an input

Explicit pools and transparent allocation

Budget capacity and placement

When it breaks

Check it quickly

Further reading

Sources

Linux overcommit budgets virtual-memory commitments

Virtual commitment differs from residence

Three admission policies

Strict accounting is not OOM immunity

When it breaks

Check it quickly

Further reading

Sources

A page fault is not always a storage read

Minor does not mean free

Major describes a different path

When it breaks

Check it quickly

Sources

Redis memory ratios need allocation context

Separate the measurements

Check the allocator and version

When it breaks

Check it quickly

Further reading

Sources

Swap trades residency for reclaim cost

Know which pages can move

When it breaks

Check it quickly

Sources

zram squeezes swap into RAM

A compressed block device in memory

Compression is the space win

It can suit small hosts and phones

When it breaks

Check it quickly

Further reading

Sources

OOM investigation starts with the allocation scope

Scope comes before the score

Interpret adjustments without promising immunity

When it breaks

Check it quickly

Sources