XB Field Notes · XB Field Notes

InnoDB deadlocks roll back the whole victim transaction

Thu, 20 Apr 2028 00:00:00 +0000

An InnoDB deadlock occurs when transactions form a cycle of incompatible lock requests. With deadlock detection enabled, InnoDB chooses a victim and rolls back that entire transaction so another participant can proceed. The application receives error 1213 with SQLSTATE 40001. Retrying only the statement that reported the error omits earlier work that was rolled back with it.

When it breaks

Do not assign one rollback rule to every lock error. Under the default innodb_rollback_on_timeout=OFF, a lock-wait timeout rolls back the waiting statement while earlier statements in the transaction remain pending. Enabling that server option changes timeout rollback to the whole transaction. A deadlock already rolls back the whole victim transaction regardless of that default timeout behavior.

Victim selection is not a contract that a particular session always loses. Applications must inspect the actual error and retry the complete logical transaction with bounded attempts. Keep irreversible external effects out of an automatically replayed transaction unless they have a separate deduplication contract. A disconnected commit response is also not equivalent to a confirmed deadlock rollback.

Either participant can be the deadlock victim. Timeout comparison assumes rollback-on-timeout off and REPEATABLE READ in every session; shown outcomes require manual checks.

Check it quickly

The unchanged three-session test was not rerun here. Retained reviews report MySQL unavailability; no raw MySQL log was recovered in the retained validation files. The version and selected victim below remain original article reports.

Use three fresh administrative MySQL 8.4 client sessions, A, B, and observer O, against a disposable instance. Start the clients in autocommit mode with REPEATABLE READ isolation in every session; a changed server default can otherwise alter visibility or turn the later read into a lock wait. Disable client automatic reconnection so session identity remains stable. The recorded run used MySQL 8.4.11. In O, create an unused fixture schema and verify detection is enabled and timeout rollback uses its default statement scope:

CREATE DATABASE xbcatm_deadlock_lab;
USE xbcatm_deadlock_lab;
CREATE TABLE accounts (id INT PRIMARY KEY, balance INT NOT NULL) ENGINE=InnoDB;
INSERT INTO accounts VALUES (1,100),(2,100);
SELECT @@innodb_deadlock_detect,@@innodb_rollback_on_timeout;

The settings should be one and zero. In A, start an operation adding ten to both accounts, initially locking account one:

USE xbcatm_deadlock_lab;
SET SESSION innodb_lock_wait_timeout=30;
SET @increment=10;
START TRANSACTION;
UPDATE accounts SET balance=balance+@increment WHERE id=1;

In B, start an operation adding one hundred, initially locking account two:

USE xbcatm_deadlock_lab;
SET SESSION innodb_lock_wait_timeout=30;
SET @increment=100;
START TRANSACTION;
UPDATE accounts SET balance=balance+@increment WHERE id=2;

In A, request the row B holds. This statement waits:

UPDATE accounts SET balance=balance+@increment WHERE id=2;

Before A’s thirty-second lock timeout, run the opposing request in B:

UPDATE accounts SET balance=balance+@increment WHERE id=1;

One session receives error 1213; identify it as the victim. The other update completes. Run this only in the surviving session:

COMMIT;

In O, inspect the committed state:

SELECT id,balance FROM accounts ORDER BY id;

Both balances are 110 if A survived, or 200 if B survived. The victim’s earlier update is absent too. The recorded run selected B, leaving 110 in both rows. In the victim’s existing session, its session variable still records the original increment. Retry the whole operation, now acquiring rows in consistent order:

SELECT @increment;
START TRANSACTION;
UPDATE accounts SET balance=balance+@increment WHERE id=1;
UPDATE accounts SET balance=balance+@increment WHERE id=2;
COMMIT;

In O, verify the combined outcome:

SELECT id,balance FROM accounts ORDER BY id;

Both balances must now be 210. Next, contrast a timeout. In O, reset the fixture:

UPDATE accounts SET balance=100;

In A, hold account one:

START TRANSACTION;
UPDATE accounts SET balance=balance+1 WHERE id=1;

In B, change account two and set a short wait:

SET SESSION innodb_lock_wait_timeout=1;
START TRANSACTION;
UPDATE accounts SET balance=balance+5 WHERE id=2;

Still in B, request A’s row:

UPDATE accounts SET balance=balance+5 WHERE id=1;

This produces error 1205 with SQLSTATE HY000. In B, inspect its own pending work:

SELECT id,balance FROM accounts ORDER BY id;

B reads 100 and 105. In O, committed readers still see 100 and 100:

SELECT id,balance FROM accounts ORDER BY id;

The timeout did not undo B’s earlier five-unit change. Explicitly discard the remaining transactions, first in B:

ROLLBACK;

Then in A:

ROLLBACK;

Finally, in O, confirm both original balances and remove the fixture:

SELECT id,balance FROM accounts ORDER BY id;
DROP DATABASE xbcatm_deadlock_lab;

Manually compare every error, setting and row result; the SQL prints values without assertions. The intended comparison distinguishes rollback scope. MySQL DDL commits implicitly, so interruption can leave the schema even after transaction rollback. Stop on unexpected results and roll back open fixture transactions before cleanup.

Reduce cycles without assuming they disappear

Acquire shared business resources in consistent order, keep transactions short, and use appropriate indexes to avoid locking more records than the operation needs. Range operations and secondary-index maintenance can add lock interactions beyond the two primary-key rows shown here. Consistent application ordering reduces a known cycle; it is not proof that every possible deadlock has been removed.

Use SHOW ENGINE INNODB STATUS to examine the latest detected deadlock, and capture broader history when incident frequency warrants it. Correlate the involved statements and lock order with the application transaction, not just the final failed statement. Retry complete work within a deadline, distinguish timeout semantics, and verify the business result before treating recovery as successful.

Sources

MySQL 8.4 InnoDB error handling, complete deadlock rollback and configurable timeout rollback scope, checked 2026-09-12.
MySQL 8.4 deadlock example, reproducing a cycle and examining the selected victim, checked 2026-09-12.
MySQL 8.4 InnoDB locking, record and range locks involved in concurrent operations, checked 2026-09-12.

dnsmasq separates forwarded cache entries from configured local names

Sat, 15 Apr 2028 00:00:00 +0000

dnsmasq can forward DNS questions upstream, retain supported answers in a bounded cache, and serve names configured locally. These are different sources of data. A repeated answer may be a cache hit or a local configuration result. Establishing which source supplied it is essential when a hostname disagrees with the authority or continues resolving after a cache flush.

Identify the source of each answer. Forwarded cache and configured local names have different lifetimes: local TTL zero does not erase configuration. SIGHUP clears cache and reloads selected data sources, not generally the main configuration. Counts and addresses reproduce the article’s historical example; it was not rerun.

When it breaks

Starting dnsmasq does not automatically make every application use it. Clients must direct their DNS queries through the intended listener, and some applications use other resolvers. Likewise, a correct local listener can forward to the wrong upstream or read unexpected host configuration. Diagnose the actual query path before enlarging the cache or restarting a daemon that never received the failing query.

A local name can deliberately override upstream data. Clearing the forwarding cache does not delete that configured name, so repeated flushing cannot make the upstream address win. Treat a local record as configuration with an owner and purpose. An answer with local TTL zero can still be returned repeatedly because dnsmasq generates it from configuration rather than retaining a forwarded response.

Cache capacity and freshness solve different problems. A larger cache can retain more eligible entries, but it does not extend their valid TTLs or make every possible query answerable during an upstream outage. New names and expired entries still need the appropriate resolution path. Cache-size tuning also does not directly specify the maximum number of simultaneous upstream queries or reserve application-level availability.

Check it quickly

Save the script as check-dnsmasq.py and run python3 check-dnsmasq.py with Docker. The pinned environment uses dnsmasq 2.90 and BIND 9.18.49. The test forwarders ignore ambient resolver and hosts files, bind explicit loopback listeners, and forward only to the disposable authority. They publish no host ports and use no DHCP configuration:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
with lab('cached 20 IN A 127.0.0.2\nuncached 20 IN A 127.0.0.3\nlocal 20 IN A 127.0.0.4') as (_,update,cache,queries):
 daemon,log=cache(options=('--host-record=local.lab.test,127.0.0.9',))
 for _ in range(3):assert values(query('cached.lab.test.',port=1054))==['127.0.0.2']
 wait(lambda: queries('cached.lab.test')==1)
 cache(port=1055,size=0)
 for _ in range(3):assert values(query('uncached.lab.test.',port=1055))==['127.0.0.3']
 wait(lambda: queries('uncached.lab.test')==3)
 local=query('local.lab.test.',port=1054);assert values(local)==['127.0.0.9'] and local.answer[0].ttl==0
 assert queries('local.lab.test')==0
 authoritative=query('local.lab.test.');assert values(authoritative)==['127.0.0.4']
 wait(lambda: queries('local.lab.test')==1)
 update('cached 20 IN A 127.0.0.8\nuncached 20 IN A 127.0.0.3\nlocal 20 IN A 127.0.0.4')
 stale=query('cached.lab.test.',port=1054);assert values(stale)==['127.0.0.2'] and queries('cached.lab.test')==1
 previous=log.read_text().count('cleared cache');daemon.send_signal(signal.SIGHUP)
 wait(lambda: log.read_text().count('cleared cache')>previous)
 refreshed=query('cached.lab.test.',port=1054);assert values(refreshed)==['127.0.0.8']
 wait(lambda: queries('cached.lab.test')==2)
 retained=query('local.lab.test.',port=1054);assert values(retained)==['127.0.0.9']
 print(json.dumps({'three_lookups_upstream_counts':{'cache_enabled':1,'cache_disabled':3},'local_record':show(local),
 'same_name_authority':show(authoritative),'before_SIGHUP':show(stale),'after_SIGHUP':show(refreshed),
 'configured_local_record_retained':show(retained)},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Three queries for cached.lab.test through the cache-enabled listener all return 127.0.0.2, while the authoritative log records one query. A second listener uses cache-size=0. Three queries for a separate name through it generate three upstream queries. This directly verifies the cache’s effect on forwarding work; the experiment does not infer hits from response speed or assert a particular latency improvement.

The first listener also has host-record=local.lab.test,127.0.0.9. Its answer is 127.0.0.9 with TTL zero, and no upstream query is logged for that lookup. A direct authoritative query for the same name returns 127.0.0.4 and increments the authority’s counter. The differing results show an explicit local override, with a positive control confirming that the query counter detects real upstream traffic.

Next, the authority changes cached.lab.test to 127.0.0.8. The cache still returns its earlier 127.0.0.2 answer within the remaining TTL. The script sends SIGHUP to this specific dnsmasq process and waits for its cache-cleared log entry. The following query reaches the authority and returns 127.0.0.8, increasing the corresponding authoritative count from one to two.

After that same signal, local.lab.test continues returning the configured 127.0.0.9 value. The signal clears forwarded cache state while leaving the explicitly configured local record available. This is why “flush DNS” is incomplete troubleshooting advice: the expected result depends on which process owns which data, and whether that data is cached, configured, or stored somewhere else entirely.

Operate the listener and its data sources deliberately

Inspect the effective configuration, listener address, upstream destinations, and client settings together. Avoid creating a forwarding loop between two local resolvers. The example removes ambient inputs to isolate behavior; a real installation may intentionally use hosts files, DHCP leases, or resolver-management software. Record those dependencies so an automatic rewrite or a local override is visible during troubleshooting.

For dnsmasq, SIGHUP clears its cache and reloads certain data sources described by the manual, but it does not generally reread the main configuration file. Use the documented service procedure for the kind of change being made. A successful cache clear cannot be treated as proof that edited listener, forwarding, or other startup configuration has been activated.

Limit listener exposure to intended clients, especially when forwarding recursion on a reachable network. DHCP, DNSSEC validation, and encrypted upstream transport require their own configuration and verification; none is demonstrated by this cache experiment. Query logging is useful during diagnosis but can reveal client naming activity, so select its scope and retention according to the environment rather than leaving detailed logs indefinitely by habit.

Finally, compare cached answers with the authority when it is the expected source. For a configured local name, compare against its configuration. Track upstream failures, hits, and entry churn separately. Identify whether the discrepancy concerns capacity, lifetime, the upstream service, or a local naming rule.

Sources

dnsmasq manual, forwarding, cache-size, host-record, local TTL, and SIGHUP behavior, checked 2026-09-12.
RFC 1035 domain names, DNS resource records and cached lifetimes, checked 2026-09-12.

LVM snapshots capture a block device in time

Mon, 10 Apr 2028 00:00:00 +0000

A classic LVM snapshot uses a copy-on-write exception store attached to an origin logical volume. lvcreate -s -L 5G -n snap vg/data requests a 5 GiB store, subject to allocation granularity and available volume-group space. It avoids copying the whole origin at creation, but does not guarantee zero work or instant completion. Thin snapshots use a different shared-pool mechanism.

The exception store must cover distinct changed chunks and metadata for the entire retention window, including writes during backup. A burst can consume it earlier than a quiet-workload estimate predicts. If a classic store fills, the snapshot becomes invalid and reads fail. Monitor validity and growth early enough to extend or finish the backup; a nominal percentage alone is not a recovery plan.

An exception store, not a copy

The origin uses the snapshot-origin target, while snapshot combines unchanged origin chunks with preserved exceptions. Writes to the snapshot itself go to its COW device. Repeated changes to an already-excepted chunk do not require another original copy for that snapshot. The earlier million-write, sixteen-chunk, fifteen-exception result has no recovered trace; touching all sixteen distinct chunks would require sixteen data exceptions, plus metadata.

lvs -a -o lv_name,lv_attr,origin,data_percent,lv_size exposes type, state and usage. swi-a-s--- is illustrative, not one universal attribute string; the fifth character I means invalid snapshot. Alert thresholds must leave room for observed write rates and response time. lvextend needs available VG extents. lvchange --monitor y enables monitoring, not guaranteed extension: dmeventd, configured autoextend policy and free capacity are also required.

Exhaustion is invalidation

Kernel documentation says an exhausted classic COW store is disabled and returns errors. The lvcreate manual’s twenty-percent-of-origin command is an example, not a general sizing recommendation. Account for chunk granularity, metadata, workload bursts and the full backup duration. Increasing capacity before exhaustion preserves a valid snapshot; an invalid recovery point must not be treated as complete merely because its LV still exists.

Origin writes normally continue when a classic snapshot overflows, but this does not guarantee that unrelated origin or storage failures cannot occur. An invalid snapshot is unusable as the recovery point; it did exist and may have supplied an earlier verified backup. Do not confuse current snapshot validity with the separate integrity of a copy already completed elsewhere.

Merging, and why thin changes the failure

lvconvert --merge vg/snap merges a persistent COW snapshot back into its origin and consumes the snapshot. It starts immediately when both LVs are closed; otherwise LVM defers it until an activation when both are closed. The origin may serve I/O after merging begins, with affected writes deferred as required. This is not a guarantee of interruption-free rollback, and newer origin data can be discarded.

A thin snapshot shares data and metadata capacity with other thin LVs instead of owning a separately sized COW store. Data exhaustion may queue writes or return errors according to policy; metadata exhaustion is another failure mode. Such failures can affect multiple volumes and filesystems, but do not imply instant destruction of every volume. Monitor both pool percentages, policy and extension capacity.

When it breaks

A block snapshot is not automatically an application-consistent backup. Filesystem and database buffers, logging and data spread across volumes require coordinated handling and recovery. During copying, check snapshot validity and propagate read failures; verify the resulting contents and a restore. A successful command exit or mount alone does not establish completeness, and this review performed no LVM creation, exhaustion, extension or merge experiment.

Before a merge, identify open users of both origin and snapshot and plan the application outage or activation boundary. Rehearse on disposable volumes and keep an independent backup. Size for all distinct changes across the retention window, not just its busiest hour. Confirm the actual resulting volume and restored application state; do not infer success solely from declining allocation counters.

LVM keeps changed chunks — mechanism diagram.

Check it quickly

The unchanged lines below remain held. They contain environment-specific names and require authorized LVM access; lvextend changes allocation, so this is not a read-only diagnostic block. No device-mapper, kernel, volume or mount operation was executed:

lvs -a -o lv_name,vg_name,lv_attr,origin,data_percent,lv_size
lvs -a -o lv_name,data_percent,metadata_percent vg/thinpool
dmsetup status vg-snap
lvextend -L +2G vg/snap

Classic snapshot Data% and its validity state differ from a thin pool’s Data% and Meta%. dmsetup status output is target-specific: normal snapshot allocation need not fall, while snapshot-merge allocation can decrease toward its metadata remainder. The final line neither proves free VG capacity nor revives an invalid snapshot. A separately held complete exercise must bind targets and verify each claimed transition.

Sources

lvcreate(8), documents -s/--snapshot, COW versus thin snapshots and illustrative sizing commands, checked 2026-09-16.
lvs(8), defines the lv_attr state characters including invalid snapshot, checked 2026-09-16.
Device-mapper snapshot support, the exception store, invalidation and lvconvert --merge behaviour, checked 2026-09-16.
lvmthin(7), thin snapshots, pool exhaustion and automatic extension, checked 2026-09-16.
lvconvert(8), immediate versus deferred merge and open-volume conditions, checked 2026-10-18.
PostgreSQL filesystem backup, consistent snapshot scope, WAL and multi-volume recovery requirements, checked 2026-10-18.

Consul discovery reflects health policy and freshness

Thu, 06 Apr 2028 00:00:00 +0000

Consul discovery reports registered endpoints and recorded health observations. It cannot promise that every returned endpoint will answer the caller’s next request. A useful operating model separates four things: what is registered, which checks apply, which states the query accepts, and how old those observations may be. A successful lookup establishes none of those guarantees by itself.

Registration and health are separate

The catalog stores registrations; health endpoints add check information. An unfiltered /v1/health/service/web response can include instances with critical checks. A failed check does not automatically erase its service registration. Explicit deregistration and configured critical-service removal are separate lifecycle mechanisms, and ACL filtering can also change which records a reader sees.

Registered checks normally start critical unless an initial status is supplied. That does not mean every registered service necessarily has a meaningful readiness check. For one service instance, consider its own service checks together with node-level checks such as serfHealth. A failed database-service check on the same node does not automatically fail an unrelated web-service check. A critical node check, however, affects services on that node.

Check outcomes are not simply Boolean. Script exit zero means passing, one means warning, and other exit codes mean critical. HTTP checks classify 2xx as passing and 429 as warning after redirect handling; redirects are followed by default unless disable_redirects is set. Other final responses indicate failure. The choice of probe determines what those states actually establish. A TCP handshake or process-existence test does not prove an application can finish a request.

DNS is not always passing-only

The HTTP health endpoint’s passing=true option requires relevant checks to be passing. Consul DNS service lookups use a different default policy: warning instances can remain eligible, while critical instances are excluded. Setting dns_config.only_passing=true excludes warning states too. Therefore, a warning instance appearing in DNS but missing from the passing-only API response need not indicate a broken check.

When comparing outputs, use the same service, datacenter, namespace where applicable, and observation window. Consul normally listens for DNS on port 8600; querying the system resolver without appropriate forwarding is a different path. Querying Consul directly removes that intermediary, but does not turn the answer into a probe of the application from the caller’s network.

Freshness has several stages

Probe intervals, probe timeouts, consecutive-failure thresholds, agent-to-server propagation, read consistency, and client caching can all contribute delay. A 30-second interval is not a guaranteed 30-second end-to-end removal bound. A TTL check expires relative to its most recent update; a failure halfway through its TTL does not necessarily leave a whole additional TTL of eligibility.

Anti-entropy repairs differences between agent state and the catalog. Its documented periodic interval is approximate and depends on cluster size. Registration changes also trigger updates. Failed synchronization can continue across multiple intervals, so the periodic schedule is neither the only update mechanism nor a maximum convergence deadline.

HTTP reads normally use the default consistency mode; DNS normally permits stale reads. A consistent HTTP read verifies leadership with a quorum, at additional cost. Even that stronger catalog read cannot discover a service failure that no check has observed yet. Catalog consistency and real-world readiness remain distinct.

When it breaks

A passing probe may test the wrong dependency, run from a different network, or miss a deadlock. Conversely, coupling every instance’s readiness to one shared dependency can remove the whole pool during a common outage. Choose checks according to useful request capability and the intended degraded behavior, then retain client timeouts and bounded recovery paths.

Do not diagnose an API/DNS mismatch from one screenshot. Check warning policy, relevant check IDs, ACL visibility, deregistration, and cached or stale observations first. A critical instance can remain in the unfiltered health response; its continued presence there is expected, not proof that filtering was ignored.

Consul health selection: mechanism and evidence boundaries.

Check it quickly

Save this as consul_filter.py and run python3 consul_filter.py. This standard-library exercise uses synthetic records, starts no Consul agent, and makes no network requests. Keep assertions enabled, without -O or PYTHONOPTIMIZE. It tests selection and stale snapshots, not Consul itself; this review did not rerun the retained exact fixture.

from copy import deepcopy

nodes = {"n1": "passing", "n2": "critical"}
services = [
 {"ID": "web-1", "Name": "web", "Node": "n1", "Status": "passing"},
 {"ID": "web-2", "Name": "web", "Node": "n1", "Status": "warning"},
 {"ID": "db-1", "Name": "db", "Node": "n1", "Status": "critical"},
 {"ID": "web-3", "Name": "web", "Node": "n2", "Status": "passing"},
]


def health_rows(name):
 # Synthetic snapshot: only the node and THIS service contribute checks.
 return [
 {"ID": item["ID"], "Checks": [nodes[item["Node"]], item["Status"]]}
 for item in services if item["Name"] == name
 ]


def selected(rows, allowed):
 return [row["ID"] for row in rows if all(s in allowed for s in row["Checks"])]


snapshot = health_rows("web")
assert [row["ID"] for row in snapshot] == ["web-1", "web-2", "web-3"]
assert selected(snapshot, {"passing"}) == ["web-1"]
assert selected(snapshot, {"passing", "warning"}) == ["web-1", "web-2"]
print("unfiltered: web-1 web-2 web-3; passing-only: web-1; default DNS policy: web-1 web-2")

previous = deepcopy(snapshot)
services[0]["Status"] = "critical"
current = health_rows("web")
assert selected(previous, {"passing"}) == ["web-1"] # An old snapshot stays old.
assert selected(current, {"passing"}) == []
assert len(current) == 3 # Failing health is not deregistration.
assert selected(current, {"passing", "warning"}) == ["web-2"]
print("service failure changes selection, not registration; an old snapshot still selects web-1")

nodes["n1"] = "critical"
assert selected(health_rows("web"), {"passing", "warning"}) == []
print("node failure excludes both web instances on n1; db-1 never controlled their earlier health")

The assertions keep unrelated database health out of web selection, distinguish warning from critical, and preserve failed registrations in the unfiltered view. Updating current health does not rewrite a previously copied snapshot. Node failure then removes both local web instances from the modeled DNS-eligible set.

For deployment validation, compare actual unfiltered and passing-only health responses with a direct DNS query under the configured warning policy. Record check transitions and client-visible removal separately. That integration was not executed here; the offline model provides no measured detection time, DNS packet evidence, or availability guarantee.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Consul Health API, service instances, checks, passing filter, and ACL visibility, checked 2026-10-15.
Consul health checks, initial states, script and HTTP outcomes, and TTL expiry, checked 2026-10-15.
Consul DNS configuration, only_passing and relevant checks, checked 2026-10-15.
Consul DNS behavior, port 8600 and caching controls, checked 2026-10-15.
Consul consistency, agent updates and approximate anti-entropy intervals, checked 2026-10-15.
Consul consistency modes, HTTP and DNS defaults and quorum checks, checked 2026-10-15.
Consul health-check reference, timeouts, thresholds, and critical-service deregistration, checked 2026-10-15.

The scheduler accounts for requests, not momentary utilization

Sun, 02 Apr 2028 00:00:00 +0000

Kubernetes scheduling answers whether a Pod can fit a node under the applicable resource requests and placement constraints. It does not simply look at a current CPU utilization chart and place work wherever the line is low. Requests provide accounting inputs for future demand; actual usage is a separate observation. Diagnose both before deciding that an idle-looking cluster has capacity for a particular Pod.

Read the admitted requests. API rejection, an accepted Pending Pod, and successful placement are different stages.

Account for the admitted specification

For ordinary application containers, the scheduler accounts for their requested resources when testing node fit. Initialization requirements, Pod overhead and supported Pod-level resource configuration can also affect the effective request. The final admitted Pod is the relevant input, including defaults and mutations, rather than just the source manifest that an operator remembers submitting.

Node allocatable is also different from the hardware’s total capacity. It represents the resource amount available to Pods under the node’s configuration. Other admitted workloads consume scheduling headroom through their requests even when their processes happen to be idle. A request does not mean that physical memory has already been allocated to the application or that CPU cores are exclusively assigned.

Limits constrain running workloads through the execution stack; they are not generally substituted for an explicitly smaller request in scheduler fit checks. A container can therefore have a CPU limit larger than one node’s CPU capacity and still schedule with a small request. That accepted limit does not manufacture additional processors or promise the process can reach the declared ceiling.

When it breaks

A Pending Pod may be blocked by insufficient requested resources, but also by taints, affinity, storage, topology or other constraints. Read the PodScheduled condition and scheduling events before changing resource values. Lowering requests cannot repair an incompatible node selector or unavailable volume, and blindly lowering an honest request can create runtime contention instead of a sound placement.

Defaulting makes omission easy to misread. When a limit is specified without a request, and no admission-time mechanism supplies another request, Kubernetes copies that resource’s limit into its request. Setting only a large limit can therefore produce a large scheduling demand. Inspect both maps on the stored object, including any LimitRange effects, before interpreting a failed placement.

The API also rejects a request greater than its corresponding limit. That is an admission failure, not a scheduler decision about cluster capacity. Keeping those failure stages separate helps explain why one manifest never creates a Pod while another produces an accepted but unscheduled Pod that can be inspected.

QoS is not an immunity promise. Under conventional per-container rules, Guaranteed requires CPU and memory requests and limits to be present and equal for every relevant container. Matching only memory is insufficient. Even a Guaranteed Pod can encounter resource failure; the class does not replace priority, capacity planning, application correctness or actual node-pressure analysis.

Check it quickly

The fixture and observations below are retained from the original article. This review did not rerun the Kubernetes integration; reported success is historical, not a new execution result.

Use a disposable Kubernetes cluster with at least one Ready, untainted worker, kubectl and python:3.11-slim-bookworm on its nodes. Save this as request-placement.py and run python3 request-placement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11, and the selected worker reported ten allocatable CPUs.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

from decimal import Decimal, ROUND_CEILING

def cpu_quantity(value):
 return Decimal(value[:-1]) / 1000 if value.endswith('m') else Decimal(value)

namespace = 'notes-requests-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 workers = [n for n in obj(namespace, 'nodes')['items'] if
 not n['spec'].get('unschedulable') and not n['spec'].get('taints')
 and any(c['type'] == 'Ready' and c['status'] == 'True' for c in n['status']['conditions'])]
 assert workers, 'a Ready worker without taints is required'
 worker = workers[0]
 allocatable = cpu_quantity(worker['status']['allocatable']['cpu'])
 too_large = str(int(allocatable.to_integral_value(rounding=ROUND_CEILING)) + 1)
 for name, request in [('burst', '10m'), ('oversized', too_large)]:
 podspec = spec()
 podspec['nodeSelector'] = {'kubernetes.io/hostname': worker['metadata']['labels']['kubernetes.io/hostname']}
 podspec['containers'][0]['resources'] = {
 'requests': {'cpu': request, 'memory': '24Mi'}, 'limits': {'cpu': too_large, 'memory': '96Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name}, 'spec': podspec})
 burst = wait_for(lambda: obj(namespace, 'pod', 'burst'), pod_ready)
 assert burst['spec']['nodeName'] == worker['metadata']['name']
 def insufficient(p):
 return any(c['type'] == 'PodScheduled' and c['status'] == 'False'
 and c.get('reason') == 'Unschedulable' and 'Insufficient cpu' in c.get('message', '')
 for c in p.get('status', {}).get('conditions', []))
 pending = wait_for(lambda: obj(namespace, 'pod', 'oversized'), insufficient)
 assert pending['status']['phase'] == 'Pending' and not pending['spec'].get('nodeName')
 only_limit = spec()
 only_limit['containers'][0]['resources'] = {'limits': {'cpu': '200m', 'memory': '64Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'defaulted'}, 'spec': only_limit})
 defaulted = wait_for(lambda: obj(namespace, 'pod', 'defaulted'), pod_ready)
 resources = defaulted['spec']['containers'][0]['resources']
 assert resources['requests'] == resources['limits'] == {'cpu': '200m', 'memory': '64Mi'}
 assert defaulted['status']['qosClass'] == 'Guaranteed'
 invalid = spec()
 invalid['containers'][0]['resources'] = {'requests': {'cpu': '200m'}, 'limits': {'cpu': '100m'}}
 denied = k('create', '-n', namespace, '-f', '-', payload={
 'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'invalid'}, 'spec': invalid}, expected=None)
 assert denied.returncode != 0 and 'must be less than or equal to cpu limit' in denied.stderr
 print(json.dumps({'node_allocatable_cpu': str(allocatable), 'limit_cpu': too_large,
 'small_request_with_large_limit_scheduled': True,
 'larger_than_node_request_pending': True,
 'limit_only_defaulted_resources': resources,
 'defaulted_qos': defaulted['status']['qosClass'],
 'request_above_limit_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script uses nodeSelector to target that worker while leaving the scheduler responsible for assignment. Both comparison Pods have an eleven-CPU limit. The first requests only ten millicpu and becomes Ready on the selected node. The second requests eleven CPUs, remains Pending without nodeName, and reports Unschedulable with Insufficient cpu. Neither Pod tries to consume eleven CPUs; their processes are idle.

A third Pod specifies only 200m CPU and 64Mi memory limits. The stored object contains identical requests, and its observed QoS class is Guaranteed. A fourth attempted Pod requests 200m while limiting CPU to 100m; the API rejects it. The original article reports that all assertions passed, and the namespace was removed.

This separates successful placement, scheduler rejection, defaulting and API validation with real objects. It does not test actual peak capacity, resource-manager exclusivity or every scheduling plugin. Size requests from workload evidence and review actual admitted demand alongside usage, instead of turning an idle chart or a familiar ratio into a placement guarantee.

Sources

Kubernetes: Resource Management, request-based placement, limits and admission defaulting, checked 2026-10-15.
Kubernetes: Pod Quality of Service Classes, Guaranteed requirements and the limits of QoS protection, checked 2026-10-15.

Baggage carries context across an explicit boundary

Mon, 27 Mar 2028 00:00:00 +0000

OpenTelemetry baggage is a context-associated collection of string key-value pairs. It lets information known near a request’s entry point travel to downstream operations: a deployment region or a rollout cohort, for example. Unlike a span attribute, baggage is intended for propagation. Unlike trace context, it describes application-selected information rather than the identifiers and flags connecting spans. These mechanisms can travel together without being the same data structure.

Propagation still needs a working boundary. An outgoing client injects context into a carrier such as HTTP headers; the receiver extracts that carrier into its local context. Instrumentation can perform these steps automatically when configured, while manually instrumented code must do them explicitly. Merely creating baggage in one process does not deliver it to an unrelated process, a background task, or an external service.

Choose what crosses and what gets recorded

Baggage entries do not automatically become span, log, or metric attributes. A downstream operation must read the value and copy it deliberately, or use a configured processor implementing that policy. This distinction matters when a team expects a searchable attribute but has only enabled propagation. It also prevents the opposite mistake: copying every incoming key into telemetry without considering its meaning, size, or cardinality.

Context operations return a new context rather than changing every existing reference. Keep the returned context and pass or attach it at the intended boundary. When removing entries for an outgoing request, preserve the trace context that should accompany that request. Treat an outbound allowlist and a telemetry-attribute allowlist as explicit decisions; the values appropriate for one receiver need not be suitable for every later destination.

When it breaks

Baggage has no built-in proof that a value was issued by a trusted component. A caller can supply a header claiming an expensive account tier or a privileged role. Parsing that header successfully establishes its representation, not its authority. Authentication and authorization must use their own trusted inputs. Small, plausible-looking values deserve the same trust analysis as obviously suspicious ones.

Unbounded baggage also increases request-header sizes and can spread sensitive data to unintended recipients, including third-party APIs reached by automatic instrumentation. Limit names and values, review the destinations that receive them, and use propagators for wire encoding. A successful round trip between two copies of one SDK is narrower evidence than interoperability across every language and proxy in a production path. Test those actual boundaries when they matter.

Baggage propagation and trust: mechanism and evidence boundaries.

Check it quickly

Create a virtual environment, install opentelemetry-sdk==1.35.0, save the script as baggage.py, and execute it with that environment’s Python. The author reported an earlier Python 3.14.4 run; retained exact records cover Python 3.12.14 with SDK 1.35.0. This source review did not rerun it. It uses the real W3C trace-context and baggage propagators with an in-memory span exporter. An explicit always-on sampler avoids inherited sampling settings. Keep assertions enabled, without -O or PYTHONOPTIMIZE: force_flush is called inside assert. It does not start an HTTP server: the dictionary is the carrier that an HTTP client would subsequently send.

import json

from opentelemetry import baggage, trace
from opentelemetry.context import Context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ALWAYS_ON
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

base = Context()
raw = baggage.set_baggage("region", "eu-west", context=base)
raw = baggage.set_baggage("release", "canary/2", context=raw)
raw = baggage.set_baggage("internal_note", "synthetic-only", context=raw)
assert not baggage.get_all(base)
propagator = CompositePropagator([TraceContextTextMapPropagator(), W3CBaggagePropagator()])
provider = TracerProvider(sampler=ALWAYS_ON)
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("baggage-lab")
with tracer.start_as_current_span("parent", context=Context()) as parent:
 outgoing = trace.set_span_in_context(parent, baggage.clear(context=raw))
 for key in ("region", "release"):
 outgoing = baggage.set_baggage(key, baggage.get_baggage(key, raw), context=outgoing)
 carrier = {}
 propagator.inject(carrier, context=outgoing)
 received = propagator.extract(carrier, context=Context())
 assert set(baggage.get_all(received)) == {"region", "release"}
 assert baggage.get_baggage("region", received) == "eu-west"
 assert baggage.get_baggage("release", received) == "canary/2"
 assert baggage.get_baggage("internal_note", raw) == "synthetic-only"
 assert trace.get_current_span(received).get_span_context().trace_id == parent.get_span_context().trace_id
 with tracer.start_as_current_span("inherited", context=received):
 pass
 with tracer.start_as_current_span("copied", context=received) as span:
 span.set_attribute("deployment.region", baggage.get_baggage("region", received))
assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert not spans["inherited"].attributes
assert spans["copied"].attributes == {"deployment.region": "eu-west"}
independent = propagator.extract({"traceparent": "invalid", "baggage": "region=eu-west"},
 context=Context())
assert not trace.get_current_span(independent).get_span_context().is_valid
assert baggage.get_baggage("region", independent) == "eu-west"
untrusted = propagator.extract({"baggage": "tier=gold"}, context=Context())
assert baggage.get_baggage("tier", untrusted) == "gold"
print(json.dumps({"wire_fields": sorted(carrier),
 "wire_baggage": carrier["baggage"],
 "automatic_span_attributes": dict(spans["inherited"].attributes),
 "explicit_span_attributes": dict(spans["copied"].attributes),
 "baggage_survives_invalid_traceparent": True,
 "caller_supplied_tier_was_parsed": True}, indent=2))
provider.shutdown()

The original context contains region, release, and a synthetic internal note. The outgoing context keeps only the first two entries while retaining the parent span. The extracted region is eu-west, and the release value canary/2 round-trips through the encoded header. Assertions also confirm that constructing the filtered context did not remove the note from the original context or change the empty base context.

Two child spans demonstrate the recording boundary. inherited receives baggage but has no attributes; copied contains only the explicitly assigned deployment.region. A malformed traceparent is tested separately: its trace context is invalid, while the independently encoded baggage still extracts successfully. Finally, a caller-supplied tier=gold is accepted as data without any identity check. These results cover serialization, filtering, and recording in this SDK fixture; the trace assertion compares trace_id, not every trace-context field. Before relying on the same policy across services, verify the real client instrumentation, receiver extraction, outbound filtering, and the attribute mapping used by the destination backend.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
OpenTelemetry baggage, baggage is separate from telemetry attributes and lacks built-in integrity checks, checked 2026-10-15.
W3C Baggage, HTTP representation, value encoding, and propagation limits, checked 2026-10-15.
W3C Trace Context, trace identity travels separately from application baggage, checked 2026-10-15.
OpenTelemetry Python sampling, explicit sampler selection and environment defaults, checked 2026-10-15.

The SYN backlog holds incomplete handshakes

Fri, 24 Mar 2028 00:00:00 +0000

A listening TCP socket separates incomplete handshakes from established connections waiting for accept(). The real setting is tcp_max_syn_backlog; this article retains its historical URL spelling. It concerns remembered incomplete requests, whereas the listen() backlog concerns the accept queue. Diagnose which stage is under pressure before choosing a remedy.

SYN cookies provide a fallback when normal handshake state cannot be retained. They do not make servers immune to overload, guarantee delivery, or remove the need to understand the SYN queue. Packet loss, filtering, allocation failures and accept-side pressure can still prevent connections with cookies enabled.

The SYN queue holds incomplete handshakes; the accept queue holds established connections awaiting accept(). With listen backlog configured as 8192 and somaxconn as 4096, the accept-side cap is 4096. Buffering more waiting work does not itself increase processing rate. The source article remains held/pending. Whole article remains held/pending; this diagram does not endorse its disputed claims.

The backlog argument is the accept queue

Linux changed the listen() backlog meaning in 2.2. Its configured value is capped by somaxconn; with arguments 8192 and 4096 respectively, that cap is 4096. This is a configuration example, not a measurement of exact admission count. Read the actual listener and namespace rather than assuming defaults or inferring capacity from a short burst.

Kernel documentation describes tcp_max_syn_backlog as a per-listener limit on remembered requests in SYN_RECV. Its initial value depends on memory and kernel implementation. Old man-page memory thresholds should not be treated as the current host’s configuration. Neither queue length alone identifies the application’s processing rate.

SYN cookies remain a fallback

With kernel support, tcp_syncookies=1 enables the overflow fallback; value 2 is a testing mode that sends cookies unconditionally. The listen manual’s statement that cookies remove the logical bound describes stateless admission beyond remembered requests. It does not mean unlimited memory, CPU, bandwidth or established sockets. The kernel documentation still recommends considering SYN-backlog tuning for legitimate overload, alongside the application’s ability to drain work.

Do not promise that every cookie connection loses SACK and window scaling. The pinned Linux v6.12 implementation can encode these options in timestamp bits when timestamps are negotiated. Which options survive depends on implementation and negotiation. That qualified behavior does not turn an emergency mechanism into a capacity plan, and disabling cookies is not a general fix.

The accept queue is where the pressure lands

Accept-side pressure can coexist with SYN-side pressure. In Linux v6.12’s ordinary handshake path, failure to create a child socket can reach an overflow branch: with tcp_abort_on_overflow disabled, the server can retain the request rather than immediately reset it. A client may already consider the connection established while application traffic stalls. This is a possible state mismatch, not a guarantee that every connect() succeeds.

A larger queue absorbs a finite burst and may increase waiting time. It does not establish higher steady-state service capacity. Measure arrival rate, accept latency and application completion together; there is no measured 20,000-connections-per-second result in this article.

When it breaks

Rising overflow counters justify investigation, but do not uniquely blame a worker thread. Accept-loop scheduling, resource exhaustion, other listeners in the same network namespace and a burst between samples can change the interpretation. Historical nonzero totals are not evidence of a current incident.

SYN loss also occurs with cookies enabled. Client retry timing depends on kernel version, retry settings, socket behavior and application deadlines; 67 or 131 seconds is not a universal timeout contract. A successful TCP connection likewise does not prove that the application request completed.

Check it quickly

Use the following observation exercise only on an authorized Linux system with an existing listener on port 8443, in its network namespace. It requires ss, nstat and sysctl already installed. Do not start listeners, generate overload, change settings or elevate privileges for this exercise. These commands were not run in this review.

ss -lnt 'sport = :8443'
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPReqQFullDrop TcpExtTCPReqQFullDoCookies
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_syncookies net.ipv4.tcp_abort_on_overflow

Take two snapshots around an already-approved ordinary workload. For a listening TCP socket, Recv-Q reports queued established connections and Send-Q the effective backlog limit. Empty output means no matching visible listener, not a zero-length queue. A changed sysctl does not retroactively prove how an existing listener was configured.

nstat -az shows absolute counters, including zeros, and may update its local history; compare deltas over the interval. The named counters separate useful overflow, request-drop and cookie-fallback signals, but they are namespace-wide and are not mutually exclusive per-listener verdicts. Record kernel version, listener context and request outcomes before proposing tuning. Missing tools, access or an isolated test environment leave the runtime conclusion unresolved.

Sources

Kernel ip-sysctl documentation, SYN request limits, cookie modes and overload guidance; earlier check retained 2026-09-16, checked 2026-10-19.
Linux listen(2), accept backlog semantics and the configured somaxconn cap; earlier check retained 2026-09-16, checked 2026-10-19.
Linux tcp(7), version-sensitive settings and retry behavior; earlier check retained 2026-09-16, checked 2026-10-19.
Linux v6.12 cookie implementation, timestamp encoding of SACK and window scaling, checked 2026-10-19.
Linux v6.12 handshake implementation, conditional listen_overflow behavior, checked 2026-10-19.
iproute2 nstat manual, absolute-counter and zero-value options, checked 2026-10-19.

ZFS snapshots give the filesystem a rewind button

Tue, 21 Mar 2028 00:00:00 +0000

A ZFS snapshot preserves a point-in-time dataset view through copy-on-write references instead of a full duplicate of its data. Existing blocks remain available while references need them. Creation still has metadata and synchronization work; neither zero allocation nor completion before a shell redraw is a measured guarantee. This review did not run ZFS or recover the earlier claimed fixture’s raw output.

That makes retained history useful for local recovery, but every snapshot of tank/data remains in the same pool. A failure that makes that pool unavailable can also make its snapshots unavailable. This is a shared failure boundary, not a claim that every controller fault atomically destroys all blocks. Independent backups or replication need their own retention and verified recovery.

A name for one consistent tree

Snapshots atomically capture the dataset’s filesystem state; recursive -r snapshots include descendants at the same instant. Without recursion, child datasets are omitted rather than automatically captured later. Filesystem consistency is not an application checkpoint: database buffers, logs and files outside the selected datasets still require the application’s supported backup or crash-recovery procedure. Coordinate all required datasets before treating the set as recoverable.

Names such as tank/data@daily-2026-09-16 can help retention, alongside creation times, properties and backup records. Naming alone does not prove eligibility for deletion. zfs list -t snapshot -o name,used,refer,creation -s creation inspects snapshots and their accounting; dependencies and holds must also be checked before a pruning decision.

Where the space is, in zfs list -o space

zfs list -o space tank/data selects NAME, AVAIL, USED, USEDSNAP, USEDDS, USEDREFRESERV and USEDCHILD; it does not include REFER. USED decomposes into the four used-by categories, not REFER plus USEDSNAP. REFER is all data accessible through the dataset and can include shared blocks. Snapshot USED measures uniquely referenced space; deleting another snapshot can change it, so individual values do not simply sum to all recoverable space.

The earlier thirty-snapshot, 64 MiB dataset and 2 MiB daily-churn figures lack a recovered ZFS trace. The 60 MiB versus 1920 MiB comparison is illustrative arithmetic, not verified allocation. Metadata, compression, block size, sharing and the exact retention timeline affect actual cost. A recent snapshot may have little unique data while the whole retained set still prevents substantial reclamation.

Holds, diff, and comparing versions

A user hold prevents destruction until released. zfs hold keep tank/data@daily-2026-09-16 sets one; zfs holds tank/data@daily-2026-09-16 inspects it, and zfs release keep tank/data@daily-2026-09-16 releases that tag. Holds can protect an in-progress transfer, but clone-origin dependencies are separate and need not appear as named hold tags. Release only after understanding the dependent operation.

zfs diff tank/data@daily-2026-09-15 tank/data@daily-2026-09-16 reports changes between the retained versions: created, removed, modified and renamed paths. Filesystem snapshots are mounted on demand beneath .zfs/snapshot/<name>; snapdir=visible exposes that directory in listings. Snapshot access and a destructive dataset rollback are different recovery choices, and neither verifies application correctness by itself.

When it breaks

A successful send/receive into independently protected storage can survive loss of the source pool. The shown incremental transfer requires an existing destination whose latest snapshot matches the source base; a bare incremental stream is not a complete standalone backup. A second pool on the same machine can still share power, controller or administrator risks. Verify received data and a usable restore before claiming protection.

Growing USEDSNAP warrants capacity and retention review, but eighty-percent occupancy is not a universal write-failure threshold. Quotas, reservations and actual pool availability matter. A clone depends on its origin snapshot; deleting that origin or rolling back past newer dependent versions needs deliberate dependency handling. Rollback can discard newer data, and destroying clones is not the only possible recovery strategy.

ZFS history has dependencies — mechanism diagram.

Check it quickly

This unexecuted standalone Bash recipe requires exclusively controlled disposable datasets, existing source snapshots, and a receiver previously seeded from the base. The destination must be unchanged since that base, mounted at the bound path without intervening mounts or symlinks; the hold tag must be unused. Before taking the target snapshot, prepare regular-file probe.txt in the source and keep a trusted external copy at expected. Listing is recursive; transfer covers only tank/data, not descendants.

(
 set -euo pipefail
 src='tank/data'
 dst='backup/data'
 base='daily-2026-09-15'
 next='daily-2026-09-16'
 tag='example-transfer'
 mount='/mnt/zfs-lab/received'
 expected='/mnt/zfs-lab/expected-v2.txt'

 zfs list -r -t snapshot -o name,used,refer,creation -s creation "$src" "$dst"
 zfs list -o name,avail,used,usedsnap,usedds,usedrefreserv,usedchild,refer "$src"
 zfs holds "$src@$base" "$src@$next"
 zfs diff "$src@$base" "$src@$next"

 latest=$(zfs list -H -r -t snapshot -o name -s createtxg "$dst" |
 awk -F@ -v dataset="$dst" '$1 == dataset { last=$0 } END { print last }')
 test "$latest" = "$dst@$base"
 source_guid=$(zfs get -Hp -o value guid "$src@$base")
 target_guid=$(zfs get -Hp -o value guid "$dst@$base")
 test -n "$source_guid"
 test "$source_guid" = "$target_guid"
 target_mount=$(zfs get -H -o value mountpoint "$dst")
 target_mounted=$(zfs get -H -o value mounted "$dst")
 test "$target_mount" = "$mount"
 test "$target_mounted" = yes
 test -f "$expected"
 test ! -L "$expected"

 zfs hold "$tag" "$src@$base" "$src@$next"
 zfs send -i "$src@$base" "$src@$next" | zfs receive "$dst"
 source_guid=$(zfs get -Hp -o value guid "$src@$next")
 target_guid=$(zfs get -Hp -o value guid "$dst@$next")
 test -n "$source_guid"
 test "$source_guid" = "$target_guid"
 test -f "$mount/.zfs/snapshot/$next/probe.txt"
 test ! -L "$mount/.zfs/snapshot/$next/probe.txt"
 cmp -- "$expected" "$mount/.zfs/snapshot/$next/probe.txt"
)

The latest receiver snapshot and base GUID must match; equal names alone are insufficient. Failed checks stop the subshell, and pipefail exposes either transfer failure. Received GUID and fixture-byte comparisons still do not establish a valid application restore: verify the complete manifest, metadata and application recovery separately. Any holds created remain after failure or success; after validation and dependency review, release only this exercise’s tag with zfs release example-transfer tank/data@daily-2026-09-15 tank/data@daily-2026-09-16. No force rollback or automatic cleanup is included.

Sources

zfs-snapshot(8), snapshot creation semantics and recursive snapshots, checked 2026-09-16.
zfs-list(8), the space column set and separate REFER property, checked 2026-09-16.
zfs-hold(8), holds, zfs holds, and the EBUSY returned by destroy, checked 2026-09-16.
zfs-send(8), incremental send streams and their base-snapshot requirements, checked 2026-09-16.
zfsprops(7), used-by decomposition, shared references, quotas and reservations, checked 2026-10-18.
zfs-clone(8), clone dependence on its origin snapshot, checked 2026-10-18.
zfs-rollback(8), discarded newer data and dependent clone handling, checked 2026-10-18.
zfs-receive(8), matching destination base for incremental receive, checked 2026-10-18.
PostgreSQL filesystem backup, consistent snapshot scope, WAL and multi-volume recovery requirements, checked 2026-10-18.
ZFS concepts, clone dependency on its origin snapshot and shared storage boundaries, checked 2026-10-18.
zfs-get(8), exact property reads used for GUID and mountpoint checks, checked 2026-10-19.

Hugo publication flags decide what enters the artifact

Sat, 18 Mar 2028 00:00:00 +0000

Hugo decides which pages to render when a build runs. Front matter supplies publication metadata, configuration and command flags supply inclusion rules, and the build clock supplies the time used for evaluation. The deployed directory then stays unchanged until another artifact replaces it. A date reaching midnight is not itself a deployment event, and a file without draft: true is not automatically a draft.

Separate publication metadata from build options

Under the default rules, a page marked draft: true is excluded. A future publication date also excludes it, while an expired page is excluded after its expiry date. Hugo resolves dates according to its front matter configuration; an explicit publishDate can differ from the displayed date. Use complete timestamps with a time zone for a boundary that must mean the same thing on a laptop and a build worker.

--buildDrafts includes drafts, --buildFuture includes future content, and --buildExpired includes expired content. Their short forms are -D, -F, and -E. These switches address separate conditions: a future draft needs both draft and future inclusion before the date arrives. Project or environment configuration can enable the same behavior, so checking only a page’s front matter is insufficient.

This site’s configuration deliberately enables buildFuture: true to retain its published article inventory, including articles dated in the future. Consequently, those dates are not an embargo here. A tutorial about Hugo defaults must not be mistaken for a claim about this site’s active settings. Compare the effective options with the expected URL inventory before changing publication policy.

A preview artifact can still be published

A preview flag changes the generated output, not its sensitivity or deployability. If a pipeline uploads a directory built with -D -F, those pages can become public. Use a separate destination for preview builds and construct the release artifact with the intended options. A Git release tag identifies source history; it neither changes Hugo’s inclusion rules nor certifies that a particular generated directory was reviewed.

Scheduled publication requires a later build and deployment after the publication boundary. Conversely, changing a public page back to draft cannot withdraw HTML that is already deployed or cached. Rebuild, replace the served artifact, and inspect the public route. Hugo normally leaves unrelated files in an existing destination, so a fresh output directory or an appropriate clean build is part of verifying that excluded pages are absent.

When it breaks

A preview build copied over a release directory can leave a draft accessible by its direct URL even after the navigation no longer lists it. Inspect generated files, feeds, sitemaps, and the actual served route; hiding an index entry is not removal. A draft section page also does not automatically make all descendants drafts. Use explicit child metadata or the documented cascade mechanism when exclusion must apply to a subtree.

Time-zone mismatches cause a different failure: two workers include different pages from the same commit. Pin the generator, use an explicit build clock for boundary tests, and record the options. Do not repair the discrepancy by changing publication dates until the date resolution rules and worker time zones are understood.

Hugo inclusion gates: mechanism and evidence boundaries.

Check it quickly

With Python 3 and Hugo 0.165.0 available, this isolated experiment creates five pages and eight fresh build destinations. Run Python with assertions enabled: omit -O and unset PYTHONOPTIMIZE. It neither changes this repository nor deploys anything. This source review did not execute it.

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess

with TemporaryDirectory(prefix="hugo-publication-") as directory:
 root = Path(directory)
 (root / "content/posts").mkdir(parents=True)
 (root / "layouts/_default").mkdir(parents=True)
 (root / "hugo.yaml").write_text("baseURL: https://example.org/\n")
 (root / "layouts/index.html").write_text("<h1>Publication test</h1>")
 (root / "layouts/_default/list.html").write_text("{{ .Title }}")
 (root / "layouts/_default/single.html").write_text("{{ .Title }}")
 for name, extra in {
 "visible": "",
 "draft": "draft: true\n",
 "future": "publishDate: 2030-01-01T00:00:00Z\n",
 "future-draft": "draft: true\npublishDate: 2030-01-01T00:00:00Z\n",
 "expired": "expiryDate: 2025-12-01T00:00:00Z\n",
 }.items():
 (root / f"content/posts/{name}.md").write_text(
 f"---\ntitle: {name}\ndate: 2025-01-01T00:00:00Z\n"
 + extra + "---\nPublication fixture.\n"
 )
 cases = [
 ("normal", "2026", [], {"visible"}),
 ("drafts", "2026", ["-D"], {"visible", "draft"}),
 ("future", "2026", ["-F"], {"visible", "future"}),
 ("preview", "2026", ["-D", "-F"], {"visible", "draft", "future", "future-draft"}),
 ("expired", "2026", ["-E"], {"visible", "expired"}),
 ("all", "2026", ["-D", "-F", "-E"], {"visible", "draft", "future", "future-draft", "expired"}),
 ("boundary", "2030", [], {"visible", "future"}),
 ("later", "2031", [], {"visible", "future"}),
 ]
 for name, year, flags, expected in cases:
 destination = root / f"output-{name}"
 subprocess.run(
 ["hugo", "--source", str(root), "--destination", str(destination),
 "--clock", year + "-01-01T00:00:00Z", *flags],
 check=True, capture_output=True, text=True,
 )
 actual = {p.parent.name for p in destination.glob("posts/*/index.html")}
 assert actual == expected, (name, actual, expected)
 print(name, sorted(actual))

The assertions verify default exclusion, each inclusion flag, their combination, and publication at and after the boundary. Both drafts remain absent from later normal builds. A future draft appears early only with both -D and -F, while -E independently includes the expired page. This checks Hugo’s rendering decisions under controlled inputs; it does not test the deployment scheduler, CDN invalidation, or this site’s future-inclusive configuration. Those need separate artifact and public-route evidence at release time.

Keep these cases when changing publication options: ordinary, future, draft, future-draft, and expired pages exercise different inclusion conditions. Check newly generated files rather than assuming that missing navigation links prove exclusion.

Sources

Hugo: Basic usage, draft, future, expired content, and destination behavior, checked 2026-09-06.
Hugo: Front matter, publication metadata and cascading settings, checked 2026-09-06.
Hugo command reference, inclusion switches and the build clock, checked 2026-09-06.
Python assert statement, optimization removes assertions and their expressions, checked 2026-10-18.

iotop shows which process is burning the disks

Wed, 15 Mar 2028 00:00:00 +0000

iotop connects task-level kernel accounting with a system-wide storage investigation. It can identify tasks reporting substantial IO or waiting, but those are different observations. A task blocked behind other work may be a victim rather than the source of contention. Kernel counters do not provide perfect attribution for every filesystem, cache, or storage layer.

Use iotop beside device and application measurements. Neither an active writer nor a high wait value alone proves a storage bottleneck. Compare the same time window before deciding which component deserves closer inspection.

Per-process counters

Check the installed implementation and version. In the cited iotop-c, main-process rows already include thread activity; -P hides separate thread rows. Other variants need their own manual. The -o option filters for observed IO activity; it does not classify harmful work. Read/write byte rates and IO-wait percentages have different units and should not be ranked as interchangeable measures.

Linux read_bytes accounts storage-layer reads; cached reads need not produce them. write_bytes is accounted when pages are dirtied, not necessarily when the device completes a write. Cancelled writes and later writeback can separate task totals from device throughput. Delay accounting records task waiting, including synchronous block-IO delay; CPU iowait is a separate aggregate and is not a reliable per-task blame measure.

See whatever sweeps the disk

Choose an explicit sort column and observe long enough to distinguish sustained work from a burst. A high write rate can be an expected backup, index build, or logging workload. Correlate process ownership, application role, and current work before changing scheduling or limits. A high rank identifies a candidate, not the correct remedy.

Resolve the task in the relevant PID namespace and verify that it still refers to the intended process. Command lines, parent relationships, and open files provide context, but an open file is not proof that all observed writes went there. PIDs can be reused between measurements; a stale number can point at unrelated work.

Watch before you act

Observation has costs and access requirements. Additional syscall tracing may perturb the process and expose arguments, so choose a separately authorized, scoped investigation when needed. An append flag belongs to opening a file; it is not a flag on each write syscall. Avoid presenting an imagined combination of outputs as a measured diagnosis.

Record collection conditions for a useful baseline: implementation, interval, process/thread mode, accumulation mode, and comparable workload. With the documented default one-second delay, -b -n 60 is roughly a minute, not five minutes. Device await includes queueing and service time; near-100-percent utilization does not establish saturation on every parallel SSD or RAID device.

When it breaks

Missing permissions, kernel accounting options, or enabled task delay accounting can cause failure or unavailable fields. An empty display does not exonerate either your processes or other users’ processes. Record the actual limitation and required environment; do not silently change capabilities or kernel settings to make the tool work. Some delay-accounting changes affect only tasks started afterward.

Bursty activity can disappear in averages, and short-lived tasks may escape sampling. Varying intervals changes what is visible. Accumulated mode summarizes observed accounting within its applicable lifetime; it does not recover an unrecorded previous hour or guarantee complete coverage of exited tasks. State the observation window with every conclusion.

Compare task bytes, task waiting and device observations over the same interval. Capture and identity checks remain unexecuted; a fresh snapshot cannot prove identity at an earlier sample.

Check it quickly

This unexecuted proposal targets the cited iotop-c interface and sysstat iostat. Confirm the installed implementations first. It captures full reports, stderr and both statuses without selecting a process from text. A fresh configuration directory prevents saved iotop preferences from silently changing the view.

(
 capture=$(mktemp -d) || exit 1
 printf 'Evidence directory: %s\n' "$capture"
 iotop --version || exit $?
 iostat -V || exit $?
 LC_ALL=C iostat -x -y -t 2 5 > "$capture/iostat.txt" 2> "$capture/iostat.err" &
 stat_pid=$!
 if LC_ALL=C XDG_CONFIG_HOME="$capture" iotop -b -P -k -t -d 2 -n 5 \
 --no-only --no-accumulated --no-accum-bw \
 > "$capture/iotop.txt" 2> "$capture/iotop.err"; then
 task_status=0
 else
 task_status=$?
 fi
 if wait "$stat_pid"; then stat_status=0; else stat_status=$?; fi
 printf 'iotop_status=%s iostat_status=%s\n' "$task_status" "$stat_status"
 cat "$capture/iotop.err" "$capture/iostat.err"
 [ "$task_status" -eq 0 ] && [ "$stat_status" -eq 0 ] || exit 1
 cat "$capture/iotop.txt" "$capture/iostat.txt"
)

In iotop-c, -P hides separate thread rows; their activity is already included in process totals. Fixed kilobyte units and non-accumulated mode support comparisons. This batch capture does not promise descending write order. Compare the labelled DISK WRITE values within each report; for an interactive ranking, explicitly select that column and descending direction. Never treat the first printed row as the largest writer.

Compare overlapping timestamped intervals; launch times are not perfectly synchronized. -y excludes iostat’s since-boot report. Missing counters, nonzero status or incomplete windows invalidate the comparison.

Save this helper as inspect_io_process.py. Select a process PID manually from the full report, in the same PID namespace. Python 3.9+, Linux pidfd support and readable proc entries are required:

"""Read-only Linux identity snapshot; requires Python 3.9+ and pidfd support."""
import json
import os
import re
import select
import sys


def snapshot(token, proc_root="/proc", ops=os, poll_factory=select.poll):
 if not re.fullmatch(r"[1-9][0-9]*", token):
 raise ValueError("PID must be a positive decimal process ID")
 pid = int(token)
 pidfd = ops.pidfd_open(pid, 0)
 directory = None
 try:
 directory = ops.open(f"{proc_root}/{pid}", os.O_RDONLY | os.O_DIRECTORY)
 watcher = poll_factory()
 watcher.register(pidfd, select.POLLIN)

 def read(name, limit):
 fd = ops.open(name, os.O_RDONLY, dir_fd=directory)
 with ops.fdopen(fd, "rb") as source:
 data = source.read(limit + 1)
 if len(data) > limit:
 raise ValueError(f"{name} exceeds the snapshot limit")
 return data

 def identity():
 head, tail = read("stat", 65536).rsplit(b")", 1)
 if int(head.split(b" ", 1)[0]) != pid:
 raise ValueError("stat PID mismatch")
 fields = tail.split() # field 3 is index 0; field 22 is index 19
 if fields[0] in (b"Z", b"X", b"x"):
 raise ProcessLookupError("process has exited")
 return int(fields[19])

 before = identity()
 executable = ops.readlink("exe", dir_fd=directory)
 raw_command = read("cmdline", 65536)
 # Remove one final NUL; empty input means no argument data.
 command = raw_command[:-1] if raw_command.endswith(b"\0") else raw_command
 command = command.split(b"\0") if raw_command else []
 after = identity()
 if before != after or watcher.poll(0):
 raise ProcessLookupError("identity changed or process exited")
 return {"pid": pid, "starttime_ticks": before, "executable": executable,
 "argv": [arg.decode("utf-8", "backslashreplace") for arg in command]}
 finally:
 if directory is not None:
 ops.close(directory)
 ops.close(pidfd)


if __name__ == "__main__":
 try:
 if len(sys.argv) != 2:
 raise ValueError("usage: inspect_io_process.py PID")
 print(json.dumps(snapshot(sys.argv[1]), ensure_ascii=True))
 except (OSError, ValueError, IndexError, AttributeError) as error:
 sys.exit(f"Identity snapshot failed: {error}")

python3 inspect_io_process.py "${PID:?Set PID from the reviewed process row}"

A successful snapshot reports an executable, arguments and stable start-time value while checking a pidfd for exit. Permission, missing-process or unavailable-pidfd errors stop the check. PID reuse before acquiring the handle cannot be excluded; this snapshot does not prove identity at the earlier sample. Proc fields can change and are not atomic. Recheck before any later action; nothing is killed or reprioritized here.

Sources

iotop man page, rendered iotop-c manual; verify the installed variant, checked 2026-10-18.
vmstat man page, documents system-wide CPU and IO-wait reporting, checked 2026-09-05.
Linux delay accounting, taskstats delay scope and enablement prerequisites, checked 2026-10-18.
Linux proc filesystem, storage byte accounting, cancelled writes, and CPU iowait limits, checked 2026-10-18.
iostat manual, first report, await, and parallel-device utilization limits, checked 2026-10-18.
iotop-c upstream manual, process aggregation, batch options, configuration and interactive sorting, checked 2026-10-19.
sysstat upstream iostat manual, -y omits the since-boot report and -t timestamps interval reports, checked 2026-10-19.
pidfd_open manual, bounded identity or controlled-operation contract, checked 2026-10-19.
Python os interfaces, bounded identity or controlled-operation contract, checked 2026-10-19.
proc_pid_stat manual, bounded identity or controlled-operation contract, checked 2026-10-19.

A microcache serves fresh entries briefly and coalesces misses explicitly

Sun, 12 Mar 2028 00:00:00 +0000

A microcache keeps reusable responses for a short freshness lifetime, often a few seconds. Requests for the same selected representation can reuse a fresh entry during that window. The response might lag a new origin update, but it is not HTTP-stale merely because it came from a cache. Expiration, permission to serve stale content, and concurrent cache filling are separate policies.

When it breaks

A short TTL does not make a shared response safe for every user. If personalized requests read or populate a generic entry, one user’s representation can be served in another context. proxy_cache_bypass prevents reading from cache under its conditions; proxy_no_cache prevents saving the response. Using only one side can leave the other path inconsistent with the intended privacy boundary.

The cache key also defines which requests share a response. Ignoring a meaningful query or host dimension can merge different representations. Including arbitrary varying data can instead destroy reuse. The fixture below uses one backend authority and preserves the complete request URI, including arguments. A service varying by tenant, language, or other inputs must establish its own representation identity.

Caching alone does not guarantee one origin request during a cold burst. Several requests can miss the same entry before any response is stored. proxy_cache_lock coordinates filling a new entry, with finite timeout and age controls. That lock is not the TTL, and a brief freshness lifetime does not implicitly turn lock behavior on.

Freshness and filling are separate. Static illustration of the source and its limits. Reported fixture outcomes were not rerun for this graphic.

Check it quickly

The integration results below are historical observations; this revision did not rerun the fixture.

Save the script as check-microcache.py and run it with Python, Docker, and the pinned images. The recorded environment used nginx 1.28.0 on Linux arm64. A disposable backend reports per-path request counts, and nginx uses a two-second validity setting with explicit cookie and authorization bypass conditions. The listener is restricted to host loopback:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
from concurrent.futures import ThreadPoolExecutor
import threading
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from urllib.parse import urlsplit,parse_qs
import json,threading
lock=threading.Lock();counts={};gates={'/lock':threading.Event(),'/nolock':threading.Event()}
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def log_message(self,*args):pass
 def do_GET(self):
 parsed=urlsplit(self.path)
 if parsed.path=='/control/state':
 with lock:data=dict(counts)
 elif parsed.path=='/control/release':
 gates[parse_qs(parsed.query)['path'][0]].set();data={'released':True}
 else:
 with lock:counts[self.path]=counts.get(self.path,0)+1;count=counts[self.path]
 if parsed.path in gates:assert gates[parsed.path].wait(15)
 data={'path':self.path,'count':count,'cookie':self.headers.get('Cookie'),'authorization_present':bool(self.headers.get('Authorization'))}
 body=json.dumps(data,sort_keys=True).encode();self.send_response(200)
 if parsed.path=='/cookie':self.send_header('Set-Cookie','fixture=public-example; Path=/')
 if parsed.path=='/private':self.send_header('Cache-Control','private')
 if parsed.path=='/no-store':self.send_header('Cache-Control','no-store')
 self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
ThreadingHTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''
extra='''
proxy_cache_path /tmp/cache levels=1:2 keys_zone=pages:1m inactive=30s max_size=10m;
map $http_cookie $skip_cookie { default 1; "" 0; }
map $http_authorization $skip_auth { default 1; "" 0; }
proxy_cache pages; proxy_cache_key "$scheme|$proxy_host|$request_uri"; proxy_cache_valid 200 2s;
proxy_cache_bypass $skip_cookie $skip_auth; proxy_no_cache $skip_cookie $skip_auth;
proxy_cache_lock on; proxy_cache_lock_timeout 5s; proxy_cache_lock_age 5s;
add_header X-Cache $upstream_cache_status always;
'''
locations='''
location /control/ { proxy_cache off; proxy_pass http://backend:8081; }
location = /nolock { proxy_cache_lock off; proxy_pass http://backend:8081; }
location / { proxy_pass http://backend:8081; }
'''
with lab(config(locations,extra),backend=backend) as (_,request):
 def get(path,headers=None):
 status,h,b=request(path,headers=headers);assert status==200;return h.get('x-cache'),json.loads(b)
 results={}
 first=get('/item?x=1');second=get('/item?x=1');assert first[0]=='MISS' and second==('HIT',first[1])
 other=get('/item?x=2');assert other[0]=='MISS' and other[1]['path']!=first[1]['path']
 results['fresh']=[first[0],second[0]];results['different_query']=other[0]
 for name,headers in [('cookie',{'Cookie':'session=public-fixture'}),('authorization',{'Authorization':'Bearer public-fixture'})]:
 private=get('/item?x=1',headers);assert private[0]=='BYPASS' and private[1]!=first[1]
 assert get('/item?x=1')==('HIT',first[1])
 new='/new-'+name;assert get(new,headers)[0]=='BYPASS'
 public=get(new);assert public[0]=='MISS' and public[1]['cookie'] is None and not public[1]['authorization_present']
 results[name]={'warm_key':'BYPASS','public_after_private_miss':public[0]}
 for path in ['/cookie','private','no-store']:
 path='/'+path.lstrip('/');a=get(path);b=get(path)
 assert a[0]==b[0]=='MISS' and a[1]['count']==1 and b[1]['count']==2
 results[path]=[a[0],b[0]]
 deadline=time.monotonic()+6
 while time.monotonic()<deadline:
 value=get('/item?x=2')
 if value[0]=='EXPIRED':break
 assert value[0]=='HIT';time.sleep(.15)
 else:raise AssertionError('TTL did not expire')
 assert value[1]['count']==2;results['after_ttl']={'status':value[0],'count':value[1]['count']}
 for path,number,origin_count in [('/lock',6,1),('/nolock',4,4)]:
 barrier=threading.Barrier(number+1)
 def client():barrier.wait();return get(path)
 with ThreadPoolExecutor(max_workers=number) as pool:
 futures=[pool.submit(client) for _ in range(number)];barrier.wait()
 wait(lambda:get('/control/state')[1].get(path)==origin_count)
 time.sleep(.3);before=get('/control/state')[1][path]
 assert before==origin_count and not any(f.done() for f in futures)
 get('/control/release?path='+path);responses=[f.result(timeout=10) for f in futures]
 assert get('/control/state')[1][path]==origin_count
 statuses=sorted(r[0] for r in responses)
 if path=='/lock':assert statuses==['HIT']*5+['MISS'] and all(r[1]==responses[0][1] for r in responses)
 else:assert statuses==['MISS']*4
 results[path]={'origin_requests_while_held':before,'cache_statuses':statuses}
 print(json.dumps(results,sort_keys=True))

The first public request to /item?x=1 reports MISS, and the next reports HIT with the same backend result. /item?x=2 has its own MISS and distinct path in the response. Polling that second entry eventually produces EXPIRED and a backend count of two. This observes actual expiration using the server’s clock; it does not assume an exact subsecond transition time.

Requests carrying the example session cookie or authorization marker report BYPASS even when a public entry is already warm. A later public request still returns the original public HIT. A second control first visits an unused key with those headers, then visits it publicly: the public request is MISS, proving that the preceding private-context response did not populate a reusable entry.

Backend responses containing Set-Cookie, Cache-Control: private, or Cache-Control: no-store each produce two successive MISS results and increasing origin counts. The configuration does not ignore those response controls. This distinguishes client-side bypass policy from origin-provided storage restrictions; the short TTL is not used as an excuse to override either privacy signal.

For a cold locked entry, six clients start together while the backend deliberately withholds the response. The backend sees one request before release; afterward, one client reports MISS and five report HIT with the same result. For a separate entry with the lock disabled, four held clients create four origin requests and all report MISS. The gate makes overlapping misses directly observable.

Choose freshness and overload behavior separately

The two-second setting applies to eligible 200 responses in this experiment. Relevant origin cache headers can take priority over configured validity. Decide which responses are shareable before selecting the lifetime, and keep the cache key aligned with the inputs that actually change their content. A high hit ratio is not evidence that the selected representation was correct.

Lock waiters exchange extra waiting for reduced origin work. This fixture releases the origin within the configured lock limits; it does not establish a permanent single-request guarantee when those limits expire or a backend fails. Review lock timeout, lock age, and any stale-serving policy together with the service’s acceptable latency and failure behavior.

Observe cache status alongside origin counts and response identity. MISS means the requested representation was not served as a cache hit, while BYPASS identifies the explicit read exclusion shown here. Fresh HIT, EXPIRED refill, and intentionally stale service should not be collapsed into one “cached” result. Their different effects on correctness and backend load are the reason to measure them separately.

Sources

nginx proxy module, cache validity, cache locks, bypass, no-cache, and response header controls, checked 2026-10-15.
nginx content caching guide, cache keys, status observation, and caching behavior, checked 2026-10-15.
RFC 9111 HTTP Caching, fresh versus stale responses and private/no-store directives, checked 2026-10-15.

etcd elects a leader so writes happen in order

Thu, 09 Mar 2028 00:00:00 +0000

etcd uses Raft to order changes to its replicated state. Replicas can temporarily lag; consensus does not mean every member holds identical bytes at every instant. A leader coordinates replication, while the voting majority determines progress. Keep election, commitment, local application, and client acknowledgment separate when diagnosing whether an operation succeeded.

One leader per term, with voting restrictions

A candidate needs a majority of votes, and each voter grants at most one vote in a term. Log freshness also matters: a newer term alone does not make an outdated candidate eligible for every vote. Randomized election timeouts reduce repeated split votes. Terms order leadership attempts; they do not measure elapsed time or directly identify a network partition.

For an entry from its current term, a leader can advance commitment after replication to a majority. Earlier-term entries become committed indirectly as that current-term entry commits; merely counting old copies is not the complete Raft rule. Within the protocol’s storage and failure assumptions, later leaders preserve committed entries. Simultaneous loss of persistent data or restoring an old backup is a different disaster-recovery problem.

An election interrupts progress until a leader is established. Client requests can wait, time out, or fail according to their deadlines and connectivity. A timeout does not establish that a proposed operation never committed. Applications need an appropriate retry or transaction design rather than interpreting every failed response as permission to repeat a non-idempotent effect.

Count voters rather than processes

A voting configuration of N members needs floor(N/2)+1 for a majority. Three voters tolerate one unavailable member; four still tolerate one but need three participants. Five and six each tolerate two. An even count does not add another tolerated failure over the preceding odd count, although temporary even configurations can occur during controlled membership changes.

A learner receives replication but does not vote. Adding it leaves the voting quorum unchanged; promotion changes that count after catch-up checks. This avoids immediately depending on an empty new replica for voting, but catch-up traffic still consumes resources and can affect availability. An unchanged quorum is not a guarantee that adding a learner cannot contribute to timeouts or an election.

When it breaks

Without a reachable majority, the cluster cannot make normal write progress. Serializable reads can use one member’s local state and may return stale values. Linearizable reads need coordination that establishes current state; they are not simply local reads with a nicer label. They also need not append a new log entry for every read, so calling all reads replicated writes obscures the implementation.

The minority of a network partition cannot independently commit a competing history. Recovery behavior depends on the actual configuration. etcd 3.5 enables pre-vote by default to reduce disruption when a disconnected member rejoins. A healed partition therefore does not inevitably force a higher-term election and write outage. Inspect configuration, timing, and logs instead of asserting that every isolated member must have raised its term.

Losing a follower may leave enough voters to continue, but lag still reduces redundancy and affects local reads. Do not dismiss it until the majority disappears. Likewise, a failed endpoint probe can reflect client routing, TLS, permissions, timeouts, or an active alarm rather than a dead voter. Client reachability and peer-to-peer quorum connectivity are different observations.

Replication, then commitment. Source-based mechanism illustration. Model values and source-reported observations are not a new runtime test.

Check it quickly

Run this Python 3 arithmetic fixture without contacting a cluster. It prints voting thresholds and checks the learner distinction; it does not implement election or replication:

def quorum(voters):
 return voters // 2 + 1

for voters in (1, 2, 3, 4, 5, 6):
 required = quorum(voters)
 print(f"voters={voters}, quorum={required}, tolerated={voters-required}")
assert quorum(3) == 2 and quorum(4) == 3
assert 2 < quorum(4) # Neither side of a 2+2 partition has a majority.
# A learner replicates data but is not included in the voting count.
voters, learners = 3, 1
assert quorum(voters) == 2
print("PASS: adding a learner leaves the voting quorum unchanged")

The output gives quorums one, two, two, three, three, four for one through six voters. A two-plus-two split of four voters has no majority. Three voters plus one learner still require two votes. PASS establishes these calculations only, not the behavior of a live cluster under failure.

The original operational example below is retained, not rerun. These private addresses and the /registry/ prefix are placeholders: adapt them only to an authorized test cluster, with its TLS/authentication settings and non-sensitive fixture keys. No authenticated cluster or partition test was executed in this review:

etcdctl --endpoints=https://10.0.1.11:2379 endpoint status --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379,https://10.0.1.12:2379,https://10.0.1.13:2379 endpoint health --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 member list --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 --consistency=s get /registry/ --prefix --keys-only | head

The first command queries only one endpoint, so its table cannot prove that the cluster has exactly one leader. Collect status from every member before comparing leader identity, terms, and applied-index progress; samples need not be simultaneous. The final query intentionally uses local serializable semantics. head limits displayed lines, not the server-side range read. Success does not certify quorum health. Preserve probe errors and avoid changing membership merely to make a table look healthy.

Sources

etcd failure modes, leader loss, majority loss, and partitions, checked 2026-10-15.
Runtime reconfiguration, voting membership and learner promotion, checked 2026-10-15.
etcd learner design, non-voting replication and catch-up load, checked 2026-10-15.
etcd API guarantees, linearizable and serializable operations, checked 2026-10-15.
etcd 3.5 configuration, pre-vote enabled by default, checked 2026-10-15.
Raft paper, sections 5.2 and 5.4 on voting and current-term commitment, checked 2026-10-15.
etcdctl 3.5.21 endpoint checks, health read and alarm checks, checked 2026-10-15.

BPF tracepoints peek inside the running kernel

Mon, 06 Mar 2028 00:00:00 +0000

BPF programs can attach to selected events in a running kernel without rebuilding the application or restarting its workload. A userspace loader still loads and manages the probe. Collection changes resource use, so “live” does not mean invisible or free. Other profilers can also observe running programs; choosing BPF depends on the question and available instrumentation.

Define the event, aggregation key, observation interval, and excluded work before interpreting output. A syscall count, a scheduling delay, and a device latency distribution answer different questions. An entry count alone cannot reveal successful operations, time spent waiting, or which storage path served a request.

Probe without pausing

An eBPF program can accumulate counts or histograms in kernel maps while the workload continues. Aggregation reduces userspace output, but every observed event may still execute probe work. The verifier checks program safety under kernel rules; acceptance is not proof of correct measurement, negligible overhead, or an impossible kernel failure.

Start with a bounded question. Counting syscall entries needs an entry event; measuring syscall duration normally requires matching entry and exit state. Block-device latency needs its own event lifecycle. High-level tools such as bpftrace help express these measurements, but a short program does not establish that its units, filters, and correlations are correct.

Tracepoints are named events

Tracepoints expose named kernel events and are often less coupled to implementation details than probes on internal functions. They are not a stable ABI: names and fields can change across kernels. Kprobes can attach by symbol, rather than only by a hardcoded address, but internal function contracts can change too. Pin maintained probes to tested versions.

Inspect the available event and its field format on the intended kernel. A listing establishes visibility, not permission to load and attach a program. If an event is unavailable, record that limitation rather than silently substituting a different event whose count or timing means something else.

Read a histogram, not a firehose

Use aggregation to reveal distributions without printing every event. A latency histogram can suggest fast and slow paths, but its shape is evidence for investigation, not a diagnosis by itself. It cannot establish which path caused an outlier unless the probe measures the relevant distinction. Counts are not latency histograms.

Keep keys deliberate. Grouping by thread name can merge unrelated tasks with the same name; grouping by identifiers or arguments changes cardinality and cost. Apply relevant filters early, bound collection, and preserve the exact probe and workload context when reporting a finding. Compare against independent observations before attributing an application problem to one component.

When it breaks

Loading can fail because of permissions, kernel configuration, unavailable events, or tool-specific BTF or header requirements. Preserve the actual error. Listing tracepoints is not a successful attachment, and absent counts can mean an inactive workload, a mismatched filter, or failed collection. Container access also depends on the host kernel and its security policy; do not weaken protections to force a demonstration.

Hot-path probes can distort the workload even when they aggregate in-kernel. Compare cost under representative conditions and narrow collection when needed. Changing a filter can change the population itself, so a different distribution is not automatically proof that the cheaper observation is more accurate.

Entry attempts merge by current thread name. Illustrative counts are not measured output; the proposed scoped fixture remains unexecuted.

Check it quickly

This proposed foreground fixture targets bpftrace 0.24 and an already authorized host; it was not run here. Confirm that version, the exact tracepoint, and /bin/cat before continuing. Keep the evidence directory, stderr and status; do not add privileges to make it pass.

(
 work=$(mktemp -d) || exit 1
 printf 'Evidence directory: %s\n' "$work"
 cd "$work" || exit 1
 printf 'controlled input\n' > input || exit 1
 bpftrace --version || exit $?
 bpftrace -lv 'tracepoint:syscalls:sys_enter_openat' || exit $?
 if bpftrace -o counts.log -c '/bin/cat input' -e '
 tracepoint:syscalls:sys_enter_openat /pid == cpid/ {
 @opens[comm] = count();
 }' > workload.out 2> collector.err; then
 collector_status=0
 else
 collector_status=$?
 fi
 printf 'collector_status=%s\n' "$collector_status"
 cat collector.err
 if [ "$collector_status" -ne 0 ]; then exit "$collector_status"; fi
 cmp input workload.out || exit 1
 cat counts.log
)

In the reviewed 0.24 implementation, -c starts this finite child after probe attachment; pid == cpid restricts the event population. The observation follows the child lifetime, not a ten-second sleep. The child inherits the collector’s privileges. The shell records the collector’s exact foreground status, not the workload’s exit code. cmp checks copied bytes separately; it does not establish that exit code.

Expect entry counts for cat, including startup opens and failed attempts, rather than exactly one successful open. Equal thread names still merge within the selected population. Empty maps or errors are inconclusive. Attachment, shutdown and workload-exit behavior still need a permitted runtime review; source inspection is not execution evidence.

Sources

bpftrace project, project overview and tracing examples, checked 2026-10-18.
eBPF introduction, Official eBPF documentation, checked 2026-06-01.
bpftrace 0.24 language, tracepoint fields and count versus histogram aggregation, checked 2026-10-18.
bpftrace 0.24 standard library, comm identifies the current thread name, checked 2026-10-18.
Linux BPF design Q&A, tracepoint and kprobe attachment is not a stable ABI, checked 2026-10-18.
Linux BPF verifier, verification checks rather than measurement or overhead guarantees, checked 2026-10-18.
GNU Bash manual, wait without an explicit child ID does not return that child’s failure, checked 2026-10-18.
bpftrace 0.24 CLI, child lifetime, privileges, cpid filtering and separate tracing output, checked 2026-10-19.
bpftrace v0.24.0 runtime source, attachment precedes starting the non-USDT child in BPFtrace::run, checked 2026-10-19.

InnoDB covering indexes carry the query's columns

Fri, 03 Mar 2028 00:00:00 +0000

A covering index contains the columns needed to evaluate a particular query and return its requested values. In InnoDB, secondary-index records also carry the primary-key columns, so a query can be covered without explicitly listing the primary key again in the secondary-index definition. Coverage is a relationship between one query and an index, not a permanent property that applies to every query against the table.

When it breaks

Adding a projected column can remove coverage. An index that supplies an identifier and timestamp may not contain the large payload that a later application change begins selecting. The optimizer can still use that same index to locate matching rows, then obtain missing values from the clustered records. Index use and covering access are different facts.

Coverage also does not guarantee a selective access path. A wide scan can be fully covered while examining many entries. Column order, predicates, sorting requirements, and row estimates still matter. Do not interpret Using index as evidence that a query is fast, or confuse it with Using index condition, which describes index condition pushdown rather than complete projection coverage.

InnoDB visibility checks qualify the usual index-only explanation. Secondary records do not contain the same hidden transaction metadata as clustered records. Delete-marked secondary entries or pages changed by newer transactions can require clustered-record lookups to obtain the correct visible version. A covering plan is therefore not a universal promise of zero clustered access under concurrent MVCC activity.

Projection coverage and MVCC visibility are separate. Historical MySQL plans lack a recovered raw log; neither coverage nor pushdown proves speed.

Check it quickly

The unchanged fixture was not rerun in this source review. The retained review records say MySQL 8.4 was unavailable, and no raw MySQL run log was recovered in the retained validation files; the reported plans remain attributed historical observations.

Use a fresh MySQL 8.4 instance and a client session with permission to create a disposable schema. The recorded run used MySQL 8.4.11. Ensure xbcatm_index_lab is unused, stop on unexpected SQL errors, and keep the same session so its selected database remains active. Create ten thousand rows, with one hundred paid orders and a deliberately wider payload:

CREATE DATABASE xbcatm_index_lab;
USE xbcatm_index_lab;
CREATE TABLE orders (
 id INT PRIMARY KEY,
 status VARCHAR(10) NOT NULL,
 created_at DATETIME NOT NULL,
 payload VARCHAR(1000) NOT NULL,
 INDEX by_status_created (status,created_at)
) ENGINE=InnoDB;
INSERT INTO orders
WITH digits(n) AS (
 SELECT 0 UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3
 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6
 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9
)
SELECT number+1,IF(MOD(number+1,100)=0,'paid','open'),
 TIMESTAMP '2026-01-01 00:00:00' + INTERVAL (number+1) SECOND,
 REPEAT('x',640)
FROM (SELECT a.n+10*b.n+100*c.n+1000*d.n AS number
 FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d) AS numbered;
ANALYZE TABLE orders;
SELECT COUNT(*),SUM(status='paid') FROM orders;

Manually verify that the final counts are ten thousand and one hundred; these SELECT statements print results without automated assertions. The secondary index contains status and creation time explicitly; InnoDB supplies the primary identifier implicitly. Inspect and run a query that needs only those values:

EXPLAIN SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded tabular plan reports by_status_created and Using index. The analyzed plan identifies a covering index lookup and returns twenty rows. Result identifiers are one hundred, two hundred, and so on through two thousand, ordered by the timestamp and identifier. These are checked results on populated data, not a plan inferred from an empty schema.

Now request the payload as well:

EXPLAIN SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded plan still uses by_status_created, but loses Using index; the analyzed plan describes an index lookup without the covering qualification. This isolates the effect of changing the requested columns while keeping the filter, ordering, and index unchanged. The diagnostic execution does run the SELECT, so account for that when applying EXPLAIN ANALYZE to expensive real queries.

On success, remove the schema created for this exercise. MySQL DDL can commit implicitly, so an error or interruption does not roll the setup back; inspect and remove only this fixture schema afterward:

DROP DATABASE xbcatm_index_lab;

The timings from this tiny warm experiment are not a throughput or latency benchmark. Scheduling, caching, and instrumentation can outweigh the difference between two twenty-row reads. The verified conclusion concerns access shape and returned values, not a promised percentage reduction in physical I/O.

Pay for coverage deliberately

Appending columns to a secondary index consumes storage and buffer-pool capacity and increases maintenance when those indexed values change. Long primary keys enlarge secondary entries too. Cover the important narrow query rather than automatically copying a large payload into every index. Keep projected columns explicit so later application changes make their access requirements visible in review.

Compare representative plans, examined rows, foreground latency, and write cost before retaining a new index. Test with realistic selectivity and concurrent activity; a quiet fixture does not measure MVCC fallback frequency or replica maintenance cost. An index should earn its place through the workload it serves, not through the appearance of a desirable label in one explanation.

Sources

MySQL 8.4 clustered and secondary indexes, primary-key columns carried by secondary records, checked 2026-09-12.
MySQL 8.4 EXPLAIN output, index-use and covering-plan indicators, checked 2026-09-12.
MySQL 8.4 InnoDB multiversioning, clustered lookups required for secondary-index visibility checks, checked 2026-09-12.

DNS TTL trades repeated upstream queries against retained answers

Wed, 01 Mar 2028 00:00:00 +0000

A DNS TTL is a lifetime attached to cached data, not a refresh job scheduled for every second. A cache can reuse a valid answer while it retains that entry. After expiry, a subsequent request usually requires fresh resolution unless another configured behavior applies. An idle record need not produce an upstream query merely because its TTL has reached zero.

The article’s controlled example uses three lookups 1.2 seconds apart: TTL 1 produces three upstream queries and TTL 20 one. A downstream cache receives remaining lifetime, not a reset TTL. These diagrams reproduce source-described behavior; no resolver experiment was run for this visual.

When it breaks

Shortening TTL increases potential upstream work when clients keep asking for the same data. It does not multiply all authoritative traffic by one universal factor: query frequency, resolver sharing, cache capacity, prefetching, and record diversity affect the result. A short TTL on one popular name behaves differently from a large stream of names that are each queried only once.

A long TTL permits continued use of an answer after the authority changes its data. That cached answer may still be valid under the lifetime originally received even though it no longer matches the newest zone content. Calling every mismatch an expired or broken cache obscures the actual contract. TTL expresses permitted retention; it does not promise continuous synchronization with the authority.

Multiple forwarding layers do not ordinarily receive a fresh copy of the original TTL at every hop. A cache returns remaining lifetime, which the next cache then ages. Adding full TTLs per layer invents a propagation delay that ordinary compliant forwarding does not create. Local TTL overrides and serve-stale policies are separate behaviors that should be identified explicitly when present.

Check it quickly

Save this as check-ttl-cost.py and run python3 check-ttl-cost.py with Docker. The pinned container uses BIND 9.18.49 and dnsmasq 2.90, with no external runtime network. The authoritative server logs actual queries. One forwarder initially serves two names, and the script later starts a second forwarder behind the first to inspect the TTL handed downstream:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
with lab('short 1 IN A 127.0.0.2\nlong 20 IN A 127.0.0.3\nchain 6 IN A 127.0.0.4') as (_,update,cache,queries):
 cache();samples=[]
 for i in range(3):
 pair={name:show(query(name+'.lab.test.',port=1054)) for name in ('short','long')};samples.append(pair)
 if i<2:time.sleep(1.2)
 wait(lambda: queries('short.lab.test')==3 and queries('long.lab.test')==1)
 first=query('chain.lab.test.',port=1054);assert first.answer[0].ttl==6
 wait(lambda: queries('chain.lab.test')==1)
 time.sleep(2.2);cache(port=1055,upstream=1054)
 downstream=query('chain.lab.test.',port=1055);upstream=query('chain.lab.test.',port=1054)
 assert values(downstream)==['127.0.0.4'] and 0<downstream.answer[0].ttl<6
 assert abs(downstream.answer[0].ttl-upstream.answer[0].ttl)<=1
 assert queries('chain.lab.test')==1
 update('short 1 IN A 127.0.0.2\nlong 20 IN A 127.0.0.3\nchain 6 IN A 127.0.0.5')
 wait(lambda: values(query('chain.lab.test.',port=1055))==['127.0.0.5'],timeout=8)
 wait(lambda: queries('chain.lab.test')==2)
 print(json.dumps({'three_lookups_each':samples,'authority_query_counts':{'short':queries('short.lab.test'),'long':queries('long.lab.test')},
 'cache_chain':{'original':show(first),'downstream_first':show(downstream),'upstream_remaining':show(upstream),'authority_queries_after_expiry':queries('chain.lab.test')}},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The short.lab.test record has TTL 1 and long.lab.test has TTL 20. The program queries each three times, waiting 1.2 seconds between rounds. The authoritative log records three queries for the short-lived record and one for the long-lived record. Both produce successful answers on every round. The difference is upstream work under this controlled request schedule, not a measured application-latency improvement.

The returned TTL for the long-lived record decreases across rounds, while each refreshed short-lived answer starts at 1 again. The script checks the server log rather than assuming a fast response means a cache hit. On loopback, even an uncached query can be quick, so response time alone would not establish whether the authority was contacted or the answer came from memory.

For the layered case, the first forwarder caches chain.lab.test with TTL 6. After 2.2 seconds, a newly started second forwarder queries through it. The recorded downstream answer has TTL 3, matching the upstream remaining value within integer-second timing. It does not regain TTL 6. The authoritative count remains one, confirming that this downstream fill reused the first forwarder’s cached answer.

The authority then changes the chain record’s address. Queries through the second forwarder eventually receive the new address, and the authoritative count rises to two. This checks successful refresh through both layers after the retained data expires. The exact expiry fraction is implementation-dependent; the asserted properties are preserved remaining lifetime, no premature extra authoritative query, and eventual acquisition of the changed value.

Choose TTL from observed demand and change needs

Use expected change frequency and acceptable old-answer overlap alongside query volume. A busy shared resolver can amortize one answer across many clients, whereas isolated caches duplicate fills. Eviction can make an entry disappear before its TTL, so a long configured lifetime does not guarantee a hit. Measure hits, misses, upstream requests, and available capacity instead of treating TTL as a cache-size control.

For a planned move, lower TTL early enough for earlier longer-lived copies to expire. Keep the destination overlap aligned with the actual retained data. Raising TTL after stability reduces repeated resolution for busy records, but it also extends the next change’s ordinary waiting window. The relevant unit is the RRset: records of the same name, class, and type should have a consistent TTL.

Finally, compare answer data and remaining lifetime at each resolver actually used by the application. A direct authoritative query and a cached recursive answer reveal different states. Application caches and existing connections may add their own behavior outside this DNS experiment. Separate those lifetimes before estimating how quickly users will discover a changed endpoint or how much authoritative capacity a TTL adjustment requires.

Sources

RFC 1035 domain names, TTL as a cache lifetime and treatment of cached records, checked 2026-09-12.
RFC 2181 DNS clarifications, TTL limits and consistent lifetime within an RRset, checked 2026-09-12.
dnsmasq manual, cache sizing, forwarding, and TTL options, checked 2026-09-12.

Bash history is your searchable command log

Mon, 28 Feb 2028 00:00:00 +0000

A useful command history saves retyping, but it is not a complete record of everything executed. Bash normally enables history in interactive shells, and settings can exclude entries or limit retention. The searchable list in a running shell and the history file on disk are separate states. That distinction explains why a command may be visible in one terminal, absent in another, and lost when an unwritten session ends unexpectedly.

History is local to each running shell until explicit file operations connect those states. Appending does not refresh another shell; importing the whole file can duplicate entries, while replacement can discard other sessions’ records. Shared history is not a reliable global execution log.

Session memory differs from the file

HISTSIZE limits in-memory history entries; HISTFILESIZE controls file retention using a line-based target. The current Bash manual allows complete multiline entries to exceed that target rather than retaining partial entries. Multiline commands and timestamp records also mean raw line counts are not command counts. HISTFILE selects the file, normally the user’s Bash history file. An unset or empty value prevents ordinary saving, and an unwritable location can leave apparently healthy in-memory recall with no persistent record.

Bash records command text before parameter expansion but after history expansion, subject to HISTCONTROL and HISTIGNORE. A saved line containing a variable therefore does not necessarily reveal the value used during the original execution. Current directories, aliases, functions, environment values, and external files can all change before a command is recalled. History preserves useful input, not a reproducible execution environment or a trustworthy audit trail.

Appending does not refresh other terminals

On normal exit, Bash saves history; histappend selects appending rather than overwriting the file. Without earlier writes, abnormal termination can lose all unsaved entries, not merely the final command. history -a appends newly entered lines that this session has not already appended. Saving before exit reduces that loss window, but does not promise durable storage through every system failure.

Another running shell has its own list. It needs an import such as history -n to read newly appended lines; enabling histappend alone does not make another terminal’s searches update live. history -r reads the entire file into the existing list and can introduce duplicates. history -w replaces the file with the current list and can discard other sessions’ records. These operations should not be treated as interchangeable synchronization commands.

If append and import operations are integrated into PROMPT_COMMAND, preserve existing prompt hooks and any use of the preceding command’s status. Replacing a configured hook wholesale can break other shell behavior. Concurrent sessions also do not create a reliable global execution order merely by sharing a file. Keep separate operational logs when ordering, exit status, or accountability matters.

Search, inspect, then execute

In the usual Emacs-style Readline bindings, Ctrl-R searches backward incrementally, and pressing it again finds older matches. A movement key leaves the search for editing; Ctrl-G cancels it. With default isearch-terminators, Ctrl-J ends incremental search and leaves the match on the editing line without executing it. Enter accepts the line. Outside that search state, Ctrl-J normally accepts a line too, so check the current mode and bindings before relying on it as a preview key.

Inspect the hostname, directory, options, and intended target whenever reusing an old command. A command that was correct last month may address a different environment today. Prefer a distinctive search fragment and review the whole line, rather than trusting that the first matching command is the right operation.

When it breaks

Missing commands can result from filtering, retention limits, a different history file, failed saving, or an unrefreshed session. Check those causes separately. ignoredups excludes consecutive duplicates, not every duplicate in the entire list. ignoreboth combines that behavior with ignorespace; it does not produce a globally unique command archive.

A leading space excludes an entry only when the relevant setting is active. It does not remove terminal recordings, process arguments, or logs written elsewhere, so do not put credentials directly into command lines and rely on history filtering for secrecy. Diagnostic examples should use synthetic entries and a temporary history file, rather than printing someone’s real recent commands.

Check it quickly

Run this with Python 3, Bash, and standard Unix utilities. The environment points Bash to a temporary file before it starts. Only synthetic entries are inspected. The brace group avoids recording the demonstration’s own control commands after history is enabled.

import os
import subprocess
import tempfile
from pathlib import Path

script = r'''{
set -eu
set -o history
history -c
history -s 'demo original'
history -w "$HISTFILE"
printf 'first-file:\n'
cat "$HISTFILE"
history -c
history -s 'demo replacement'
history -w "$HISTFILE"
printf 'replacement-file:\n'
cat "$HISTFILE"
history -r "$HISTFILE"
printf 'memory-after-reread:\n'
history | sed 's/^[[:space:]]*[0-9][0-9]*[[:space:]]*//'
unset HISTFILE
}'''
with tempfile.TemporaryDirectory(prefix="bash-history-demo-") as directory:
 history_path = Path(directory) / "history"
 env = dict(os.environ, BASH_ENV="", HISTFILE=str(history_path), HISTSIZE="100",
 HISTFILESIZE="100", HISTCONTROL="", HISTIGNORE="",
 HISTTIMEFORMAT="")
 subprocess.run(["bash", "--noprofile", "--norc", "-c", script],
 env=env, check=True)

The first file contains demo original; after replacement it contains only demo replacement. Reading that file into a list already containing the replacement produces two identical entries in memory. This verifies overwrite and import semantics without executing the stored strings. It does not claim to test keyboard bindings or crash durability, which require their own observations.

A 2026-10-19 rerun passed this exact temporary-file exercise using Python 3.12.14 and Bash 5.2.37 on Linux 6.18.55. It did not exercise interactive keys or inspect user history.

Sources

Bash history facilities, session lists and saving to the history file, checked 2026-09-07.
Bash history builtins, append and replacement versus importing entries, checked 2026-09-07.
Readline history commands, search and command-acceptance bindings, checked 2026-09-07.
Bash reference manual, HISTFILESIZE whole-entry retention and section 8.2.5 search-state termination, checked 2026-10-19.

Protobuf evolution preserves meaning only within a defined contract

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC rollout can leave old producers, new consumers, and intermediate services running together. Protobuf helps them exchange binary messages by identifying fields with numbers and wire types. That tolerance is narrower than application compatibility: a message can parse successfully while a consumer misunderstands its values, ignores a new instruction, or loses the distinction between absent and explicitly zero.

Separate numbers, meaning, and bytes

A binary tag combines the field number with its wire type; the field name is not transmitted there. Keeping the number and type while renaming a field can preserve binary decoding. It still changes generated accessors and may change JSON or text interfaces. A gRPC method name or service name belongs to another contract and is not protected by a field-number rule.

Never reuse a retired number for another meaning. Different declared types can share a wire type: signed and unsigned integers both use varints, yet a negative signed value can become a large unsigned value. A successful parse is therefore insufficient validation. Reserve retired numbers and names in the schema so later declarations cannot accidentally reclaim them; reserved numbers are unavailable, not placeholders to activate later.

Unknown fields can survive a parse-and-serialize binary relay. However, copying only known fields or converting through JSON can discard them. Preservation of decoded information also does not promise byte-identical serialization. Field order and encoding choices can differ, including across implementations. Compare required values and presence in compatibility tests; do not use ordinary reserialization as a canonical digest algorithm.

Presence carries a separate bit of meaning

In proto3, singular scalar fields without optional generally use implicit presence. Their default value is indistinguishable from absence through the normal API. Explicit optional scalars track whether a value was supplied, including zero. Message fields and oneof members already have presence; repeated and map fields do not distinguish an empty collection from an absent one.

Changing a scalar between implicit and explicit presence can remain binary-readable while changing round-trip semantics. An older reader that knows the field with implicit presence can parse explicit zero and then omit it when forwarding. Adding optional at one endpoint cannot force every intermediary to preserve that distinction. Test the entire path that actually modifies or forwards the message.

When it breaks

A new consumer may require a field that old writers never supplied. Although proto3 has no required declaration, application validation can create exactly this rolling-upgrade failure. Introduce an agreed fallback or sequence the rollout before enforcing the requirement. Optional syntax alone does not define a sensible default, authorize an operation, or make an old service implement a new feature.

Also test persisted messages, delayed jobs, and rollback versions. Their useful lifetime can exceed the deployment window. A schema check against only the immediately previous release misses older records that can still reappear. Keep retired numbers reserved even when current traffic no longer uses them.

Both relayed results return zero, but the unknown-field relay retains explicit presence while the known implicit-presence relay loses it. The historical Python fixture passed within its local scope; it did not test reserved declarations, real gRPC calls or cross-language clients. No example was rerun for this drawing.

Check it quickly

Run the complete block with Python and Google’s protobuf package 7.36.2 available. It was executed with Python 3.12.14. Separate descriptor pools create real old and new message classes without a compiler, filesystem fixture, gRPC server, or network connection. This checks the Python binary runtime, not cross-language interoperability or generated-client compatibility.

from google.protobuf import descriptor_pb2 as schema, descriptor_pool, message_factory


def message(fields, explicit=()):
 file = schema.FileDescriptorProto(name='probe.proto', package='probe', syntax='proto3')
 record = file.message_type.add(name='Record')
 for name, number, kind in fields:
 field = record.field.add(name=name, number=number, type=kind, label=1)
 if name in explicit:
 field.oneof_index = len(record.oneof_decl)
 record.oneof_decl.add(name='_' + name)
 field.proto3_optional = True
 pool = descriptor_pool.DescriptorPool()
 pool.Add(file)
 return message_factory.GetMessageClass(pool.FindMessageTypeByName('probe.Record'))


INT32 = schema.FieldDescriptorProto.TYPE_INT32
UINT32 = schema.FieldDescriptorProto.TYPE_UINT32
STRING = schema.FieldDescriptorProto.TYPE_STRING
Old = message([('id', 1, INT32), ('name', 2, STRING)])
New = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)], ('count',))
Implicit = message([('event_id', 1, INT32), ('name', 2, STRING), ('count', 3, INT32)])

old = Old(id=7, name='alpha')
new_reader = New.FromString(old.SerializeToString())
assert new_reader.event_id == 7 and not new_reader.HasField('count')
# The added explicit zero is unknown to Old but retained in binary forwarding.
new_writer = New(event_id=7, name='alpha', count=0)
relay = Old.FromString(new_writer.SerializeToString())
returned = New.FromString(relay.SerializeToString())
assert returned.event_id == 7 and returned.HasField('count') and returned.count == 0
print('old/new binary values survive; unknown explicit zero survives the old relay')

# An implicit-presence reader knows field 3, so zero is omitted on serialization.
implicit_relay = Implicit.FromString(new_writer.SerializeToString())
lost_presence = New.FromString(implicit_relay.SerializeToString())
assert lost_presence.count == 0 and not lost_presence.HasField('count')
print('an implicit-presence relay preserves zero but loses its explicit presence')

Signed = message([('value', 1, INT32)])
Unsigned = message([('value', 1, UINT32)])
assert Unsigned.FromString(Signed(value=-1).SerializeToString()).value == 4294967295
# Distinct field order is legal; compare decoded meaning, not byte equality.
reordered = b'\x18\x00' + old.SerializeToString()
assert New.FromString(reordered) == new_writer
print('reusing an integer tag changes meaning; legal field orders decode equivalently')

The first output confirms that an old writer remains readable and that an unknown explicit zero survives the old binary relay. The next output exposes a different result when the relay knows that field with implicit presence: its value remains zero, but HasField becomes false after forwarding. These are different compatibility questions, not contradictory observations.

The final assertions decode negative one as 4294967295 under an incompatible unsigned interpretation and accept a legal alternative field order. They demonstrate why wire readability does not establish semantic equivalence or canonical bytes. The descriptor exercise does not compile reserved declarations; compiler checks and real gRPC request/response behavior remain separate rollout tests.

Sources

Protobuf: encoding, tags, varints and serialization ordering, checked 2026-10-15.
Protobuf: proto3 language guide, binary changes, reserved fields and unknown-field retention, checked 2026-10-15.
Protobuf: field presence, implicit and explicit presence across relays, checked 2026-10-15.
Protobuf: dos and don’ts, retired numbers and compatibility discipline, checked 2026-10-15.

Kubernetes CronJobs need a history policy

Fri, 25 Feb 2028 00:00:00 +0000

A CronJob creates Jobs on a schedule, and each Job manages the Pods that perform the work. Finished objects can preserve useful status, but retention is bounded by several independent mechanisms. In particular, CronJobs already have default history limits. Omitting explicit history settings does not make ordinary completed runs accumulate forever.

Count and TTL do not guarantee minimum retention when the other permits earlier deletion. Cleanup is asynchronous, and active-job limits do not prove a business operation is safe to retry. The retained diagnostics were not rerun.

Separate history from log retention

The default CronJob history keeps up to three successful Jobs and one failed Job. The controller reconciles those categories independently. A zero limit requests no retained Jobs in that category, but cleanup is asynchronous rather than an exact completion-time deletion. Active Jobs are not completed-history entries and need separate attention.

Thirty days of hourly scheduling means 720 scheduled opportunities, not necessarily 720 retained Jobs or exactly 720 Pods. Missed schedules, concurrency policy, retries and parallelism change those relationships. A Job can own several Pods. State the assumptions before using run counts as storage or API-capacity estimates.

Keeping a Pod object also does not guarantee its logs remain available. Node failure, rotation, container cleanup and eviction affect log access. Export the operational evidence required by your retention policy to a suitable logging system. Kubernetes objects are useful investigation records, but they are not an archival guarantee.

Large retained histories still have costs: object storage, list sizes and investigation noise. A watch receives relevant changes rather than repeatedly emitting every unchanged object merely because it exists. Measure actual object counts and control-plane behavior instead of describing retained status as continuous activity.

Count and age are independent policies

The CronJob history limits constrain finished Job counts, not their minimum age. A busy schedule can replace the last failed record quickly, while a quiet schedule may keep it much longer. Choose counts from the inspection interval and failure frequency, rather than treating one retained failure as enough evidence for every workload.

TTL-after-finished is configured on the Job. For future CronJob-created Jobs, place ttlSecondsAfterFinished inside the Job template’s spec. The timer starts when a Job reaches Complete or Failed; expiry makes it eligible for cascading deletion. A zero TTL makes it eligible immediately, while an unset TTL supplies no age-based cleanup.

History cleanup and TTL can each remove a Job. Consequently, neither a large history count nor a long TTL guarantees minimum retention when the other mechanism permits earlier deletion. Finalizers and controller progress can delay physical removal. Editing a CronJob template affects future Jobs; it does not rewrite the policy of every existing Job.

Ownership determines cascading behavior

CronJob-created Jobs reference their owner, and Jobs reference their Pods through dependent ownership. Normal cascading deletion can remove that hierarchy. Explicit orphan deletion instead preserves dependents and removes the relevant ownership link; it does not intentionally leave an ordinary dangling reference for the collector to follow.

After orphaning Jobs, the former CronJob no longer trims their history. A configured TTL can still apply to each finished Job. Directly created Jobs also need an explicit retention decision. Check the actual deletion propagation policy and ownership rather than assuming every client uses identical defaults or that every orphan will soon disappear.

When it breaks

A Job that never reaches a terminal condition does not become eligible through finished-job TTL. History limits do not bound active runs either. An appropriate activeDeadlineSeconds can bound a Job’s runtime, while concurrencyPolicy controls overlapping schedules. These settings address different failure modes and do not prove the underlying operation is safe to retry.

Zero history can remove the failure record before anyone investigates it. Preserve required logs and result artifacts elsewhere before shortening retention. Conversely, a missing owner relationship or an absent TTL deserves investigation, not automatic deletion: another controller or retention requirement may deliberately own the cleanup decision.

Check it quickly

The following retained read-only diagnostics were not rerun in this review. Select a disposable test context and namespace. Compare the stored CronJob with the Jobs it actually owns, their completion conditions and creation times; namespace-wide totals below include unrelated workloads and active Jobs.

kubectl get cronjob hourly-rollup -o jsonpath='{.spec.successfulJobsHistoryLimit} {.spec.failedJobsHistoryLimit} {.spec.jobTemplate.spec.ttlSecondsAfterFinished}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp --no-headers | wc -l
kubectl get pods --field-selector=status.phase=Succeeded --no-headers | wc -l
kubectl get jobs -o jsonpath='{range .items[*]}{.metadata.name} ttl={.spec.ttlSecondsAfterFinished} owner={.metadata.ownerReferences[0].name}{"\n"}{end}'

The first line usually shows API-defaulted history values even when the manifest omitted them. Blank output is not proof of unlimited retention; check lookup errors and the complete object. The last line displays only the first owner reference. An absent TTL and owner mean these two particular mechanisms provide no demonstrated cleanup, not that deletion is impossible. Repeat inspection across completed runs, expecting bounded owned histories after reconciliation while active runs and unrelated objects are assessed separately.

Sources

Kubernetes Job, “Clean up finished jobs automatically” and the TTL mechanism for finished Jobs, checked 2026-10-15.
Kubernetes CronJob, default history limits and the job template, checked 2026-10-15.
Kubernetes garbage collection, owner references and cascading deletion, checked 2026-10-15.
Primary reference, TTL eligibility and finalizers, checked 2026-10-15.
Primary reference, log rotation and node-level retention, checked 2026-10-15.

A Redis hash stores many fields in one key

Thu, 24 Feb 2028 00:00:00 +0000

A Redis hash is a field-addressable record under one key, not a schema-checked object or an automatically nested document. It is useful when callers need a few attributes without replacing the whole record. That advantage depends on access patterns, field sizes, and the meaning of replies, not merely on choosing the hash type.

Read the fields your caller actually needs

HGET retrieves one field. HMGET retrieves selected fields in the requested order, while HGETALL returns the entire mapping. With redis-py’s binary responses, an absent field becomes None and an existing empty value becomes empty bytes. A truthiness check incorrectly treats those two states as equivalent. An absent key also produces missing values through HMGET.

Field names and values are bytes; a name containing a dot is still a literal name. Redis does not interpret name.first as traversal through a nested object. Likewise, a string containing JSON remains a Redis string: HGET does not parse it. Decide serialization and validation in the application and preserve the distinction between an omitted attribute and an intentionally empty one.

Updates have precise return values

HSET changes the specified fields and leaves others untouched. A multi-field call is one command, but its integer reply counts newly added fields, not successful replacements. Updating an existing name can correctly return zero. Treating that zero as failure can cause needless retries or misleading error reports.

HINCRBY changes an integer field in one command and starts from zero when the field is absent. It rejects noninteger values and overflow beyond signed 64-bit range. This avoids a read-modify-write race for that increment; it does not make surrounding application checks atomic. Separate HGET, local arithmetic, and HSET can still overwrite another client’s update.

HDEL counts fields actually removed, and deleting the final field removes the key. Do not use an empty hash as a persistent placeholder for an entity. Whole-key expiration also removes its fields. Current Redis additionally supports field deadlines; the exercise below assigns no expirations and does not test lifetime behavior.

Compact representation is an implementation choice

Small hashes can use listpack encoding instead of a hash table. Redis 8.10.1’s standard thresholds are 512 entries and 64 bytes per field name or value; deployed settings can differ. Crossing an applicable limit changes the representation, not the public hash operations or the stored values.

Compact storage trades internal scanning work for reduced overhead and locality. Raising thresholds substantially requires measurement: a command’s advertised complexity is not a promise that arbitrarily large fields or changed configuration have no cost. Inspect OBJECT ENCODING when diagnosing representation, but do not make application correctness depend on that internal label or an assumed memory-saving ratio.

When it breaks

HGETALL grows with the whole hash and transfers both names and values. Fetching thousands of fields when the caller needs two wastes work, even if they share one key. HMGET’s work grows with the requested field count; it is not free either. Never use returned map iteration order as an application ordering contract.

WRONGTYPE indicates that the key holds another Redis type, not that a field is missing. Noninteger counters require an explicit data decision, not a retry loop. One huge hash also remains one key rather than automatically spreading its fields across cluster nodes. Choose record boundaries around actual access and ownership rather than packing unrelated data together.

HMGET preserves request order and missing-versus-empty values. HSET counts added fields. The 4-field/8-byte limits are private test settings, distinct from Redis 8.10.1 standard 512/64 limits.

Check it quickly

The original exercise reports macOS, Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0; this review did not rerun it. Use POSIX, a C compiler, make, curl, tar, and Python 3.10+. Setup builds Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a private Unix-socket server with TCP and persistence disabled. Only this server receives deliberately small thresholds: four entries and eight bytes. The test closes its client, terminates its own process, and removes the temporary runtime directory; it never connects to a default local Redis.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-hash-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--hash-max-listpack-entries", "4", "--hash-max-listpack-value", "8",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 assert client.hset("profile", mapping={"name": "Ada", "visits": "0", "empty": b""}) == 3
 assert client.hset("profile", "name", "Ada L") == 0
 assert client.hmget("profile", ["visits", "name", "missing", "empty"]) == [
 b"0", b"Ada L", None, b"",
 ]
 assert client.hmget("absent", ["a", "b"]) == [None, None]
 assert client.hgetall("profile") == {b"name": b"Ada L", b"visits": b"0", b"empty": b""}
 assert client.hincrby("profile", "visits", 1) == 1
 assert client.hget("profile", "name") == b"Ada L"
 print("HSET counts new fields; HMGET preserves order and distinguishes missing from empty")

 client.set("json", '{"name":"Ada"}')
 try:
 client.hget("json", "name")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a JSON string is not a hash")
 client.hset("literal", "name.first", b"Ada")
 assert client.hget("literal", "name") is None
 assert client.hget("literal", "name.first") == b"Ada"
 client.hset("invalid", "counter", "3.5")
 try:
 client.hincrby("invalid", "counter", 1)
 except redis.ResponseError as error:
 assert "integer" in str(error)
 else:
 raise AssertionError("an integer counter cannot contain 3.5")
 assert client.hget("invalid", "counter") == b"3.5"
 client.hset("overflow", "counter", 2**63 - 1)
 try:
 client.hincrby("overflow", "counter", 1)
 except redis.ResponseError as error:
 assert "overflow" in str(error)
 else:
 raise AssertionError("signed 64-bit integer overflow must fail")
 assert client.hget("overflow", "counter") == str(2**63 - 1).encode()
 print("wrong types and invalid or overflowing counters fail; dotted fields are literal")

 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fourth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fifth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"hashtable"
 client.hset("value-limit", "field", b"abcdefgh")
 assert client.object("ENCODING", "value-limit") == b"listpack"
 client.hset("value-limit", "field", b"abcdefghi")
 assert client.object("ENCODING", "value-limit") == b"hashtable"
 assert client.hget("profile", "visits") == b"1"
 assert client.hget("value-limit", "field") == b"abcdefghi"
 print("private 4-field and 8-byte thresholds each trigger listpack to hashtable conversion")

 assert client.hdel("value-limit", "absent") == 0
 assert client.hdel("value-limit", "field") == 1
 assert client.exists("value-limit") == 0
 print("deleting the last field removes the hash key")

The assertions distinguish added fields from replacements, preserve HMGET ordering and missing-versus-empty values, and verify unchanged neighboring fields. A JSON string raises WRONGTYPE, a dotted field stays literal, and invalid or overflowing counters remain unchanged after rejection.

At four short fields the profile remains listpack; the fifth switches it to hashtable. A separate hash switches when its value grows from eight to nine bytes. Reads still return the expected values, and deleting the final field removes that key. These results demonstrate configured encoding transitions and command semantics, not production memory savings, latency, concurrent-load behavior, or cluster distribution.

Sources

Redis hashes, field-oriented operations and current expiration capabilities, checked 2026-09-08.
Redis HSET, partial updates and added-field counts, checked 2026-09-08.
Redis HMGET, requested ordering and missing-field replies, checked 2026-09-08.
Redis HGETALL, full-hash reads and complexity, checked 2026-09-08.
Redis HINCRBY, integer increments and supported range, checked 2026-09-08.
Redis HDEL, removal counts and final-field deletion, checked 2026-09-08.
Redis OBJECT ENCODING, internal representations and conversion, checked 2026-09-08.
Redis memory optimization, compact encoding thresholds and measurement guidance, checked 2026-09-08.
Redis 8.10.1 configuration, default hash thresholds, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Grafana variables need the query language to agree

Wed, 23 Feb 2028 00:00:00 +0000

Grafana dashboard variables let one dashboard select an environment, service, or instance without duplicating every panel. A query variable obtains options from a data source; a custom variable can hold a deliberately fixed list. Neither choice is universally better. Use query-backed options when discovery should follow the source, and a maintained static list when the allowed choices are intentionally stable and small.

A variable is substituted into a query before the data source executes it. That makes its formatting part of query correctness. For Prometheus multi-value selections, use a regular-expression label matcher such as environment=~"$environment". Grafana’s Prometheus interpolation handles selected values as regular-expression alternatives with the required escaping. An exact matcher treats the resulting combined text as one label value and therefore does not implement the intended multi-selection. The variable editor’s label_values helper discovers options; it is not a PromQL function to paste into a panel expression.

Check the resulting expression

There are two relevant syntactic layers: the regular expression and the PromQL string containing it. A literal dot in production.eu must not become a wildcard that also matches productionXeu. Let the data source’s interpolation perform its documented formatting, and inspect the expression actually sent when using an explicit formatter. The raw formatter bypasses that data-source formatting; it is not a universal fix for difficult variable values.

The All option also has semantics. With no custom All value, this example expands the available, filtered options. Its menu includes production.eu and staging, so All covers those two environments. A custom expression such as .* would instead match whatever values the backend regex admits. Decide whether “all” means the menu’s current choices or a broader query, and test that decision with deliberately excluded data.

When it breaks

Query variables are not guaranteed to stay fresh forever. Their refresh setting determines when Grafana updates the option query, and time-dependent options may need refresh on time-range changes. Chained variables can still produce an empty downstream selection when the data contains no matching combination. Distinguish a legitimately empty result from a stale selection or a broken query before presenting “No data” as a service outage.

Variable filters are not authorization controls. A menu that hides an environment does not remove that environment from the data source or stop another permitted query from requesting it. Enforce access at the appropriate Grafana and backend boundaries. Shared dashboard URLs also carry selections rather than permissions; multiple values use repeated var-environment parameters, not an invented comma-separated permission list.

Grafana query formatting: mechanism and evidence boundaries.

Check it quickly

Save this as grafana-variables.py. With Python, a local Docker daemon, and both images preloaded, run python3 grafana-variables.py --serve. Disposable Grafana 12.1.0 and Prometheus 3.5.1 containers share a dedicated bridge; only Grafana is published on loopback, without host mounts. Open the printed URL. Serve mode lasts 120 seconds; --serve-seconds accepts 1–300. Control-C enters cleanup and remains an interruption, not normal completion. Docker calls have 30-second timeouts; readiness retries stop after a 20-second deadline plus the in-flight request. Cleanup checks exact owned names and fails on removal, inventory, or command-timeout or in-flight-interruption uncertainty. Without --serve, it checks only handwritten backend queries. Python disables proxy discovery and redirects; browser proxy policy remains a separate prerequisite.

import argparse
import json
import secrets
import subprocess
import time
import uuid
from http.client import HTTPException
from threading import Event
from urllib.error import HTTPError, URLError
from urllib.parse import quote
from urllib.request import HTTPRedirectHandler, ProxyHandler, build_opener

parser = argparse.ArgumentParser()
parser.add_argument("--serve", action="store_true")
parser.add_argument("--serve-seconds", type=int, default=120)
args = parser.parse_args()
if not 1 <= args.serve_seconds <= 300:
 parser.error("--serve-seconds must be between 1 and 300")

def require(condition, message):
 if not condition:
 raise AssertionError(message)

uncertain_commands = []

prefix = "xbcatm-variable-" + uuid.uuid4().hex[:10]
network, prom, grafana = [prefix + suffix for suffix in ("-net", "-prom", "-grafana")]
print(json.dumps({"owned_containers": [grafana, prom], "owned_network": network}), flush=True)


def command(*args):
 try:
 result = subprocess.run(["docker", *args], capture_output=True, text=True, timeout=30)
 except subprocess.TimeoutExpired as error:
 uncertain_commands.append(args)
 raise RuntimeError("Docker " + args[0] + " timed out") from error
 except KeyboardInterrupt:
 uncertain_commands.append(args)
 raise
 if result.returncode:
 raise RuntimeError(result.stderr.strip())
 return result.stdout.strip()


class NoRedirect(HTTPRedirectHandler):
 def redirect_request(self, req, fp, code, msg, headers, newurl):
 raise RuntimeError("Unexpected redirect in local fixture")


opener = build_opener(ProxyHandler({}), NoRedirect())


def get(path):
 with opener.open(origin + path, timeout=5) as response:
 require(response.status == 200, "Unexpected HTTP status")
 return json.load(response)


def eventually(check):
 deadline = time.monotonic() + 20
 while time.monotonic() < deadline:
 try:
 value = check()
 if value:
 return value
 except (HTTPError, URLError, TimeoutError, ConnectionError, HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Local lab did not become ready")


def cleanup_owned(groups):
 # Inspect exact owned names; absence is not inferred from a failed inspect.
 errors = []
 for kind, names, format_string in groups:
 try:
 listing = command(kind, "ls", *( ["--all"] if kind == "container" else [] ),
 "--format", format_string).splitlines()
 present = sorted(set(names) & set(listing))
 if present:
 command(kind, "rm", *( ["--force"] if kind == "container" else [] ), *present)
 except Exception as error:
 errors.append(f"{kind} removal: {error}")
 try:
 remaining = command(kind, "ls", *( ["--all"] if kind == "container" else [] ),
 "--format", format_string).splitlines()
 if set(names) & set(remaining):
 errors.append(f"{kind} remains: {sorted(set(names) & set(remaining))}")
 except Exception as error:
 errors.append(f"{kind} inventory unverified: {error}")
 if uncertain_commands:
 errors.append("A Docker command timed out or was interrupted; daemon-side completion remains uncertain")
 require(not errors, "Cleanup not verified: " + "; ".join(errors))
 print(json.dumps({"cleanup": "owned names absent in final inventories"}), flush=True)


source = {"type": "prometheus", "uid": "lab_prom"}
dashboard = {
 "uid": "variable-lab", "title": "Variable escaping lab", "schemaVersion": 41,
 "version": 1, "timezone": "utc", "refresh": "", "time": {"from": "now-5m", "to": "now"},
 "templating": {"list": [{
 "name": "environment", "label": "Environment", "type": "query", "datasource": source,
 "query": {"query": "label_values(lab_active_workers, environment)", "refId": "VariableQuery"},
 "definition": "label_values(lab_active_workers, environment)",
 "regex": "/^(production[.]eu|staging)$/", "refresh": 1, "sort": 1,
 "multi": True, "includeAll": True, "allValue": "", "options": [],
 "current": {"text": ["production.eu"], "value": ["production.eu"], "selected": True},
 }]},
 "panels": [{
 "id": number, "title": title, "type": "stat", "datasource": source,
 "gridPos": {"h": 7, "w": 12, "x": (number - 1) * 12, "y": 0},
 "targets": [{"refId": "A", "expr": expression, "instant": True, "range": False}],
 "options": {"reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": False}},
 "fieldConfig": {"defaults": {"decimals": 0}, "overrides": []},
 } for number, title, expression in (
 (1, "Selected workers", 'sum(lab_active_workers{environment=~"$environment"})'),
 (2, "Unescaped control", 'sum(lab_active_workers{environment=~"${environment:raw}"})'),
 )],
}
prom_script = """set -eu
mkdir -p /tmp/www
cat > /tmp/www/metrics.txt <<'METRICS'
# HELP lab_active_workers Synthetic active worker count.
# TYPE lab_active_workers gauge
lab_active_workers{environment="production.eu"} 10
lab_active_workers{environment="staging"} 2
lab_active_workers{environment="productionXeu"} 100
METRICS
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
scrape_configs:
 - job_name: lab
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']
CONFIG
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data
"""
grafana_script = """set -eu
mkdir -p /tmp/data /tmp/provisioning/datasources /tmp/provisioning/dashboards /tmp/dashboards
mkdir -p /tmp/provisioning/plugins /tmp/provisioning/alerting
cat > /tmp/provisioning/datasources/lab.yml <<'DATASOURCE'
apiVersion: 1
datasources:
 - name: Lab Prometheus
 type: prometheus
 uid: lab_prom
 access: proxy
 url: http://metrics:9090
 isDefault: true
 jsonData:
 timeInterval: 1s
DATASOURCE
cat > /tmp/provisioning/dashboards/lab.yml <<'PROVIDER'
apiVersion: 1
providers:
 - name: lab
 type: file
 options:
 path: /tmp/dashboards
PROVIDER
cat > /tmp/dashboards/lab.json <<'DASHBOARD'
""" + json.dumps(dashboard) + """
DASHBOARD
exec /run.sh
"""
try:
 # Use preloaded images; this is a local disposable fixture, not a download step.
 for image in ("prom/prometheus:v3.5.1", "grafana/grafana:12.1.0"):
 command("image", "inspect", image)
 command("network", "create", network)
 command("run", "--pull=never", "-d", "--rm", "--name", prom, "--network", network, "--network-alias", "metrics",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", prom_script)
 command("run", "--pull=never", "-d", "--rm", "--name", grafana, "--network", network,
 "-p", "127.0.0.1::3000", "-e", "GF_PATHS_DATA=/tmp/data",
 "-e", "GF_PATHS_PROVISIONING=/tmp/provisioning",
 "-e", "GF_AUTH_ANONYMOUS_ENABLED=true", "-e", "GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer",
 "-e", "GF_AUTH_DISABLE_LOGIN_FORM=true", "-e", "GF_AUTH_BASIC_ENABLED=false",
 "-e", "GF_SECURITY_ADMIN_PASSWORD=" + secrets.token_urlsafe(24),
 "-e", "GF_ANALYTICS_REPORTING_ENABLED=false", "-e", "GF_ANALYTICS_CHECK_FOR_UPDATES=false",
 "-e", "GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES=false", "-e", "GF_PLUGINS_PREINSTALL_DISABLED=true",
 "--entrypoint", "/bin/sh", "grafana/grafana:12.1.0", "-c", grafana_script)
 origin = "http://" + command("port", grafana, "3000/tcp").splitlines()[0]
 health = eventually(lambda: get("/api/health"))
 require(health["version"] == "12.1.0" and health["database"] == "ok", "Unexpected Grafana health")
 loaded = eventually(lambda: get("/api/dashboards/uid/variable-lab"))
 require(loaded["dashboard"]["templating"]["list"][0]["multi"] is True, "Dashboard provisioning mismatch")
 checks = {}
 cases = ((r"production\\.eu", 10), ("staging", 2),
 (r"(production\\.eu|staging)", 12), ("production.eu", 110),
 ("production.eu,staging", None), (".*", 112))
 # Wait for the synthetic series to be scraped before asserting empty results.
 def ready_series():
 result = get("/api/datasources/proxy/uid/lab_prom/api/v1/query?query=" +
 quote("count(lab_active_workers)", safe=""))["data"]["result"]
 return len(result) == 1 and float(result[0]["value"][1]) == 3
 eventually(ready_series)
 for pattern, expected in cases:
 expression = 'sum(lab_active_workers{environment=~"' + pattern + '"})'
 result = get("/api/datasources/proxy/uid/lab_prom/api/v1/query?query=" +
 quote(expression, safe=""))["data"]["result"]
 value = None if not result else float(result[0]["value"][1])
 require(len(result) <= 1 and value == expected, (expression, result))
 checks[expression] = value
 print(json.dumps({"app_url": origin + "/d/variable-lab/variable-escaping-lab",
 "grafana_version": health["version"],
 "backend_handwritten_query_checks": checks,
 "dashboard_interpolation": "NOT CHECKED",
 "serve_seconds": args.serve_seconds if args.serve else 0}), flush=True)
 if args.serve:
 Event().wait(args.serve_seconds)
finally:
 cleanup_owned((("container", (grafana, prom), "{{.Names}}"),
 ("network", (network,), "{{.Name}}")))

Interrupted or timed-out commands remain uncertain despite empty inventories. Cleanup failures chain the primary exception. Repeated interruption or abrupt death can prevent cleanup; final inventories cannot exclude delayed daemon work or concurrent changes.

Explicit checks remain active under Python optimization. The Python checks submit handwritten PromQL through the data-source proxy; they do not exercise dropdown interpolation. The comma-joined and wildcard controls expect no series and 112 respectively; neither is a dropdown test. The synthetic series hold ten workers for production.eu, two for staging, and one hundred for productionXeu. With only production.eu selected, “Selected workers” must show ten, while the deliberately unescaped control shows one hundred ten. Select both available options, then All: the correct panel must show twelve in each case. The raw control becomes empty because its comma-joined text does not match the intended alternatives.

For each browser selection, retain Query Inspector’s outgoing expression and returned value alongside the displayed selection. Check repeated URL parameters when selecting both values. Backend output explicitly labels dashboard interpolation “NOT CHECKED.” The earlier author reports Chrome 152 checks of these selections and fields. Exact browser captures were not recovered for this review, and neither containers nor browser were rerun. Those historical observations concern the provisioned synthetic dashboard, not tenant isolation or changing-discovery freshness. When adapting the dashboard, keep the excluded environment as a regression case and verify the final expression whenever variable type, formatter, refresh policy, or data source changes.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
Grafana variables, variable types and dashboard reuse, checked 2026-09-13.
Grafana Prometheus template variables, multi-value regex matching and refresh behavior, checked 2026-09-13.
Grafana variable syntax, data-source interpolation and explicit formatting overrides, checked 2026-09-13.
Grafana dashboard URL variables, repeated URL parameters for multiple selected values, checked 2026-09-13.
Python urllib.request, private opener and disabled automatic proxies, checked 2026-10-19.
Python subprocess, subprocess timeout scope, checked 2026-10-19.

XB Field Notes · XB Field Notes

InnoDB deadlocks roll back the whole victim transaction

When it breaks

Check it quickly

Reduce cycles without assuming they disappear

Sources

dnsmasq separates forwarded cache entries from configured local names

When it breaks

Check it quickly

Operate the listener and its data sources deliberately

Sources

LVM snapshots capture a block device in time

An exception store, not a copy

Exhaustion is invalidation

Merging, and why thin changes the failure

When it breaks

Check it quickly

Sources

Consul discovery reflects health policy and freshness

Registration and health are separate

DNS is not always passing-only

Freshness has several stages

When it breaks

Check it quickly

Sources

The scheduler accounts for requests, not momentary utilization

Account for the admitted specification

When it breaks

Check it quickly

Sources

Baggage carries context across an explicit boundary

Choose what crosses and what gets recorded

When it breaks

Check it quickly

Sources

The SYN backlog holds incomplete handshakes

The backlog argument is the accept queue

SYN cookies remain a fallback

The accept queue is where the pressure lands

When it breaks

Check it quickly

Sources

ZFS snapshots give the filesystem a rewind button

A name for one consistent tree

Where the space is, in zfs list -o space

Holds, diff, and comparing versions

When it breaks

Check it quickly

Sources

Hugo publication flags decide what enters the artifact

Separate publication metadata from build options

A preview artifact can still be published

When it breaks

Check it quickly

Sources

iotop shows which process is burning the disks

Per-process counters

See whatever sweeps the disk

Watch before you act

When it breaks

Check it quickly

Sources

A microcache serves fresh entries briefly and coalesces misses explicitly

When it breaks

Check it quickly

Choose freshness and overload behavior separately

Further reading

Sources

etcd elects a leader so writes happen in order

One leader per term, with voting restrictions

Count voters rather than processes

When it breaks

Check it quickly

Sources

BPF tracepoints peek inside the running kernel

Probe without pausing

Tracepoints are named events

Read a histogram, not a firehose

When it breaks

Check it quickly