Shell pipes stream between processes

Tue, 28 Sep 2027 00:00:00 +0000

A shell pipeline connects one command’s standard output to the next command’s standard input. The commands can run concurrently, so the producer need not finish before the consumer starts. This supports incremental processing, but the pipe itself does not guarantee that either program processes data incrementally. Buffering, early termination, and the status of each stage determine whether the complete operation does what you intended.

A pipe transports bytes between processes. Buffering, the lifetime of every open endpoint, and each stage’s status remain separate concerns.

A byte stream with finite capacity

A pipe transports bytes rather than records. A reader is not promised one read per line or one read per producer write. Programs must implement any framing they need. Standard error normally remains separate, which is why an upstream diagnostic can appear on the terminal without entering the downstream filter. Decide explicitly whether a consumer should receive diagnostics as data.

The kernel provides finite buffering. A blocking writer waits when the pipe fills, and a reader waits when the pipe is empty while writers remain open. This backpressure limits how far a producer can advance through that particular pipe. Capacity depends on the operating system and configuration; it is not universally sixty-four kibibytes. Applications can still accumulate arbitrarily large internal data structures, so a pipeline alone does not bound total memory.

User-space buffering adds another delay. A program that flushes every line to a terminal may buffer larger chunks when stdout is a pipe. An apparently idle consumer may therefore be waiting for data still buffered in the producer. Check the producer’s documented streaming and flush controls before changing kernel limits or adding arbitrary sleeps. Some transformations, such as a complete sort, inherently need more input before producing useful output.

Atomic writing is a separate property from total capacity. Linux guarantees noninterleaving for appropriately sized writes up to PIPE_BUF, with blocking-mode qualifications; that does not create message boundaries for readers. Several short writes may be returned together, and one write may need several reads. Do not design a parser around observed chunk sizes.

When it breaks

A reader sees end-of-file only after every descriptor referring to the write end is closed and all buffered bytes have been read. Closing the writers does not discard data already in the pipe. A child that inherited an unused write descriptor can keep a consumer waiting even after the apparent producer exited. In the other direction, writing after all read ends are closed generates SIGPIPE, or yields an EPIPE error when the signal is ignored. A consumer that deliberately stops early can therefore cause an otherwise healthy producer to terminate unsuccessfully.

By default, Bash returns the last stage’s status. An upstream failure followed by a successful cat can look successful overall. With pipefail, the aggregate result is the rightmost nonzero stage status, not the first failure in time or the leftmost failing command. The PIPESTATUS array retains individual stage results briefly. Capture it immediately: even an assignment or printf can replace the observation.

Most Bash pipeline stages run in subshell environments. An assignment in a loop on the right side of a pipe normally does not update the caller’s variable. Bash’s lastpipe option can alter the final stage’s execution when job control is inactive, so state assumptions need an explicit shell configuration. A redirected loop can avoid a pipeline when the caller must retain the loop’s state.

Finally, a foreground Bash pipeline waits for its stages to terminate. It does not return merely because the consumer read its first line. A producer that catches a broken pipe and continues unrelated work can delay completion. Treat a pipeline as several processes with connected descriptors and separate outcomes, rather than one command that necessarily succeeds or stops as a unit.

Check it quickly

Run the following Bash block with cat, yes, and head available. It compares default and pipefail statuses using a producer that deliberately exits seven. It then demonstrates two failing stages and an early-closing consumer. Errexit is disabled only while collecting expected nonzero results, then restored before assertions.

bash --noprofile --norc <<'BASH'
set -e
set +o pipefail
set +e
(printf 'partial\n'; exit 7) | cat >/dev/null
plain=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${plain[*]} == '0 7 0' ]]
set -o pipefail
set +e
(printf 'partial\n'; exit 7) | cat >/dev/null
strict=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${strict[*]} == '7 7 0' ]]
set +e
(exit 7) | (exit 4)
multiple=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${multiple[*]} == '4 7 4' ]]
set +e
yes | head -n 1 >/dev/null
early=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${early[0]} -ne 0 && ${early[1]} -ne 0 && ${early[2]} -eq 0 ]]
printf 'default: %s\npipefail: %s\nrightmost: %s\nearly close: %s\n' \
 "${plain[*]}" "${strict[*]}" "${multiple[*]}" "${early[*]}"
printf 'PASS: aggregate and individual stage outcomes\n'
BASH

Each reported array starts with the aggregate status, followed by the stages in source order. Expect 0 7 0, 7 7 0, and 4 7 4 for the first three. The final consumer succeeds while yes returns nonzero; the test deliberately does not require a platform-specific signal number. A production pipeline must distinguish this intentional early closure from lost required output. Measure throughput and memory separately; these status assertions do not measure buffering performance.

Sources

GNU Bash: Pipelines, process execution and aggregate status rules, checked 2026-09-06.
Linux man-pages: pipe(7), byte streams, descriptor lifetime, capacity, and SIGPIPE, checked 2026-09-06.
GNU Bash: Special Parameters, the immediately preceding command’s status, checked 2026-09-06.

Pipes · XB Field Notes

Shell pipes stream between processes

A byte stream with finite capacity

When it breaks

Check it quickly

Sources