Linux Pipes — Why EOF Does Not Arrive When the Producer Exits

By /

The producer has exited with status 0. The consumer has read its output, yet the next read still waits. Adding a delay changes nothing. This is the point to stop asking “is the child still running?” and ask “who still holds the write end open?”.

This article examines an ordinary anonymous pipe on Linux, one reader, and positive-size reads. Sockets, reopening FIFOs, and pipe2() packet mode are outside its scope. Python serves as a thin interface to file descriptors, not as a model of text-stream buffering.

EOF describes the channel, not the process status

An empty pipe has two distinct states: some write end remains open, or none does. In the first case, a blocking read waits and a nonblocking read reports EAGAIN. In the second, a read returns 0: EOF. One producer exiting is insufficient if another write descriptor remains open elsewhere. This rule is documented in pipe(7).

A read() result smaller than the requested size does not mean the stream has ended. Receiving four of the requested 64 bytes is a valid result. Conversely, calling read() with a count of 0 cannot establish EOF: it may return zero without testing the condition you want to investigate. Every read request in the example asks for 64 bytes. See read(2).

A descriptor number does not own the entire pipe

A file descriptor is a handle within a process. dup() creates another handle referring to the same open file description. Closing the original number does not close its duplicate. For a pipe’s write end, an otherwise unused handle can therefore keep the channel open. The sharing rules are described in dup(2).

This explanation uses a model of reachable handles, not a literal count of every descriptor inside the kernel. Do not infer that each dup() increments a particular implementation-level “writers” counter. The diagnostic question is whether a reference still keeps the write end open.

After fork(), parent and child have descriptors referring to the same open file descriptions. In a typical parent-reader, child-writer arrangement, the parent therefore closes its unused write end and the child its unused read end. Closing the child’s handle does not remove the parent’s copy. This applies the rules of fork(2), explained more broadly in the article on fork(), copy-on-write, and file descriptors.

Experiment: the producer has exited, but EOF has not arrived

Save the complete program below as pipe_eof.py. It requires Linux and Python 3.10 or later and uses only the standard library. It creates one child process and two small pipes, with no working files, sudo, or system configuration changes. Run it as a standalone script.

The parent deliberately retains extra, a duplicate of the write end. subprocess.run() returns after the child exits. The original handle w is then closed, while extra stays open. Nonblocking reads demonstrate the missing EOF without hanging the program. ExitStack arranges descriptor cleanup even on exceptions; explicit checks remain active when Python runs with -O.

#!/usr/bin/env python3
"""Bounded Linux pipe experiment; creates only a four-byte child output."""
import os
import select
import subprocess
import sys
from contextlib import ExitStack


def require(condition: bool, message: str) -> None:
    if not condition:
        raise RuntimeError(message)


def experiment() -> None:
    if not sys.platform.startswith("linux"):
        raise SystemExit("This experiment requires Linux.")
    with ExitStack() as cleanup:
        r, w = os.pipe()
        cleanup.callback(os.close, r)
        writers = ExitStack()
        cleanup.callback(writers.close)
        writers.callback(os.close, w)
        extra = os.dup(w)
        extra_writer = ExitStack()
        cleanup.callback(extra_writer.close)
        extra_writer.callback(os.close, extra)
        os.set_blocking(r, False)
        child = subprocess.run(
            [sys.executable, "-I", "-c",
             "import os; n = os.write(1, b'DATA'); raise SystemExit(n != 4)"],
            stdout=w, stdin=subprocess.DEVNULL, close_fds=True,
            check=True, timeout=5,
        )
        writers.close()
        print(f"producer exit: {child.returncode}")
        first = os.read(r, 64)
        require(first == b"DATA", f"Unexpected payload: {first!r}")
        print(f"payload: {first!r}")
        try:
            result = os.read(r, 64)
        except BlockingIOError:
            print("extra writer open: EAGAIN")
        else:
            raise RuntimeError(f"Expected EAGAIN, got {result!r}")
        extra_writer.close()
        result = os.read(r, 64)
        require(result == b"", f"Expected EOF, got {result!r}")
        print(f"last writer closed: {result!r} (EOF)")

    with ExitStack() as cleanup:
        r, w = os.pipe()
        cleanup.callback(os.close, r)
        writers = ExitStack()
        cleanup.callback(writers.close)
        writers.callback(os.close, w)
        os.set_blocking(r, False)
        require(os.write(w, b"TAIL") == 4, "Short fixture write")
        writers.close()
        poller = select.poll()
        poller.register(r, select.POLLIN)
        events = poller.poll(0)
        mask = next((mask for fd, mask in events if fd == r), 0)
        require(bool(mask & select.POLLHUP), "Expected POLLHUP")
        print("buffered after close: POLLHUP")
        tail = os.read(r, 64)
        require(tail == b"TAIL", f"Lost buffered bytes: {tail!r}")
        print(f"buffered payload: {tail!r}")
        require(os.read(r, 64) == b"", "Expected EOF after draining")
        print("drained: EOF")


if __name__ == "__main__":
    experiment()

python3 pipe_eof.py

TEST EXECUTED: Python 3.12.14, Linux 6.18.55, x86_64, isolated environment. The following is actual execution output, not a performance measurement:

producer exit: 0
payload: b'DATA'
extra writer open: EAGAIN
last writer closed: b'' (EOF)
buffered after close: POLLHUP
buffered payload: b'TAIL'
drained: EOF

The first four lines separate three observations: the child exited successfully, its four bytes were received, and the next read still reported EAGAIN. Only closing extra changed the result to an empty byte string. We used neither sleep() nor an assumption that a process “should have exited by now”.

The program uses os.read(), whose result does not pass through a text buffer. os.set_blocking(r, False) sets O_NONBLOCK. In the Python version used here, os.pipe() and os.dup() create descriptors that are non-inheritable across exec; that does not prevent the parent itself from retaining extra. Documentation: os.

POLLHUP is not permission to discard the remaining bytes

The program’s second part writes TAIL and closes the write end before reading. poll() reports POLLHUP, but the subsequent read still returns TAIL. EOF appears only after those bytes have been consumed. This is a separate condition from the sender closing, consistent with the POLLHUP description in poll(2).

The implication for an event loop is that pending data still needs handling. Closing the reader immediately on HUP can lose the last part of a response. Our program requires the POLLHUP bit but does not compare the entire mask against a single constant: other bits may be present simultaneously. The case deliberately excludes a competing reader that could consume the data between observation and reading.

Why close-on-exec cannot fix everything

FD_CLOEXEC takes effect on a successful execve(), not on fork() itself. It limits unwanted handles reaching a new program, but it does not close a handle in the parent or in a child that never executes exec. Descriptor preservation is documented in execve(2).

In the example, close_fds=True restricts unintended descriptor passing to the launched program; stdout=w is intentional redirection. Neither option removes the parent’s extra. The practical fix must address the owner of the unnecessary handle. Setting a flag in another process is no substitute.

Do not confuse missing EOF with a full-buffer deadlock

This experiment waits for the producer before reading only because the producer writes exactly four bytes into an initially empty pipe. Do not copy that ordering when capturing unbounded output. The child can block on writing while the parent waits for its termination. The subprocess documentation warns against this use of Popen.wait() with PIPE; communicate() handles data exchange but buffers the received data in memory.

Separate the hypotheses “a writer has not been closed” and “nobody is consuming the produced data”. We reproduced the first case; we deliberately avoid creating the second. For a large stream, design continuous consumption and memory limits. A timeout helps detect lack of progress, but does not identify which hypothesis is correct.

How to find the forgotten handle

Start with resource ownership: who creates the pipe, who calls dup()/dup2(), which processes inherit handles, and on which path each handle is closed. Include error handling and helper processes. Naming a variable “producer” does not prove that only that process holds the write end.

On a machine that permits tracing your own processes, you can run:

strace -f -e trace=pipe,pipe2,dup,dup2,dup3,fcntl,read,write,close,poll,wait4 python3 pipe_eof.py

TRACING BLOCKED: in the test environment, strace could not obtain PTRACE_TRACEME: Operation not permitted. Running the program without the tracer passed its checks. We therefore provide no invented trace. The -f flag follows child processes, and the filter restricts displayed calls; see strace(1). Do not change global security settings merely for this demonstration.

Look for a relationship in the trace: pipe creation, creation of another reference, closure of one handle, and later closure of the last one. Descriptor numbers need not match between runs. Here EAGAIN is evidence of a nonblocking read’s state, not a transmission failure.

Limits of the experiment and the engineering lesson

The test demonstrates the effect of a duplicate in the parent and the preservation of data after the writer closes. It does not measure pipe capacity, scheduler overhead, or throughput, and does not test every Python or kernel version. A handle retained in another process is a documented extension of the model, not a separate measurement in this example. This code is not a general-purpose streaming library.

Waiting for EOF means auditing the lifetime of a channel. Process termination is only one operation affecting that lifetime. In practice, design descriptor ownership as explicitly as memory ownership: every additional reference needs a reason to exist and a place where it is closed.

Sources and verification scope

Sources were read on 2026-10-25. Linux man-pages 6.19; Python 3.11 API documentation, executed with Python 3.12.14. The different documentation and interpreter versions are explicit; the test validates only the interfaces used in the described environment.

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top