ptrace — How strace and gdb Really Work

You run strace on a process doing millions of syscalls per second and throughput drops by two orders of magnitude. You attach gdb to a live service — and the whole process freezes. These aren’t accidental costs of badly written tools. They’re the architecture of ptrace: a single system call that practically the entire Linux debugging ecosystem stands on. strace as a debugging primitive, gdb, ltrace, reverse-engineering frameworks — they all boil down to the same kernel interface and inherit its limitations as a package deal.

Understanding the mechanics of ptrace answers three questions everyone debugging on Linux eventually asks: why tracing syscalls is so expensive, why gdb -p can bounce off the system with Operation not permitted, and what a breakpoint actually is, given that the CPU has no idea your source code exists.

One Syscall, the Entire Debugger Ecosystem

ptrace is a multiplexer. One call — ptrace(request, pid, addr, data) — where the first argument selects the operation: PTRACE_TRACEME, PTRACE_ATTACH, PTRACE_PEEKDATA, PTRACE_GETREGS, PTRACE_CONT and a few dozen other requests, all documented in ptrace(2). The kernel doesn’t have a separate “debugger API” — it has this one syscall and a set of semantics built around stopping a process.

The model is asymmetric: tracer and tracee. The relationship is per-thread — every thread is traced separately — and it’s exclusive: one tracee has exactly one tracer. That’s why you can’t attach gdb to a process already sitting under strace; the kernel returns EPERM and that’s the end of the discussion. The communication is brutally simple: the tracee stops in one of several kinds of stops, the kernel notifies the tracer via waitpid(), the tracer pokes around the frozen process — registers, memory, signals — and decides how to resume it. Everything a debugger does, from printing syscalls to step-debugging, is a composition of this one loop: stop, inspect, resume.

Three Ways to Take Over a Process

PTRACE_TRACEME + execve()

The classic “run under a debugger” pattern. The child process, right after fork() and copy-on-write, calls ptrace(PTRACE_TRACEME, ...) — declaring: I consent to being traced by my parent — and then executes execve(). After a successful execve the kernel delivers SIGTRAP to the child, so the new program stops before executing its first own instruction. This is exactly where strace ./app and gdb ./app start their session: they have the process frozen at startup, with full control before anything happens.

PTRACE_ATTACH

Attaching to an already-running process. ATTACH sends the tracee a SIGSTOP, which has side effects: the process goes through a group-stop visible to the rest of the system, and the attach + stop sequence is racy against signals arriving in the meantime. For years debuggers lived with these ambiguities, wrapping them in heuristics — and for years this was a source of subtle bugs in the debugging tools themselves.

PTRACE_SEIZE

The newer interface, designed after the lessons of ATTACH: it takes over the tracee without stopping it. The process keeps running, and the tracer stops it only when it actually wants to — with an explicit PTRACE_INTERRUPT. SEIZE also reports stops in an unambiguously distinguishable way (PTRACE_EVENT_STOP), which eliminates a whole class of “was that my SIGSTOP or someone else’s” questions. Modern strace -p uses SEIZE under the hood. If you’re writing your own tracer in 2026 and you start with ATTACH — you’re starting with legacy.

Signal-Delivery-Stop — the Tracer Sees Signals First

Tracing changes the semantics of signal delivery. Every signal addressed to the tracee — except SIGKILL — first stops the process and lands at the tracer. Only then does the tracer, when resuming the tracee, decide: deliver the signal, swap it for a different one, or suppress it entirely. The mechanics of dispositions, handlers and interrupted syscalls are a separate topic — taken apart in the anatomy of signals and handlers in Linux — but from ptrace’s perspective the key fact is the interception itself: the debugger stands in the middle of the delivery path.

That’s why strace prints --- SIGCHLD --- lines between syscalls, why gdb can catch SIGSEGV before the process dies, and why “suppressing” a signal in a debugger is possible at all. It’s also why a process under a debugger is behaviorally a different process: delivery timings shift, EINTR shows up in places you’ve never seen it, and a heisenbug — by definition — disappears exactly when you start looking. SIGKILL remains the one exception: it kills the tracee immediately and no tracer can stop it.

Syscall Stops, or a Minimal strace in C

The heart of strace is the PTRACE_SYSCALL request: resume the tracee and stop it at the nearest system call boundary. Every syscall generates two stops — a syscall-enter-stop before entering the kernel and a syscall-exit-stop after leaving it. In the enter-stop, the syscall number and arguments sit in registers (on x86-64: orig_rax, rdi, rsi, rdx…); in the exit-stop, the result waits in rax. With the PTRACE_O_TRACESYSGOOD option, syscall stops report as SIGTRAP | 0x80, so you can tell them apart from ordinary signal delivery without guessing.

Below is a complete, minimal tracer: it runs a program and prints the number and result of every syscall. Sixty lines, zero magic.

#define _GNU_SOURCE
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/ptrace.h>
#include <sys/user.h>
#include <sys/wait.h>
#include <unistd.h>

/* Resumes the tracee to the nearest syscall-stop.
 * Signals that aren't syscall-stops get re-delivered on resume.
 * Returns 0 on a syscall-stop, -1 when the tracee is gone. */
static int wait_for_syscall(pid_t child) {
    int status, sig = 0;
    for (;;) {
        if (ptrace(PTRACE_SYSCALL, child, NULL, (void *)(long)sig) == -1) {
            perror("ptrace(PTRACE_SYSCALL)");
            return -1;
        }
        sig = 0;
        if (waitpid(child, &status, 0) == -1) {
            perror("waitpid");
            return -1;
        }
        if (WIFEXITED(status) || WIFSIGNALED(status))
            return -1;                        /* tracee is gone */
        if (WIFSTOPPED(status)) {
            int stopsig = WSTOPSIG(status);
            if (stopsig == (SIGTRAP | 0x80))  /* PTRACE_O_TRACESYSGOOD */
                return 0;                     /* syscall-stop */
            sig = stopsig;                    /* signal-delivery-stop:
                                                 inject on resume */
        }
    }
}

int main(int argc, char *argv[]) {
    if (argc < 2) {
        fprintf(stderr, "usage: %s <program> [args...]\n", argv[0]);
        return EXIT_FAILURE;
    }

    pid_t child = fork();
    if (child == -1) {
        perror("fork");
        return EXIT_FAILURE;
    }

    if (child == 0) {                         /* tracee */
        if (ptrace(PTRACE_TRACEME, 0, NULL, NULL) == -1) {
            perror("ptrace(PTRACE_TRACEME)");
            _exit(127);
        }
        execvp(argv[1], &argv[1]);            /* stop: SIGTRAP after execve */
        perror("execvp");
        _exit(127);
    }

    int status;                               /* tracer */
    if (waitpid(child, &status, 0) == -1 || !WIFSTOPPED(status)) {
        fprintf(stderr, "tracee never reached execve\n");
        return EXIT_FAILURE;
    }
    if (ptrace(PTRACE_SETOPTIONS, child, 0,
               PTRACE_O_EXITKILL | PTRACE_O_TRACESYSGOOD) == -1) {
        perror("ptrace(PTRACE_SETOPTIONS)");
        return EXIT_FAILURE;
    }

    while (wait_for_syscall(child) == 0) {    /* syscall-enter-stop */
        struct user_regs_struct regs;
        if (ptrace(PTRACE_GETREGS, child, NULL, &regs) == -1) {
            perror("ptrace(PTRACE_GETREGS)");
            break;
        }
        long long nr = (long long)regs.orig_rax;

        if (wait_for_syscall(child) != 0) {   /* syscall-exit-stop */
            fprintf(stderr, "syscall(%lld) = ?\n", nr);
            break;
        }
        if (ptrace(PTRACE_GETREGS, child, NULL, &regs) == -1) {
            perror("ptrace(PTRACE_GETREGS)");
            break;
        }
        fprintf(stderr, "syscall(%lld) = %lld\n", nr, (long long)regs.rax);
    }
    return EXIT_SUCCESS;
}

A few decisions in this code aren’t accidental. PTRACE_O_EXITKILL cleans up the relationship: when the tracer dies, the tracee gets SIGKILL instead of hanging in a stop forever. wait_for_syscall re-injects signals that aren’t syscall-stops — without that, the first stray SIGCHLD or SIGALRM breaks the enter/exit parity and the “results” stop matching the calls. Reading registers via PTRACE_GETREGS is x86-64-specific; portable code uses PTRACE_GET_SYSCALL_INFO, which returns the syscall number, arguments and result in an architecture-independent structure. Real strace adds argument decoding (strings, structs, flags), child tracing via PTRACE_O_TRACEFORK and a dozen other mechanisms on top — but the loop in the middle is exactly this one.

How gdb Sets a Breakpoint

A breakpoint isn’t a CPU feature reserved for debuggers (apart from the limited pool of hardware watchpoints). It’s vandalism with the kernel’s consent: via PTRACE_POKETEXT, gdb overwrites the first byte of the instruction at the target address with 0xCC — the one-byte int3 opcode. When execution hits that spot, the CPU raises a trap, the kernel translates it into SIGTRAP, and SIGTRAP — like every signal of a traced process — goes to the tracer first. gdb rolls RIP back by one, restores the original byte, executes that single instruction via PTRACE_SINGLESTEP, puts 0xCC back and resumes the process. The whole illusion of “stopped at line 47” is a sequence of byte swaps and signals.

Two things follow. First: the .text section is mapped without write permission, but ptrace writes through the kernel’s interface, bypassing the page protection the process itself sees — the same memory protection mechanism that, on an ordinary userspace write, would end the way the anatomy of a segfault from MMU to core dump describes. Second: you first have to find the breakpoint’s address, and between the address in the binary and the address in memory stand ASLR, relocation and the dynamic linker — how segments land in memory before main() runs is taken apart in ELF and dynamic linking.

Where the ptrace Overhead Comes From

Let’s count the cost of one traced syscall in PTRACE_SYSCALL mode. Two stops per call; each stop means freezing the tracee, a context switch to the tracer, at least waitpid() and PTRACE_GETREGS in the tracer — more syscalls — and a context switch back. Four context switches and several system calls of overhead per one traced getpid(), which by itself costs a fraction of a microsecond. Add reading the tracee’s memory: classic PTRACE_PEEKDATA returns one machine word per call, so peeking at a 4-kilobyte write() buffer means 512 separate syscalls. Newer tracers save themselves with process_vm_readv(), which copies whole blocks in a single call — but nothing eliminates the stops.

Hence the two-orders-of-magnitude slowdowns on syscall-heavy workloads — and hence the first rule of hygiene: strace in production is a decision, not a reflex. You can make that decision smarter. strace --seccomp-bpf injects a seccomp filter into the kernel that lets uninteresting syscalls through without stopping the process — the stop happens only for calls matching -e trace=.... And eBPF goes one step further: it collects data on the kernel side without stopping the tracee at all, at the price of losing what’s most valuable in ptrace — the ability to modify process state.

YAMA, or Why gdb -p Throws “Operation not permitted”

The classic ptrace permission model was simple: same UID can attach. From a security standpoint that’s a disaster — any compromised user process could read the memory of that user’s browser, password manager or SSH agent, because “same UID, right?”. The YAMA module adds the kernel.yama.ptrace_scope sysctl with four levels: 0 — the classic model; 1 — attach only to your own direct children (the default on most distributions); 2 — attach only with CAP_SYS_PTRACE; 3 — attach disabled permanently, until reboot.

$ cat /proc/sys/kernel/yama/ptrace_scope
1
$ gdb -p "$(pidof myapp)"
ptrace: Operation not permitted.
$ sudo sysctl -w kernel.yama.ptrace_scope=0   # temporary and deliberate, not permanent

Hence the asymmetry that regularly confuses people: strace ./app always works — the process volunteers via PTRACE_TRACEME as the tracer’s child — but strace -p PID on the same machine bounces off level 1. That’s not a bug in your strace, and your permissions aren’t “broken”. It’s an LSM policy working exactly as configured. A per-process exception comes from prctl(PR_SET_PTRACER, pid) — that’s how crash handlers like Breakpad work, which have to let an external process dump their memory.

When ptrace Is the Right Tool — and When It Isn’t

ptrace is surgery on a living organism: full control, touching state, a stopped patient. That level of invasiveness is right for debugging an incident — and wrong as a permanent part of the system. Rules that hold up: strace and gdb to verify a specific hypothesis about a specific process, with the cost accepted; continuous syscall observation in production — eBPF, not ptrace. If your “observability agent” keeps processes under ptrace, that’s not observability — that’s a permanent debugger in production, with all the stops and overhead described above.

And the methodological point: ptrace gives you a microscope, not a compass. Stopping a process and staring at registers makes sense only once you know what you’re looking for — the tool serves to verify hypotheses, not generate them, which is what debugging with hypotheses instead of guessing takes apart.

The insight to take away: the entire Linux debugging ecosystem — strace, gdb, ltrace and every tracer you’ll ever write — is a loop over four operations of one syscall: stop, inspect, modify, resume. All their superpowers (signal interception, breakpoints, peeking at memory) and all their limitations (overhead, one tracer per tracee, YAMA) are properties of that loop. Understand the loop, and no debugger will ever surprise you again.