The Scaling Problem of I/O in Userspace
At high connection counts, one thread per client can cost more than you want to pay. An epoll event loop lets one thread wait on many descriptors. io_uring takes a different approach: submit an operation and collect its result. What changes in practice?
Then 2019 came along with kernel 5.1, and Jens Axboe introduced io_uring — a fundamentally different asynchronous I/O model, based not on readiness notification but on completion notification, with two shared queues between userspace and kernel.
We will compare readiness and completion models, batching and queue-management costs. There is no single RPS threshold for abandoning epoll: what matters is where your application spends time.
epoll — The Readiness-Based Model
Introduced in kernel 2.5.45 as an answer to the limitations of select() and poll(), epoll is a registry of file descriptors for which the kernel notifies userspace when they become ready for an operation. Three syscalls define the interface:
epoll_create1()— creates an epoll instance and returns its fdepoll_ctl()— adds/removes/modifies watched fdsepoll_wait()— blocks (or times out) until any fd becomes ready
The Critical Catch: epoll Notifies but Does Not Execute
epoll_wait() reports readiness but does not read data. The application calls read() or recv(). This does not mean two syscalls per operation: a wait can return multiple ready descriptors, and the application can perform multiple reads before waiting again.
import select
import socket
# Minimal TCP receiver: reads and discards data; not an echo server.
server = socket.socket()
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.setblocking(False)
server.bind(("127.0.0.1", 8080))
server.listen(128)
epoll = select.epoll()
epoll.register(server.fileno(), select.EPOLLIN | select.EPOLLET)
connections = {}
def close_client(fd):
epoll.unregister(fd)
connections.pop(fd).close()
try:
while True:
for fd, event in epoll.poll():
if fd == server.fileno():
# Drain accept until EAGAIN too.
while True:
try:
client, _ = server.accept()
except BlockingIOError:
break
client.setblocking(False)
connections[client.fileno()] = client
epoll.register(client.fileno(), select.EPOLLIN | select.EPOLLET)
elif fd in connections:
try:
while True:
data = connections[fd].recv(4096)
if not data: # EOF
close_client(fd)
break
# Process received bytes here.
except BlockingIOError:
if event & (select.EPOLLERR | select.EPOLLHUP):
close_client(fd)
except OSError:
close_client(fd)
finally:
for client in connections.values():
client.close()
epoll.close()
server.close()This shows ET reading without a protocol, client limits or writing. An echo server would also need pending-output buffers, partial-write handling and EPOLLOUT. A userspace-to-kernel transition is not the same as a scheduler context switch. Measure in your environment, starting with syscall counts from strace.
Edge-Triggered vs Level-Triggered
Level-triggered (LT) reports a descriptor while its readiness condition persists. With edge-triggered (ET), do not rely on another notification merely because you left data buffered. Use nonblocking descriptors and handle I/O until EAGAIN, including accepting connections. ET may reduce notifications; it does not guarantee a faster application.
io_uring — The Completion-Based Model
io_uring doesn’t ask the kernel “may I?”. io_uring tells the kernel “do X, ping me when done”. The mechanics rest on two shared ring buffers in memory, mmap()-ed between userspace and kernel space:
- Submission Queue (SQ) — userspace writes operations (SQE — Submission Queue Entry)
- Completion Queue (CQ) — kernel writes results (CQE — Completion Queue Entry)
Shared memory needs correct queue synchronization; liburing handles these details. The kernel interface includes io_uring_setup(), io_uring_enter() and io_uring_register(). SQPOLL can reduce submission calls, but waking its thread or waiting for completion may still require a syscall.
Anatomy of a Submitted Operation
This requires liburing headers and the library. Save as uring-read.c, build with cc -Wall -Wextra uring-read.c -luring -o uring-read and pass an existing file. It demonstrates one operation, not a batching benchmark; runtime behavior was not verified in the preparation environment.
#include <liburing.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
int main(int argc, char **argv) {
if (argc != 2) {
fprintf(stderr, "Usage: %s FILE\n", argv[0]);
return 1;
}
int fd = open(argv[1], O_RDONLY);
if (fd < 0) { perror("open"); return 1; }
struct io_uring ring;
int ret = io_uring_queue_init(8, &ring, 0);
if (ret < 0) {
fprintf(stderr, "queue_init: %s\n", strerror(-ret));
close(fd);
return 1;
}
int status = 1;
char buf[4096]; /* Must remain valid until the read completes. */
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
if (!sqe) { fprintf(stderr, "SQ full\n"); goto out; }
io_uring_prep_read(sqe, fd, buf, sizeof(buf), 0);
ret = io_uring_submit(&ring);
if (ret != 1) {
fprintf(stderr, "submit: %s\n", ret < 0 ? strerror(-ret) : "no submission");
goto out;
}
struct io_uring_cqe *cqe;
ret = io_uring_wait_cqe(&ring, &cqe);
if (ret < 0) {
fprintf(stderr, "wait_cqe: %s\n", strerror(-ret));
goto out;
}
if (cqe->res < 0) {
fprintf(stderr, "read: %s\n", strerror(-cqe->res));
} else {
printf("Read %d bytes\n", cqe->res);
status = 0;
}
io_uring_cqe_seen(&ring, cqe);
out:
io_uring_queue_exit(&ring);
close(fd);
return status;
}The read is not a separate read() syscall. The kernel executes the submitted operation and places the result in the CQ. One io_uring_enter() can submit multiple prepared operations, within queue capacity. Batching amortizes syscall overhead, but preparing and handling entries still costs work.
SQPOLL — Eliminating the Syscall from the Hot Path
IORING_SETUP_SQPOLL starts a kernel thread that reads the submission queue. While active, it can notice entries without a submission syscall. After an idle period it sleeps and needs waking; waiting for completion may also enter the kernel.
Polling consumes CPU. Since Linux 5.13, SQPOLL itself no longer requires special privileges; earlier versions had additional restrictions. Separately check security policy and io_uring availability in the runtime environment.
Batching and Multishot Instead of Fixed Syscall Counts
Multishot accept, available since Linux 5.19, lets one request produce results for successive connections. In liburing, use io_uring_prep_multishot_accept(). Check IORING_CQE_F_MORE: when absent, the request will produce no more results. This is different from IOSQE_BUFFER_SELECT.
What to Measure Before Changing the I/O Model
Compare identical work: message sizes, active connections, latency and CPU use. For file tests, control the page-cache state too. One workload’s result does not establish a migration threshold for others.
If profiling identifies the cost of submitting many small operations, or you need concurrent file I/O, try io_uring. If the time goes into a database, serialization or application logic, changing the event loop may leave the main bottleneck untouched.
Compatibility and Security
io_uring appeared in Linux 5.1, but operation and flag support depends on kernel versions and distribution patches. An API’s introduction version is not a deployment recommendation. Check:
- required operations in the target kernel and liburing;
- seccomp policy and system io_uring restrictions;
- error handling or an alternative backend;
- other-platform requirements: epoll is Linux-specific too.
Availability on your laptop does not imply availability in a production container. Check the actual host configuration and queue initialization result.
Check Your Runtime’s Backend First
In C and C++, liburing provides queue helpers. For languages with their own runtime, check the documentation of your version: networking and file backends can differ. Do not change language solely to get one I/O interface.
Conclusion: Measurement > Ideology
If the current backend meets requirements and I/O is not a measured bottleneck, migration may not justify its cost. If an io_uring experiment helps, check it under load, on errors and in the target configuration. Fewer syscalls alone do not guarantee faster responses.
An engineer who measures first the CPU profile under production load will see whether syscall overhead even appears in the top 10. An engineer who writes liburing code immediately because they read about it on Hacker News will likely spend a week debugging multi-shot accepts for a service serving 800 req/s that nobody will ever scale to 80k.
Premature optimization is the root of all evil — Knuth knew what he was talking about. Profile before optimizing isn’t a slogan, it’s engineering discipline.
If this level — concrete syscalls and what happens between userspace and the kernel — is your territory, a similar under-the-hood section runs through the article on what fork() in Linux really does. The performance-without-abstraction thread continues in the piece on cgroups v2 as a native runtime isolation primitive without Docker.
Technical references: epoll(7), SQPOLL, multishot accept.
