System Calls & the User–Kernel Boundary

Your program runs in user space, and user space is fenced off from anything that could hurt the rest of the machine. It cannot issue the privileged CPU instructions that talk to devices, it cannot read another process’s memory, and it cannot reprogram the hardware. That fence is not politeness — it is enforced by the CPU, and stepping over it directly simply faults. So how does any real work get done? Through exactly one controlled opening: the system call.

Think of the kernel as a bank vault and the syscall interface as the single teller window. You do not climb over the counter to grab cash; you slide a request through the window, the teller checks that you are allowed, does the work in the secure area, and slides a result back. Every file you open, every byte you send, every child process you spawn goes through that window. There is no side entrance.

Here is the motivating surprise. A humble printf("hello\n") looks like pure library code, but it cannot put a single character on your terminal by itself. Underneath, once its buffer is ready to flush, it calls write() — and write() is a system call. The friendly function you have typed a thousand times is, at the bottom, a request slid through the kernel’s window.

User space

Where your program runs, with restricted privileges; it cannot touch hardware or other processes directly.

Kernel space

The privileged half of the machine where the trusted kernel code runs and hardware is actually driven.

System call

The single controlled mechanism by which user code requests privileged work from the kernel.

The boundary

The hardware-enforced line between the two worlds; syscalls are the only sanctioned way across it.

A system call is not a normal function call — a normal call just jumps to another address in your own address space. A syscall has to change the CPU’s privilege level, and only a special instruction is allowed to do that. On modern x86-64 Linux that instruction is literally called syscall; older code used int 0x80 (a software interrupt). Executing it triggers a trap: the CPU stops running your code, switches into kernel mode, and jumps to a fixed kernel entry point the OS installed at boot.

Because the trap cannot pass arguments the way a C call does, there is a strict convention. The syscall number goes in a designated register (rax on x86-64), and the arguments go in a fixed sequence of registers (rdi, rsi, rdx, r10, r8, r9). The kernel reads the number, looks it up in the system-call table to find the right handler, and calls it.

Now the trusted part runs. The kernel does not blindly obey — it validates everything: is this file descriptor really yours, does this pointer actually point into your address space, are you permitted to touch this file? If anything is wrong it refuses. If all is well it does the work, places the return value back in rax, and executes a return-from-trap that flips the CPU back to user mode and resumes your code on the instruction after the syscall. On failure the wrapper turns the negative result into -1 and sets the global errno so you can find out why.

• ▸Load the syscall number and arguments into the agreed registers.
• ▸Execute the trap instruction; the CPU switches to kernel mode and jumps to the kernel entry point.
• ▸The kernel dispatches through the syscall table, validates arguments, and performs the privileged work.
• ▸The result is placed in a register; the CPU switches back to user mode and returns to your code.
• ▸A negative kernel result becomes a -1 return and a set errno in the calling program.

Almost no application executes the raw trap instruction itself. Instead it calls a thin wrapper function provided by the C library (glibc on most Linux systems). The wrapper for write() is a few lines of assembly: move the syscall number into rax, move your arguments into the right registers, execute syscall, then translate the result. Its whole job is to make a system call look like an ordinary C function so you never have to write assembly.

This is where the vocabulary matters. A library call is any function the C library exposes; a system call is specifically a request that crosses into the kernel. printf and malloc are library calls that live entirely in user space — until they decide they need the kernel, at which point printf eventually calls write() and malloc eventually calls brk() or mmap() to get more memory. So write, read, open, brk, and mmap are system calls; printf, malloc, and strlen are library calls. strlen never crosses the boundary at all.

#include <stdio.h>   /* printf  — a buffered library call */
#include <unistd.h> /* write   — a direct syscall wrapper */
#include <string.h>

int main(void) {
    printf("via printf\n");            /* buffered in user space */

    const char *msg = "via write\n";   /* fd 1 = standard output */
    write(1, msg, strlen(msg));         /* traps into the kernel now */
    return 0;
}

Wrapper

A tiny libc function that loads registers and executes the trap so a syscall looks like a normal C call.

Library call

Any C-library function (printf, malloc, strlen); it may or may not end up making a system call.

System call

A request that actually crosses into the kernel (write, read, open, mmap, brk).

glibc

The GNU C library — the usual provider of these wrappers and of the buffered stdio layer on Linux.

Linux has hundreds of system calls, but they fall into a handful of familiar families. You do not need to memorise the list; you need the map, so that when you meet a new call you know which neighbourhood it lives in and roughly what it is for.

Process control

Create, run, and end programs: fork (clone a process), exec (replace its image), exit (terminate), wait (reap a child).

File management

Work with files by descriptor: open, read, write, close, lseek, stat.

Device management

Talk to devices, which Linux exposes as files: read/write on device nodes, plus ioctl for device-specific control.

Information & maintenance

Query or set system and process state: getpid, gettimeofday/time, uname, getrlimit.

Communication

Move data between processes or machines: pipe, socket, connect, send, recv, shmget.

The example below shows the information family at its simplest: getpid asks the kernel for this process’s ID. It cannot be answered in user space — only the kernel knows the process’s identity — so even this trivial-looking call is a genuine trip across the boundary.

#include <stdio.h>
#include <unistd.h>   /* getpid */

int main(void) {
    pid_t pid = getpid();          /* syscall: kernel returns our PID */
    printf("my pid is %d\n", pid);
    return 0;
}

Let us copy a file to the screen using nothing but raw system calls — no stdio, no printf, no buffering the library does for us. This is exactly what a tool like cat does at its core, and reading it line by line teaches you the shape of almost all Unix I/O.

#include <fcntl.h>    /* open */
#include <unistd.h>  /* read, write, close */
#include <stdio.h>   /* perror */

int main(int argc, char **argv) {
    int fd = open(argv[1], O_RDONLY);
    if (fd < 0) { perror("open"); return 1; }

    char buf[4096];
    ssize_t n;
    while ((n = read(fd, buf, sizeof buf)) > 0)
        write(1, buf, n);        /* write exactly what we read */

    if (n < 0) perror("read");
    close(fd);
    return 0;
}

open(argv[1], O_RDONLY) asks the kernel to open the named file for reading. On success the kernel returns a small non-negative integer — the file descriptor — which is your handle for every later call. A negative return means failure, so we check it and use perror to print a human-readable reason drawn from errno.

The read loop is the heart of it. Each read(fd, buf, sizeof buf) pulls up to 4096 bytes into our buffer and returns how many it actually got — which may be fewer than we asked for. A return of 0 means end-of-file, so the loop stops; a negative return means an error. For every chunk we immediately write(1, buf, n) to file descriptor 1, standard output, passing n — the exact count read, never the buffer size — so we never emit garbage past the real data.

Finally close(fd) hands the descriptor back to the kernel. Descriptors are a finite per-process resource, and a long-running server that forgets to close them will eventually hit its limit and start failing to open anything — a classic, painful production bug known as a file-descriptor leak.

A system call is far from free. Compared to a plain function call — a handful of nanoseconds that never leaves your code — a syscall costs hundreds of nanoseconds or more, because so much has to happen. The CPU switches privilege mode, the kernel saves and later restores your registers, and the detour disturbs the CPU’s caches and TLB, so your code often runs a little slower even after control returns. Any single syscall is cheap in human terms; a million of them in a hot loop are not.

This single fact explains a huge amount of real-world I/O design. If you call write() once per character, you pay the crossing cost per character. If you gather characters in a user-space buffer and call write() once per 4 KB, you pay it once per 4 KB — often a hundredfold fewer crossings. That is precisely why stdio buffers: printf does not trap on every call; it fills a buffer and flushes with one write when the buffer is full, at a newline on a terminal, or at exit.

The kernel even offers a shortcut for a few calls whose answer needs no privilege. Reading the clock with gettimeofday or clock_gettime is so common that Linux maps a tiny piece of kernel data and code — the vDSO — into every process, so those calls can be answered entirely in user space with no trap at all. It is the exception that proves the rule: the crossing is expensive enough that the kernel works hard to avoid it where it safely can.

Mode switch

The privilege change into and out of the kernel — the core unavoidable cost of a syscall.

Register save/restore

The kernel must preserve and restore your CPU state around its work, adding overhead.

Cache/TLB effects

Running kernel code evicts some of your cached data and address translations, slowing you afterwards.

Buffering

Batching many small operations into one syscall to amortise the crossing cost — what stdio does for you.

vDSO

A kernel-provided user-space shim that answers a few cheap calls (like gettimeofday) without trapping.

The boundary is invisible in normal running, but strace makes it visible. It runs a program and prints every system call it makes, with the arguments and the return value, so you can literally watch your process talk to the kernel. It is one of the most useful debugging tools on Linux, and knowing it well genuinely separates competent engineers from stuck ones.

$ strace ./mycat hello.txt
execve("./mycat", ["./mycat", "hello.txt"], 0x7ffd...) = 0
openat(AT_FDCWD, "hello.txt", O_RDONLY)  = 3
read(3, "hello, world\n", 4096)          = 13
write(1, "hello, world\n", 13)           = 13
read(3, "", 4096)                         = 0
close(3)                                  = 0
+++ exited with 0 +++

Read it top to bottom and the whole story is there. openat returns 3 — our file descriptor, since 0, 1, and 2 are already taken by stdin, stdout, and stderr. read returns 13, the byte count, and shows the data; write(1, ...) sends those same 13 bytes to standard output. The second read returns 0, which is end-of-file, so the loop ends and close(3) tidies up. This is exactly the C program from earlier, seen from the kernel’s side.

$ strace -c ./mycat big.txt >/dev/null
% time     calls  syscall
------ ---------  ----------------
 61.2       2048  read
 34.7       2048  write
  1.5          1  openat
  0.4          1  close

The -c summary is where strace becomes a performance tool. Here 4096 read/write calls dominate the time — a strong hint that a bigger buffer, or letting stdio buffer for us, would cut the syscall count and speed things up. When a program is mysteriously slow or hangs, strace often shows you the exact call it is stuck in, frozen right there on the last line.

The concept of a system call is universal, but the specifics are not. Every protected OS has a user/kernel boundary and a trap mechanism to cross it, yet the numbers, names, and conventions differ. On Linux, write is syscall number 1 and the interface is stable and documented — programs rely on the exact numbers. On Windows, applications call the documented Win32 API (functions like WriteFile), which sits on top of a lower, largely undocumented native NT syscall layer that Microsoft is free to change between releases.

What keeps portable code sane across this variety is POSIX — a standard that specifies the behaviour of calls like open, read, write, and fork as a contract, independent of how any one kernel implements them. Write to the POSIX interface and the same C source compiles and runs across Linux, macOS, and the BSDs, because each provides wrappers that honour the same contract even though their kernels differ underneath.

Linux syscalls

A stable, numbered, documented interface (write = 1); programs and libc depend on the exact numbers.

Windows Win32 vs native

Apps target the documented Win32 API; the underlying native NT syscalls are lower-level and may change between versions.

POSIX

A portable contract specifying how standard calls behave, so source compiles across many Unix-like systems.

Shared concept

Different names and numbers, one idea: a guarded trap from user mode into a privileged kernel.

Step back and the shape of Part A is now complete. An OS abstracts hardware and manages resources; a privileged kernel enforces the rules; and the system call is the single, guarded doorway user code uses to ask that kernel for anything real. You can trace a call from printf across the boundary and back, name its cost, and watch it live with strace. That is the whole boundary in your hands.