Files & Directories

Strip away every convenience and a file is astonishingly simple: it is a named sequence of bytes that survives after the program that made it exits, and after the power goes off. That is the whole promise. Byte 0, byte 1, byte 2, up to some length — an array that lives on durable storage and has a name you can find it by later. Persistence is the entire point; a variable in memory vanishes when your process ends, but a file is still there tomorrow.

To the operating system, the contents are just bytes. It attaches no meaning to them. A file holding JPEG image data and a file holding UTF-8 text are, at the file-system level, identical kinds of things: a length and a run of bytes. The interpretation — "these bytes are a picture", "these are a Python script" — lives entirely in the programs that read the file, never in the file system itself. This is why you can open a .png in a text editor (and see garbage) or rename a .zip to .jpg (and confuse an image viewer): nothing checked, because there was nothing to check.

That point deserves emphasis because so many beginners believe otherwise. On Unix, a file extension is a convention among humans and applications, not a fact the kernel enforces. The kernel does not stop you renaming server.log to server.mp3, and it will not refuse to run a program because it lacks a .exe suffix — on Unix, what makes a file executable is a permission bit, which we will meet later, not its name. Tools that need to know a file’s real type inspect its leading bytes (its "magic number"), which is exactly what the file command does.

File

A named, persistent, ordered sequence of bytes stored on durable media; the OS treats the contents as opaque.

Persistence

The defining property: a file outlives the process that created it and survives reboots, unlike in-memory data.

Extension

A suffix like .txt or .jpg — a hint to humans and applications about content type, not something the kernel enforces.

Magic number

A few identifying bytes at the start of a file that let tools like file detect its true type regardless of name.

$ cp portrait.jpg mystery.txt      # rename a JPEG to look like text
$ file mystery.txt
mystery.txt: JPEG image data, JFIF standard 1.01, 1920x1080

$ file /bin/ls
/bin/ls: ELF 64-bit LSB pie executable, x86-64, dynamically linked

The verbs you use on a file are few and they have barely changed in fifty years: create it, open it, read from it, write to it, seek to a position, and close it. open is the pivotal one. You hand the kernel a path and say how you intend to use the file — read-only, write-only, read-write, create-if-missing, truncate, append — and if it agrees, it hands back a small non-negative integer: the file descriptor, or fd. From that moment you never mention the path again; the fd is your handle for every following read, write, and seek.

Behind that little integer stand three linked structures, and understanding them clears up a surprising amount of confusion. The fd is just an index into your process’s private file-descriptor table. Each used slot points to an entry in a system-wide open-file table, and that entry holds the two things an open file needs to remember: the access mode and the current file offset — the byte position where your next read or write will land. That open-file entry, in turn, points to the file’s inode, the single in-kernel object describing the actual file.

Why three layers instead of one? Because they capture three genuinely different things. The fd is per-process and cheap, so descriptors 0, 1, and 2 (standard input, output, and error) mean "my input/output", not any global file. The offset lives in the open-file entry, not the inode, so two independent opens of the same file get independent cursors — one program can be reading near the start while another reads near the end, each advancing its own position. And the inode is shared, so both see the same underlying bytes. When you read, the kernel starts at the offset, copies bytes toward your buffer, and advances the offset by the number of bytes actually transferred; lseek just moves that offset without transferring anything, which is how you jump around inside a file.

#include <fcntl.h>    /* open, O_* flags */
#include <unistd.h>  /* read, write, lseek, close */

int fd = open("notes.txt", O_RDONLY);   /* fd = 3, offset starts at 0 */
if (fd < 0) { perror("open"); return 1; }

char buf[64];
ssize_t n = read(fd, buf, sizeof buf);  /* reads n bytes, offset += n */

lseek(fd, 0, SEEK_SET);                 /* rewind: offset back to 0 */
n = read(fd, buf, sizeof buf);          /* reads the same bytes again */

close(fd);                              /* hand the descriptor back */

File descriptor (fd)

A small integer, private to a process, indexing its open-file-descriptor table; your handle to an open file.

Open-file table entry

A system-wide record for one open instance, holding the access mode and the current byte offset.

File offset

The cursor: the byte position where the next read or write happens; advanced automatically, moved manually by lseek.

Standard streams

fds 0, 1, 2 — stdin, stdout, stderr — opened for every process before your code runs.

Every file carries a bundle of information about itself that is not part of its contents — its metadata. On a Unix file system this lives in a compact on-disk record called the inode (index node). The inode is the file, as far as the kernel is concerned: give it an inode and it knows the file’s size, who owns it, what may be done to it, when it was last touched, how many names point to it, and where on the disk its data blocks live.

Look closely at that list and notice the one thing conspicuously absent: the file’s name. The inode does not store the name. This is the single most surprising fact about Unix files, and it is the key that unlocks the next three sections. The name lives in a directory, which maps a human-readable name to an inode number. The inode holds everything about the file except what it is called — which is precisely why a file can have several names at once, or be renamed, or have its name deleted while the file itself lives on.

The inode also solves the "where are the bytes" problem. A small file’s data fits in a handful of blocks, and the inode holds a dozen or so direct pointers straight to them. Larger files need more pointers than fit in a fixed-size inode, so the classic design adds indirection: a single-indirect pointer aims at a block that is itself full of pointers, a double-indirect pointer aims at a block of blocks-of-pointers, and so on. That is implementation detail we unpack next chapter; here the point is simply that the inode is the map from "this file" to "these scattered blocks".

You can read most of the inode from the command line. stat prints it in full; ls -l is a friendlier view of the same fields. Note the three timestamps, which trip people up constantly: atime is the last time the data was read, mtime is the last time the data was modified, and ctime is the last time the inode itself changed (a permission change bumps ctime but not mtime). None of them is a "creation time" in the traditional sense.

$ stat report.pdf
  File: report.pdf
  Size: 48213      Blocks: 96      IO Block: 4096   regular file
Device: 8,1        Inode: 1310726    Links: 1
Access: (0644/-rw-r--r--)  Uid: (1000/ hitesh)  Gid: (1000/ hitesh)
Modify: 2026-07-09 11:02:14   Change: 2026-07-09 11:02:14

$ ls -l report.pdf
-rw-r--r-- 1 hitesh hitesh 48213 Jul  9 11:02 report.pdf
#  ^perms   ^links ^owner ^group ^size  ^mtime      ^name (from the directory)

Inode

The on-disk record holding all of a file’s metadata and the pointers to its data blocks — everything except its name.

Inode number

The unique identifier of a file within a file system; what a directory entry actually references.

Block pointers

The inode’s map from file to data: direct pointers for small files, single/double/triple indirect for large ones.

atime / mtime / ctime

Last data access, last data modification, and last inode (metadata) change — note: not a creation timestamp.

If the inode has no name, where do names live? In directories. And here is the elegant part: a directory is itself a file. Its contents are not free-form bytes but a structured table of directory entries, each one pairing a name with an inode number. That is all a directory fundamentally is — a lookup table from names to inodes. "The directory contains the file" is a convenient fiction; what it really contains is the file’s name and a pointer (the inode number) to where the file’s inode lives.

Because directories can list other directories, they nest into the familiar tree. At the very top sits a single root directory, written /, with no parent. Every file on a Unix system is reachable by starting at the root and walking down through named directories until you reach the target — that walk is called path resolution, and the kernel does it one component at a time. To resolve /home/hitesh/notes.txt, it reads the root directory to find the inode for home, reads that directory to find hitesh, reads that to find notes.txt, and lands on its inode. Each step is a directory lookup; the slashes are the joints.

A path starting with / is absolute — it is resolved from the root and means the same thing no matter where you are. A path that does not is relative — it is resolved starting from the process’s current working directory. Two special entries make relative paths work, and they exist in every directory: "." refers to the directory itself, and ".." refers to its parent. So ../config reaches up one level and then into config, and ./run.sh insists on the run.sh right here. Those two entries are real directory entries with real inode numbers — which, incidentally, is why a brand-new empty directory already has a link count of 2 (its name in its parent, plus its own ".").

$ ls -ai /home/hitesh
     2 ..        81 .        96 projects   1310726 notes.txt
#  inode#      names --------------------------->

$ pwd                 # where relative paths start from
/home/hitesh
$ cat ./notes.txt     # "." = here;  resolves to inode 1310726
$ cat ../             # ".." = parent (/home)

Directory

A file whose contents are a table of entries, each mapping a name to an inode number.

Path resolution

Walking the tree one component at a time, doing a directory lookup at each slash, until the target inode is found.

Absolute vs relative

An absolute path starts at the root /; a relative path starts at the process’s current working directory.

. and ..

Real entries in every directory: "." is the directory itself, ".." is its parent — the machinery behind relative paths.

Now that names and inodes are separate things, a file having more than one name is not a paradox — it is just two directory entries pointing at the same inode number. That is a hard link. When you run ln a.txt b.txt, you are not copying the file; you are adding a second name for the exact same inode. Both names are equal, first-class, and indistinguishable — there is no "original". The inode keeps a link count of how many names point to it, and creating the hard link bumps it from 1 to 2.

This reframes what "deleting a file" means. The unlink system call — what rm uses — does not necessarily destroy any data. It removes one name from its directory and decrements the inode’s link count. Only when the link count hits zero (and no process still holds the file open) does the file system actually free the inode and its data blocks. So deleting b.txt while a.txt still points at the same inode simply drops the count back to 1 and leaves every byte intact. "Delete" is really "remove a name; reclaim the file only when its last name is gone".

A symbolic link (or symlink, or soft link) is a completely different animal. It is its own tiny file, with its own inode, whose entire contents are a text string: a path to something else. It does not point at an inode; it points at a name. When path resolution hits a symlink, it reads that stored path and continues resolving from there. Create one with ln -s. Because a symlink stores a path rather than an inode, it can cross file-system boundaries and point at directories — things hard links generally cannot do — but it pays for that flexibility with fragility: delete or move the target and the symlink still exists, now pointing at nothing. That is a dangling (broken) link.

$ echo "hello" > a.txt
$ ln a.txt b.txt          # hard link: second name for the same inode
$ ls -li a.txt b.txt
1310726 -rw-r--r-- 2 hitesh hitesh 6 Jul  9 12:00 a.txt
1310726 -rw-r--r-- 2 hitesh hitesh 6 Jul  9 12:00 b.txt
#  ^same inode number     ^link count = 2

$ ln -s a.txt link.txt    # symlink: a little file holding the path "a.txt"
$ ls -li link.txt
1310801 lrwxrwxrwx 1 hitesh hitesh 5 Jul  9 12:01 link.txt -> a.txt
#  ^its OWN inode  ^type l = symlink            ^stored target path

$ rm a.txt                # remove the original name
$ cat b.txt               # hard link still works — data intact
hello
$ cat link.txt            # symlink now dangles — target is gone
cat: link.txt: No such file or directory

Hard link

An additional directory entry pointing at the same inode; all names are equal, and the inode’s link count tracks how many exist.

Link count

The number of hard links (names) to an inode; the file’s data is freed only when it reaches zero.

unlink

The delete operation: removes one name and decrements the link count — data is reclaimed only at zero.

Symbolic link

A separate file whose contents are a path to another name; flexible across file systems but breaks if the target moves.

Because a Unix system is shared, every file needs to answer "who may do what to me". The model is deliberately small. Each file has an owning user and an owning group, both recorded in the inode, and three classes of accessor: the owner (user), members of the file’s group, and everyone else (other). For each class there are three permission bits — read, write, and execute — giving the nine bits you see in ls -l as three rwx triads. A dash means the bit is off.

Those nine bits are almost always written in octal, because each rwx triad is exactly three bits and so maps to one octal digit: read is 4, write is 2, execute is 1, and you add them. rwx is 4+2+1 = 7; r-x is 4+0+1 = 5; r-- is 4 = 4. So the ubiquitous 755 means "owner may read/write/execute, group and other may read/execute", and 644 means "owner read/write, everyone else read-only". Once you can convert a triad to a digit in your head, the numbers chmod wants stop being magic.

The subtle part is what the bits mean on a directory, because a directory is a file of a different kind. On a directory, read means "list the names inside", write means "create, rename, or delete entries" (note: this governs whether you can delete a file, not the file’s own write bit — deletion changes the directory, not the file), and execute means "traverse into it / use it in a path". That last one catches everyone eventually: you can hold a valid path to a file you are allowed to read, and still be denied, because you lack execute (x) on a directory somewhere along the path and so cannot walk through it.

Two special bits round it out. Setuid on an executable makes it run with the privileges of the file’s owner rather than the user who launched it — that is how an ordinary user running /usr/bin/passwd can update the root-owned password file, briefly and under a tightly controlled program. And the sticky bit on a shared directory such as /tmp restricts deletion so that even though everyone can write there, you may only remove your own files, not other people’s. You set all of this with chmod (change mode) and change ownership with chown.

$ chmod 755 deploy.sh     # rwx / r-x / r-x  — owner full, others read+run
$ chmod 644 config.json   # rw- / r-- / r--  — owner writes, others read
$ chmod u+x,go-w script   # symbolic form: add exec for user, drop write for group+other

$ chown hitesh:developers report.pdf   # set owner=hitesh, group=developers

$ ls -l /usr/bin/passwd   /tmp
-rwsr-xr-x 1 root root  ... /usr/bin/passwd   # the s = setuid: runs as root
drwxrwxrwt 1 root root  ... /tmp              # the t = sticky: delete only your own

Owner / group / other

The three accessor classes; each file has an owning user and group recorded in its inode.

rwx bits

Read (4), write (2), execute (1) per class — nine bits total, usually written as a three-digit octal like 755.

Directory permissions

On a directory: r = list names, w = add/remove entries, x = traverse into it. x is required to use it in any path.

setuid / sticky bit

setuid runs a program as its owner (e.g. passwd as root); the sticky bit on /tmp lets you delete only your own files.

A running machine usually has more than one file system: the root disk, maybe a second drive, a USB stick, a network share. The question is how they present themselves to you. Unix makes a bold choice: there is exactly one directory tree, rooted at /, and every file system is grafted into it at some directory. That grafting is called mounting, and the directory where a file system is attached is its mount point. After you mount the USB stick at /mnt/usb, walking into /mnt/usb transparently walks into the stick’s own file system — the join is invisible in normal use. cd across it and you would never know you had crossed onto different hardware.

This is a genuinely different philosophy from Windows, and the contrast is clarifying. Windows exposes each file system under its own drive letter — C:\, D:\, E:\ — so there are several independent roots and a path names its device up front. Unix hides the devices behind one namespace; the same absolute path always means the same place, and where the bytes physically live (which disk, which partition, even which machine for a network mount) is an administrative detail settled at mount time, not something baked into every path.

The unified tree has a beautiful consequence: not everything mounted into it has to be backed by a disk at all. Linux exposes kernel and process information through virtual file systems that generate their contents on the fly. /proc presents live process and kernel state as files — /proc/self/status describes the calling process, /proc/meminfo reports memory — and /sys exposes devices and kernel tunables the same way. These "files" occupy no disk blocks; reading one runs kernel code that produces the answer on demand. It is the "everything is a file" idea taken to its logical end: even the kernel’s own internals are browsable, greppable files.

$ mount /dev/sdb1 /mnt/usb        # graft the USB stick onto the tree
$ ls /mnt/usb                    # now just another directory
photos  backup.tar  notes.txt

$ mount | grep sdb1              # see where a device is attached
/dev/sdb1 on /mnt/usb type ext4 (rw,relatime)

$ cat /proc/self/status | head -3   # a "file" generated by the kernel
Name:   cat
State:  R (running)
Pid:    48213

Mounting

Attaching a file system into the directory tree at a chosen directory, so its contents appear there.

Mount point

The directory where a file system is grafted on; traversing it crosses transparently into that file system.

Unified namespace

The Unix model of one tree under a single root /, versus Windows’ separate per-device drive letters (C:, D:).

Virtual file system

A mount like /proc or /sys whose files are generated by the kernel on demand and occupy no disk blocks.

The file abstraction is clean, but real systems break in a handful of characteristic ways, and every one of them follows straight from what we have covered. Start with descriptors. They are a finite, per-process resource capped by a limit you can see with ulimit -n. Code that opens files (or sockets — sockets are file descriptors too) in a loop and forgets to close them leaks descriptors until the process hits the ceiling and every new open fails with "too many open files" (EMFILE). The fix is discipline: close what you open, and in higher-level languages lean on the construct that closes for you — Python’s with, Go’s defer, Java’s try-with-resources.

Next, safe writes. A tempting but dangerous pattern is to open your config file, truncate it, and write the new contents — because if the process crashes or the disk fills midway, you are left with a half-written, corrupt file and the old good version is already gone. The robust idiom exploits something we saw earlier: rename within a file system is atomic. So write the new contents to a temporary file, flush and fsync it to durable storage, then rename it over the target in one step. A reader either sees the entire old file or the entire new one — never a torn mixture. This "write-temp-then-rename" dance is how editors, package managers, and databases update files safely.

Two more sharp edges. When several writers share one file — the classic case is multiple processes appending to a log — opening with O_APPEND makes each write atomically seek to the true current end before writing, so lines never overwrite each other; without it, each writer trusts its own stale offset and they clobber one another. And when you truly need exclusive access, that is what file locks (flock, or fcntl byte-range locks) are for. Finally, mind the boring traps that cause real outages: text-versus-binary and newline handling differs across platforms, and paths are hostile — they contain spaces, Unicode, and "../" sequences, so never build a path by string-concatenation of untrusted input, or you invite a path-traversal escape out of the directory you meant to confine things to.

Which brings us to the punchline for engineers: the database you run every day is, at bottom, a small set of very large files. Postgres keeps a data directory of table and index files plus a write-ahead log; SQLite is famously a single file. The database does not get special storage from the OS — it gets the same open/read/write/fsync/rename primitives you just learned, and layers its own page structure, indexing, and crash-recovery on top of that flat byte-stream. Every durability guarantee a database makes ultimately cashes out as a well-chosen sequence of these file operations, with fsync at the critical moments to force bytes past the OS cache onto the disk.

int fd = open("config.json.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, data, len);      /* write the full new contents */
fsync(fd);                 /* force it to durable storage first */
close(fd);

rename("config.json.tmp", "config.json");
/* atomic: a reader sees the whole old file or the whole new file, */
/* never a half-written mix — even if we crash right here.        */

fd limit / leak

Descriptors are capped per process (ulimit -n); failing to close them leaks until open fails with EMFILE.

Atomic rename

Renaming within a file system is all-or-nothing, so write-temp-then-rename gives crash-safe file updates.

O_APPEND & locking

O_APPEND makes concurrent appends land at the true end atomically; flock/fcntl locks give exclusive access.

Databases as files

A DB is large files (tables, indexes, WAL) using the same open/read/write/fsync/rename primitives, with structure on top.