Real File Systems

Everything in the previous chapter was a toolbox: linked allocation, indexed allocation, the inode, the free-block bitmap, the journal. A real file system is just a specific, committed choice of which tools to use and how to lay them on the disk. Nothing new is invented here — the FAT chain you will see is the linked-allocation idea made real, the Unix inode is the indexed-allocation idea made real, and ext3’s journal is exactly the crash-recovery idea made real. What changes from one system to the next is which trade-off each designer decided to win.

Every file system answers the same short list of questions, and you can size up any of them by asking those questions in order. How does it find the blocks of a file (the allocation method)? Where does it keep metadata (a table, an inode, a record)? How does it track free space? How does it stay consistent after a crash (fsck, journal, or copy-on-write)? And what are its hard limits (biggest file, biggest volume, longest name)? Hold that checklist in your head and the rest of this chapter is really one questionnaire filled in five different ways.

Allocation method

How the file system records which blocks hold a file’s data — a linked table (FAT), an index block (inode), or an extent (a start block plus a run length).

Metadata store

Where per-file information (size, timestamps, permissions, block map) lives: a central table, a per-file inode, or a record in a master file.

Consistency strategy

How the file system recovers a coherent state after a crash: full scan (fsck), a journal of pending changes, or copy-on-write.

Hard limits

The ceilings baked into the on-disk format: maximum file size, maximum volume size, maximum filename length.

FAT is linked allocation with the links pulled out of the data blocks and gathered into one array at the front of the volume — the File Allocation Table it is named after. The table has one entry per cluster (a cluster is just a fixed group of sectors, the unit FAT allocates in). A directory entry records a file’s starting cluster; you look that cluster up in the table and its value is the number of the next cluster, which you look up again, and so on, following the chain until you hit a special end-of-chain marker. A zero entry means the cluster is free; another reserved value marks a bad cluster. That is the entire scheme, and its simplicity is exactly why it has outlived nearly everything else.

The number in the name is the width of a table entry, and it sets the ceilings. FAT12 (12-bit entries) was for floppies; FAT16 for early hard disks; FAT32 uses 28 usable bits per entry, so it can address enough clusters for volumes up to 2 TiB with typical sector sizes. But FAT32 stores each file’s size in a 32-bit field, which caps a single file at 4 GiB minus one byte — the limit that bites the moment you try to copy a 5 GB video onto a FAT32 stick. exFAT (2006) was Microsoft’s answer for flash: it keeps the table-of-clusters idea but widens the size fields so files and volumes are effectively unbounded for practical purposes, which is why SDXC cards ship formatted as exFAT.

The other famous FAT wart is the 8.3 filename: eight characters of name, three of extension, uppercase, from the original MS-DOS directory entry. Long filenames were bolted on later (the VFAT scheme) by hiding extra characters across several adjacent directory entries, kept backward-compatible with the old 8.3 slot. It is a hack, but a hack that has let a filename like MyHolidayVideo.mp4 live on a format designed in 1980.

# FAT32 — maximum compatibility, but 4 GiB per-file limit
$ sudo mkfs.vfat -F 32 /dev/sdb1

# exFAT — for large SD cards and files over 4 GiB
$ sudo mkfs.exfat /dev/sdb1

$ df -T /dev/sdb1
Filesystem     Type  1K-blocks  Used Available Use% Mounted on
/dev/sdb1      vfat   15630336     8  15630328   1% /media/usb

So why is FAT still everywhere in 2026 — USB sticks, SD cards, camera storage, the UEFI system partition your PC boots from — despite having no journaling, weak metadata, and slow chain-walking for large files? Because on removable media, universal support beats every other virtue. A FAT volume mounts read-write on Windows, macOS, Linux, cameras, TVs, and printers without a driver argument. Its very lack of features (no ownership, no ACLs, no journal to replay) is what makes it trivial to implement everywhere. For a chip you plug into a stranger’s device, "everyone can read it" wins.

The Unix file system from the 1970s (the V7 design) is the direct ancestor of almost every serious file system since, so it is worth seeing in its original form. The disk is divided into four regions laid out in order: a boot block at the very front, then a superblock describing the whole file system (its size, the location of the inode list, the free lists), then the inode list — a fixed array of equally-sized inode records allocated when the file system is created — and finally the large data-block region holding actual file contents.

This is indexed allocation in its purest form. Each inode holds a file’s metadata (type, owner, permissions, size, timestamps) and a small array of block pointers: a handful of direct pointers to data blocks, plus a single-indirect pointer (to a block full of more pointers), a double-indirect, and a triple-indirect. Small files need only the direct pointers and are found in one hop; huge files are still reachable by descending through the indirect blocks. Crucially, the inode number — not the name — is the file’s real identity, and directories are just files mapping names to inode numbers. Everything Unix does with hard links and mount points falls straight out of that one decision.

It worked, but the original layout had real problems that pushed the whole field forward. Inodes sat at the front of the disk while their data sat far away, so reading a file meant a long seek from inode to data and back — brutal on the spinning disks of the day. Free blocks were handed out wherever they fell, so files fragmented across the platter and grew slower over time. And recovery after a crash meant fsck scanning the entire inode list and block map to rebuild consistency, which on a large volume could take many minutes of downtime. The famous Berkeley Fast File System answered these by splitting the disk into cylinder groups — each with its own inodes and nearby data blocks — and using larger blocks, an idea you are about to see reappear under a different name.

Superblock

The master record describing the whole file system: total size, block size, and where the inode list and free lists live. Lose it and the volume is unreadable — so copies are kept.

Inode list

A fixed array of inode records set aside at format time. Its size caps how many files the volume can ever hold, regardless of free space.

Indirect blocks

Blocks full of pointers, referenced by an inode to reach files far larger than its handful of direct pointers could map alone.

fsck

The file-system check that walks all metadata to restore consistency after an unclean shutdown — correct, but slow on large disks, which motivated journaling.

ext2 is the classic Unix file system, modernised. Its headline idea is the block group: the volume is chopped into many groups, and each group carries its own inode table, its own bitmaps for free inodes and free blocks, and its own data area — exactly the Fast File System’s cylinder-group trick, keeping a file’s inode and data close together so the disk head barely moves. Bitmaps replaced the old free lists for fast, compact free-space tracking. ext2 was fast and clean, but it had the original sin: an unclean shutdown meant a full fsck.

ext3’s one big addition was journaling. Before changing the real metadata, ext3 first writes a description of the intended change to a dedicated journal area, then applies it, then marks the journal entry done. After a crash the system only has to replay (or discard) the small journal instead of scanning the whole disk — recovery drops from minutes to seconds. This is the same write-ahead idea databases use for durability, and it is the single reason Linux servers stopped dreading power loss. ext3 was otherwise deliberately identical to ext2 on disk, so the upgrade was painless.

ext4 is where the limits and performance were rebuilt for modern hardware. The biggest change is extents: instead of listing thousands of individual block pointers for a large file, an extent records "start at block X, run for N blocks" — one entry for a whole contiguous stretch. That shrinks metadata dramatically and cuts fragmentation for big files like videos and database tables. ext4 also widened block addressing to lift limits to a 16 TiB maximum file and a 1 EiB volume, indexed large directories with an internal tree (htree) so lookups stay fast in folders with millions of entries, and added delayed allocation — holding new data in memory and choosing its blocks only at flush time, so it can pick a good contiguous run in one shot.

$ sudo mkfs.ext4 /dev/nvme0n1p2
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Creating journal (131072 blocks): done

$ df -T /              # what is my root actually running?
Filesystem     Type 1K-blocks     Used Available Use% Mounted on
/dev/nvme0n1p2 ext4 102400000 41231884  55892116  43% /

Put together, ext4 is the default general-purpose file system on Linux — the one under most cloud VMs, containers, and databases — precisely because it is boring in the best way: mature, well-understood, quick to check, with predictable performance and no surprises. When you spin up a Linux server and do not choose a file system, this is almost certainly what you get, and for the vast majority of workloads it is the right call.

NTFS, the file system of the Windows world, takes the inode idea and pushes it to an extreme: everything is a record in one big table called the Master File Table (MFT). Every file and directory gets an MFT record, roughly a kilobyte each. Even the file system’s own bookkeeping — the MFT itself, the free-space bitmap, the journal, the root directory — is stored as ordinary files with their own MFT records. There is no separate "inode region" versus "data region" in the Unix sense; there is the MFT, and it describes everything, including itself.

The twist that makes NTFS more than "inodes with a different name" is that a file is a bag of typed attributes rather than a fixed record. An MFT record holds attributes like $STANDARD_INFORMATION (timestamps and flags), $FILE_NAME, $SECURITY_DESCRIPTOR (the access-control list), and $DATA (the contents). The elegant part: if a file is small enough, its $DATA attribute lives resident — right inside the MFT record — so a tiny file needs no data blocks at all and is read in the same fetch as its metadata. When the data outgrows the record, $DATA becomes non-resident and points to runs of clusters on disk — NTFS’s version of extents. A file can even have several named $DATA attributes, which is the mechanism behind alternate data streams.

On top of that record model, NTFS layers the features Windows expects: journaling for crash consistency (via a log file, the same write-ahead idea as ext3), rich per-file access-control lists rather than Unix’s nine permission bits, and transparent per-file compression and encryption (EFS) implemented as properties of the $DATA attribute. Directories are kept as B-trees for fast lookup in huge folders. The result is feature-rich and tightly integrated with Windows security — and correspondingly more complex than the lean Unix model, which is one reason full read-write NTFS support on other operating systems took so long to become trustworthy.

Master File Table (MFT)

The central array of ~1 KB records, one per file or directory, that describes every object on the volume — including the MFT itself.

Attributes

The typed parts of a file record ($STANDARD_INFORMATION, $FILE_NAME, $SECURITY_DESCRIPTOR, $DATA). A file is defined by its set of attributes, not a fixed layout.

Resident vs non-resident

Small files keep their $DATA inside the MFT record (resident); larger files store it in cluster runs on disk (non-resident) — NTFS’s equivalent of extents.

ACL

An access-control list attached to each file, far more expressive than Unix’s owner/group/other bits — the basis of Windows file security.

ISO 9660 is the file system on data CD-ROMs, and it is a lovely illustration of how a constraint reshapes a design. Optical media is written once and then only read, so the whole apparatus we have been building — free-space bitmaps, allocation policies, journals, defragmentation — is simply unnecessary. There is nothing to allocate over time and no crash to recover from. So ISO 9660 stores each file as a single contiguous extent (a start sector and a length) laid out back to back, which is the allocation method that is normally impractical on a read-write disk because files grow and leave holes. On a disc, contiguity is free, and it makes reading a file a single uninterrupted sweep of the laser.

The original standard was strict and portable to a fault — uppercase 8.3-style names, shallow directory nesting — so extensions grew on top of it: Joliet added Unicode long filenames for Windows, Rock Ridge added Unix permissions and longer names and symbolic links, and El Torito made discs bootable. (DVDs and Blu-ray largely moved on to UDF, but the same read-only, contiguous philosophy carried over.) The details matter less than the shape: when a medium cannot be rewritten, the file system gets radically simpler because most of its hard problems disappear.

That shape is not a museum piece — it keeps coming back wherever immutability is chosen on purpose. A container image is built as a stack of read-only layers, each layer a fixed set of files that is never edited in place; the running container adds one thin writable layer on top and uses copy-on-write to modify anything from the layers below. That is the ISO 9660 bargain rediscovered: make the base immutable and you gain shareability, cacheability, and trivial integrity checking. The same instinct drives append-only logs, write-once object storage, and the copy-on-write file systems in the next section.

Lined up side by side, the file systems in this chapter are the five-question checklist answered five ways. Read the table below not as facts to memorise but as a map of trade-offs: each row is a pressure, and each column made a different peace with it.

                 FAT32 / exFAT     ext4            NTFS            ZFS / Btrfs
  allocation     cluster chain     extents         cluster runs    copy-on-write
  consistency    none              journal         journal (log)   CoW + checksums
  max file       4 GiB / ~16 EiB   16 TiB          8 PB            ~16 EiB
  max volume     2 TiB / ~128 PB   1 EiB           8 PB            ~256+ ZiB
  permissions    none              Unix bits       rich ACLs       Unix bits + ACLs
  snapshots      no                no              limited (VSS)   yes, native
  best use       removable media   Linux default   Windows         NAS, storage pools

The rightmost column is where file systems are still evolving. ZFS and Btrfs are copy-on-write file systems: they never overwrite a block in place. A change is written to a fresh block, and the pointers above it are updated to point at the new version, rippling up to the root — so the on-disk state is always consistent without a separate journal, because the old tree stays valid until the new root is committed. That single mechanism gives you nearly free snapshots (just keep the old root and its blocks around), end-to-end checksums that detect silent data corruption, and pooled storage that spans many disks. The cost is more metadata churn and a tendency to fragment, which is why they shine on capacity-oriented storage servers and NAS boxes more than on latency-critical single disks.

This is not academic — the file system underneath your software is a real performance and correctness decision. Databases care because durability lives at this layer: Postgres and MySQL depend on the file system honestly flushing to disk on fsync, and a mismatch between what the file system promises and what the hardware does has caused real data-loss bugs. Containers care because their whole layering model rides on file-system features — overlay filesystems and CoW snapshots are what make image layers and fast container startup possible. And anyone running storage at scale weighs checksums and snapshots (ZFS/Btrfs) against the boring reliability of ext4. Choosing a file system is choosing which failure modes you are willing to live with.

That closes Part E. We started from the bare file abstraction, worked through directories, allocation, inodes, and journaling as ideas, and have now seen those ideas made real in the file systems you actually touch — with every difference traceable to a constraint rather than a whim. You can now open a disk, run lsblk -f or df -T, and read the result: not just which file system is mounted, but why someone chose it and what it will and will not do for you.