Disk I/O and Durability for System Design: Page Cache, fsync and Why "Saved" Isn't Saved

In 2018, Postgres developers found that Linux could quietly drop a failed disk write, and a later fsync would still report success. The database thought your data was safe. It wasn't. That bug, nicknamed fsyncgate, is the best way I know to show that "saved" means less than most engineers think.

For this blog, we'll stay on Linux and on a single machine. Networks and replicas come later, in Reads and writes.

01"Saved" has three meanings

Think about sending a letter. You write it, drop it in the office out-tray for a clerk to collect whenever they pass, and later a courier gets a signature at the other end. "Is it sent?" depends on which step you mean. Only the signed delivery survives the office burning down.

Files work the same way. Some words first:

• Page cache: RAM the kernel uses to hold file data.
• Dirty page: a page of file data changed in RAM but not yet written to the disk.
• write(): the system call that hands bytes to the kernel.
• fsync(): asks the kernel to push a file's data and metadata to the device and wait. fdatasync() skips metadata that isn't needed to read the data back.
• Durability: once the system says "saved", the data survives a crash or power cut.
• Journal: a small log the filesystem writes first, so it can repair itself after a crash.

The PostgreSQL reliability docs list the caches your bytes pass on the way down: the OS cache, the disk controller's cache and the drive's own cache. Here is the stack.

02Disks 101: HDD vs SSD, sequential vs random

A hard disk drive (HDD) is a spinning platter with a head on a moving arm, like a record player. Moving the arm to a new spot is a seek, about 8-10 ms (Norvig's table says 8 ms). Reading 1 MB in a straight line takes about 20 ms, so by our rough arithmetic one seek costs as much as reading 0.4 MB.

A solid-state drive (SSD) has no moving parts. Colin Scott's latency model puts an SSD random read at about 16 µs. By our own rough arithmetic, that's roughly 500x faster than an 8 ms seek. But as Emmanuel Goossaert explains, flash can't overwrite a small piece in place:

• It writes in pages but erases only in much bigger blocks.
• A flash translation layer (FTL) inside the drive remaps every write to a fresh page.
• Garbage collection copies live pages out of a block so it can be erased; wear levelling spreads writes evenly. That copying is write amplification: one write from you becomes several inside the drive.

Side by side:

On the Kafka team's disk array, linear writes ran at "about 600MB/sec" and random writes at "about 100kB/sec". Appends beat scattered small writes, even on SSDs.

03What write() and fsync() really do

write() copies your bytes into the page cache, marks those pages dirty and returns. The disk hasn't been touched. Kernel flusher threads write dirty pages back later, on rules in the kernel's vm settings such as dirty_expire_centisecs (how old dirty data may get) and dirty_ratio (how much dirty memory before a writer must flush itself). Pull the plug before then, and the data is gone.

That makes the page cache a write-back cache: it says "done" first and writes to the slower layer later. Kafka leans into this on purpose, letting the OS use "all free memory" as disk cache instead of caching inside the JVM.

fsync() is the courier's signature. Per the fsync(2) man page, it flushes the file's data and metadata to the device and blocks until the device reports it's done. But there's a trap in the same page: fsync "does not necessarily ensure" that the directory entry (the file's name) is on disk. A new or renamed file needs an fsync on its directory too.

So the safe way to replace a file has four steps: write a temp file, fsync it, rename it over the old one, then fsync the directory. Here's the sequence.

And even fsync has a dark corner. In 2018, Postgres learned that a failed background write could leave fsync returning success with the data already gone. That's fsyncgate, and it changed how serious databases handle errors.