What fsync actually promises

When write() returns, your bytes are in the page cache, not on the disk. That gap is where most “durable” storage bugs live, and it is wider than the man page suggests.¹1O_DIRECT skips the page cache, but not the drive’s own volatile cache. It changes where the gap is, not whether there is one.

int fd = open("log", O_WRONLY | O_APPEND);
write(fd, buf, n);   /* copied into the page cache */
fsync(fd);           /* now it's on stable storage */

The rest of this post walks through what each layer guarantees, from the syscall down to the drive.

Four places a byte can be

A successful write() moves data from the first box to the second. The kernel writes dirty pages back to the device later, on its own schedule. A drive with a volatile write cache can acknowledge that write while the data still sits in its cache.

fsync(fd) pushes the file’s dirty pages and metadata to the device and then asks the device to flush its cache. When it returns 0, the data has reached stable storage, provided the drive honours flushes.²2Most do. Some consumer drives have been caught acknowledging flushes they never performed. There is nothing userspace can do about that except buy better drives.

The directory is a file too

Creating a file and calling fsync on it makes its contents durable, but not its name. The directory entry lives in the parent directory, which has its own dirty state. The safe way to replace a file atomically:

1. Write the new contents to a temporary file in the same directory.
2. fsync the temporary file.
3. rename it over the old name.
4. fsync the directory.

Skip step 2 and a crash can leave the new name pointing at an empty file. Skip step 4 and the rename itself may not survive.

When fsync fails

The worst case is an fsync that returns an error. On Linux, a failed writeback can leave the affected pages marked clean, so the data they held is gone from the cache. A second fsync can then return success because there is nothing left to flush.³3This is what PostgreSQL ran into in 2018. Its fix was to stop retrying and crash instead, then recover from its own write-ahead log.

So an fsync error is not something to retry. Treat it as data loss: stop accepting writes, then rebuild state from something you know is durable.