ZeroFS: A Single-Process, Log-Structured Filesystem Bridging S3, NFS, 9P, and NBD
ZeroFS is a single-process userspace filesystem that exposes S3-compatible object stores as POSIX filesystems via NFS and 9P, or as raw block devices over NBD. Built on a log-structured engine, it packs writes into compressed, encrypted immutable segments and supports advanced configurations like multi-region ZFS mirroring and high-availability standby failover.
Unified Multi-Protocol Architecture
ZeroFS runs as a single userspace process that presents S3-compatible buckets (including AWS S3, Google Cloud Storage, Azure Blob, and local disks) as NFS, 9P, and NBD endpoints. Because all three servers operate within a single process, they share a unified cache and bucket state.
- NFS: Mounts natively across Linux, macOS, Windows, and BSDs. It utilizes NFS `COMMIT` semantics, allowing `fsync` calls to return to the client before data is written to the remote object store.
- 9P: Provides strict durability guarantees where `fsync` returns only after data is committed to S3. Linux deployments can use a bundled non-root FUSE client (`zerofs mount`) that manages reconnections automatically.
- NBD: Exports raw block devices, supporting file system architectures like ZFS pools or VM boot drives. The server honors `FLUSH` and `FUA` commands, holding write barriers until the payload is written to stable storage.
Log-Structured Engine and Cryptographic Pipeline
The backend storage engine uses a log-structured design. Incoming writes are split into 32 KiB extents and packed into immutable segment objects of up to 256 MiB. Before uploading to the destination bucket, each extent passes through a pipeline of compression and encryption: 1. Compression: Data is compressed using either zstd or lz4. The engine supports changing codecs on the fly; read operations dynamically detect and unpack existing segments without requiring data migration. 2. Encryption: Every extent is encrypted using XChaCha20-Poly1305. The master data key is wrapped using a key derived via Argon2id from a user-provided password.
For read performance, ZeroFS implements a tiered caching strategy utilizing local memory and disk. Space reclamation is handled via TRIM support, which propagates client-side deletions down to S3 object reclamation.
High Availability and Replication Topology
The storage engine supports a single-writer, multi-reader topology. Any number of read-only replica instances can point to the same bucket; these replicas continuously sync with the writer's metadata and return `EROFS` on write attempts.
To mitigate write latency over long-haul connections, ZeroFS can deploy an active-standby configuration. Under this model, an optional standby node caches acked-but-unflushed writes. In the event of primary failure, the standby node can execute a failover within seconds. Additionally, the system supports named, point-in-time checkpoints that can be mounted as read-only volumes.
Testing and Validation Protocols
- POSIX Semantics: Over 8,662 `pjdfstest` cases are executed to verify permissions, ownership, symlinks, and rename operations.
- Concurrency and Stress: Parallel Linux kernel compilations (`make -j$(nproc)`) run across NFS, 9P, and FUSE mounts, alongside `stress-ng` suites testing dentry churn, wide directories, and UTimes.
- Filesystem Robustness: Standard `xfstests` suites and ZFS scrub cycles verify the integrity of block devices.
- Jepsen Analysis: The Jepsen `local-fs` suite runs random operation histories against 9P with crash injection. The high-availability suite tests leader/standby failovers by terminating and pausing nodes over a local S3 instance.
These protocol layers enable geo-distributed configurations. For example, three independent ZeroFS instances in different S3 regions can present local NBD block devices to a ZFS mirror pool. If a region goes offline, the ZFS pool degrades but remains operational, resilvering automatically when the member returns.