Preventing PostgreSQL Database Crashes with Strict Linux Memory Overcommit

Operating PostgreSQL under default Linux memory overcommit configurations exposes systems to catastrophic database-wide restarts when the OOM killer terminates backend processes. Transitioning to strict memory overcommit forces graceful allocation failures via ENOMEM, protecting shared memory integrity and preventing connection drops.

Postgres Shared Memory and OOM Vulnerability

PostgreSQL utilizes a multi-process architecture where the master supervisor process, known as the postmaster, forks an individual backend process for each incoming client connection. These backend processes rely on shared memory segments to maintain vital state, including shared buffers, write-ahead log (WAL) buffers, and lock tables.

Because the operating system does not provide transactional guarantees for shared memory, an unexpected process termination can cause catastrophic failures. If the Linux Out-of-Memory (OOM) killer terminates a backend process while it is writing to a shared memory segment, the segment can be left in an inconsistent or corrupted state.

To prevent this corruption from propagating to persistent storage, the postmaster is designed to assume the worst when any child process terminates abnormally. Upon detecting a child process crash, the postmaster immediately terminates all other active backend processes, aborts all in-flight transactions, and drops every active client connection. On restart, the database must run through crash recovery, replaying WAL files to restore consistency. Under high write volumes, this recovery phase can cause prolonged system outages.

Analyzing Linux Overcommit Modes

The Linux kernel offers three virtual memory overcommit policies configured via the `vm.overcommit_memory` sysctl knob. These modes dictate how the kernel handles memory allocation requests made via system calls such as `malloc()` and `mmap()`:

• Mode 0 (Heuristic): This is the system default. The kernel permits general overcommitting but rejects individual allocation requests that are obviously larger than the system's total capacity (reclaimable page cache, slab, free memory, and swap).
• Mode 1 (Always): The kernel accepts all allocation requests regardless of physical memory limits. If processes subsequently attempt to write to (fault in) more physical pages than the hardware can provide, the kernel relies on the OOM killer to reclaim memory by killing processes.
• Mode 2 (Strict): The kernel monitors total committed virtual memory across all active processes using the `Committed_AS` metric. It enforces a strict upper limit called `CommitLimit`. Any allocation request that would cause `Committed_AS` to exceed `CommitLimit` is immediately blocked and returns an `ENOMEM` error.

When operating under Mode 2, PostgreSQL handles `ENOMEM` errors gracefully. If a backend process cannot allocate memory, it reports the error to the client, aborts the specific active transaction, and continues running. The postmaster process and all other active client connections remain unaffected, turning a potential system-wide outage into a routine, isolated transaction error.

Calculating the Commit Limit

In strict overcommit mode, the kernel determines the maximum allocatable virtual memory using two distinct configurations: `overcommit_kbytes` and `overcommit_ratio`.

If `overcommit_kbytes` is explicitly defined, the calculation is:

CommitLimit = overcommit_kbytes + swap

If `overcommit_kbytes` is not configured, the kernel falls back to `overcommit_ratio` and calculates the limit as:

CommitLimit = (overcommit_ratio / 100) * available_memory + swap

This strict approach is ideal for dedicated database hosts where the resource demands of PostgreSQL and its associated sidecar processes are well-understood. It converts unpredictable, late-stage OOM killer actions into early, predictable failures.

Operational Challenges and Kernel Constraints

While strict overcommit provides strong stability guarantees, operating it at scale introduces unique challenges. For instance, infrastructure teams have had to navigate system quirks, including a notable three-character kernel bug (also referred to as a one-character bug) that forced the temporary disabling of strict overcommit across server fleets.

Additionally, operators must carefully tune the commit limits. Common heuristics for dedicated database virtual machines involve setting the commit ratio to 80% or applying a target ratio supplemented by a static memory buffer (such as +2 GB) to accommodate the baseline virtual memory footprint of essential system sidecars.