Thread

RFC: PostgreSQL Storage I/O Transformation Hooks

Henson Choi <assam258@gmail.com> — 2025-12-28T07:49:54Z

RFC: PostgreSQL Storage I/O Transformation Hooks Infrastructure for a
Technical Protocol Between RDBMS Core and Data Security Experts

*Author:* Henson Choi assam258@gmail.com

*Date:* 2025-12-28

*PostgreSQL Version:* master (Development)
------------------------------
1. Summary & Motivation

This RFC proposes the introduction of minimal hooks into the PostgreSQL
storage layer and the addition of a *Transformation ID* field to the
PageHeader.
A Diplomatic Protocol Between Expert Groups

The core motivation of this proposal is *“Separation of Concerns and Mutual
Respect.”*

Historically, discussions around Transparent Data Encryption (TDE) have
often felt like putting security experts on trial in a foreign
court—specifically, the “Court of RDBMS.” It is time to treat them not as
defendants to be judged by database-specific rules, but as an *equal
neighboring community* with their own specialized sovereignty.

*The issue has never been a failure of technology, but rather a
misplacement of the focal point.* While previous discussions were mired in
the technicalities of “how to hardcode encryption into the core,” this
proposal shifts the debate toward an architectural solution: “what
interface the core should provide to external experts.”

   - *RDBMS Experts* provide a trusted pipeline responsible for data I/O
   paths and consistency.
   - *Security Experts* take responsibility for the specialized domain of
   encryption algorithms and key management.

This hook system functions as a *Technical Protocol*—a high-level agreement
that allows these two expert groups to exchange data securely without
encroaching on each other’s territory.
------------------------------
2. Design Principles

   1. *Delegation of Authority:* The core remains independent of specific
   encryption standards, providing a “free territory” where security experts
   can respond to an ever-changing security landscape.
   2. *Diplomatic Convention:* The Transformation ID acts as a
   communication protocol between the engine and the extension. The engine
   uses this ID to identify the state of the data and hands over control to
   the appropriate expert (the extension).
   3. *Minimal Interference:* Overhead is kept near zero when hooks are not
   in use, ensuring the native performance of the PostgreSQL engine.

------------------------------
3. Proposal Specifications 3.1 The Interface (Hook Points)

We allow intervention by security experts through five contact points along
the I/O path:

   - *Read/Write Hooks:* mdread_post, mdwrite_pre, mdextend_pre
   (Transformation of the data area)
   - *WAL Hooks:* xlog_insert_pre, xlog_decode_pre (Transformation of
   transaction logs)

3.2 The Protocol Identifier (PageHeader Transformation ID)

We allocate 5 bits of pd_flags to define the “Security State” of a page.
This serves as a *Status Message* sent by the security expert to the
engine, utilized for key versioning and as a migration marker.
------------------------------
4. Reference Implementation: contrib/test_tde A Standard Code of Conduct
for Security Experts

This reference implementation exists not as a commercial product, but to
define the *Standards of the Diplomatic Protocol* that
encryption/decryption experts must follow when entering the PostgreSQL
domain.

   1. *Deterministic IV Derivation:* Demonstrates how to achieve
   cryptographic safety by trusting unique values provided by the engine
   (e.g., LSN).
   2. *Critical Section Safety:* Defines memory management regulations that
   security logic must follow within “Critical Sections” to maintain system
   stability.
   3. *Hook Chaining:* Demonstrates a cooperative structure that allows
   peaceful coexistence with other expert tools (e.g., compression, auditing).

------------------------------
5. Scope

   - *In-Scope:* Backend hook infrastructure, Transformation ID field, and
   reference code demonstrating diplomatic protocol compliance.
   - *Out-of-Scope:* Specific Key Management Systems (KMS), selection of
   specific cryptographic algorithms, and integration with external tools.

This proposal represents a strategic diplomatic choice: rather than the
PostgreSQL core assuming all security responsibilities, it grants security
experts a *sovereign territory through extensions* where they can perform
at their best.