Thread

Re: RFC: PostgreSQL Storage I/O Transformation Hooks

Konstantin Knizhnik <knizhnik@garret.ru> — 2025-12-28T12:11:39Z

On 28/12/2025 9:49 AM, Henson Choi wrote:
>
>
>   RFC: PostgreSQL Storage I/O Transformation Hooks
>
>
>     Infrastructure for a Technical Protocol Between RDBMS Core and
>     Data Security Experts
>
> *Author:* Henson Choi assam258@gmail.com
>
> *Date:* 2025-12-28
>
> *PostgreSQL Version:* master (Development)
>
> ------------------------------------------------------------------------
>
>
>     1. Summary & Motivation
>
> This RFC proposes the introduction of minimal hooks into the 
> PostgreSQL storage layer and the addition of a *Transformation ID* 
> field to the |PageHeader|.
>
>
>       A Diplomatic Protocol Between Expert Groups
>
> The core motivation of this proposal is *“Separation of Concerns and 
> Mutual Respect.”*
>
> Historically, discussions around Transparent Data Encryption (TDE) 
> have often felt like putting security experts on trial in a foreign 
> court—specifically, the “Court of RDBMS.” It is time to treat them not 
> as defendants to be judged by database-specific rules, but as an 
> *equal neighboring community* with their own specialized sovereignty.
>
> *The issue has never been a failure of technology, but rather a 
> misplacement of the focal point.* While previous discussions were 
> mired in the technicalities of “how to hardcode encryption into the 
> core,” this proposal shifts the debate toward an architectural 
> solution: “what interface the core should provide to external experts.”
>
>   * *RDBMS Experts* provide a trusted pipeline responsible for data
>     I/O paths and consistency.
>   * *Security Experts* take responsibility for the specialized domain
>     of encryption algorithms and key management.
>
> This hook system functions as a *Technical Protocol*—a high-level 
> agreement that allows these two expert groups to exchange data 
> securely without encroaching on each other’s territory.
>
> ------------------------------------------------------------------------
>
>
>     2. Design Principles
>
>  1. *Delegation of Authority:* The core remains independent of
>     specific encryption standards, providing a “free territory” where
>     security experts can respond to an ever-changing security landscape.
>  2. *Diplomatic Convention:* The Transformation ID acts as a
>     communication protocol between the engine and the extension. The
>     engine uses this ID to identify the state of the data and hands
>     over control to the appropriate expert (the extension).
>  3. *Minimal Interference:* Overhead is kept near zero when hooks are
>     not in use, ensuring the native performance of the PostgreSQL engine.
>
> ------------------------------------------------------------------------
>
>
>     3. Proposal Specifications
>
>
>       3.1 The Interface (Hook Points)
>
> We allow intervention by security experts through five contact points 
> along the I/O path:
>
>   * *Read/Write Hooks:* |mdread_post|, |mdwrite_pre|, |mdextend_pre|
>     (Transformation of the data area)
>   * *WAL Hooks:* |xlog_insert_pre|, |xlog_decode_pre| (Transformation
>     of transaction logs)
>
>
>       3.2 The Protocol Identifier (PageHeader Transformation ID)
>
> We allocate 5 bits of |pd_flags| to define the “Security State” of a 
> page. This serves as a *Status Message* sent by the security expert to 
> the engine, utilized for key versioning and as a migration marker.
>
> ------------------------------------------------------------------------
>
>
>     4. Reference Implementation: |contrib/test_tde|
>
>
>       A Standard Code of Conduct for Security Experts
>
> This reference implementation exists not as a commercial product, but 
> to define the *Standards of the Diplomatic Protocol* that 
> encryption/decryption experts must follow when entering the PostgreSQL 
> domain.
>
>  1. *Deterministic IV Derivation:* Demonstrates how to achieve
>     cryptographic safety by trusting unique values provided by the
>     engine (e.g., LSN).
>  2. *Critical Section Safety:* Defines memory management regulations
>     that security logic must follow within “Critical Sections” to
>     maintain system stability.
>  3. *Hook Chaining:* Demonstrates a cooperative structure that allows
>     peaceful coexistence with other expert tools (e.g., compression,
>     auditing).
>
> ------------------------------------------------------------------------
>
>
>     5. Scope
>
>   * *In-Scope:* Backend hook infrastructure, Transformation ID field,
>     and reference code demonstrating diplomatic protocol compliance.
>   * *Out-of-Scope:* Specific Key Management Systems (KMS), selection
>     of specific cryptographic algorithms, and integration with
>     external tools.
>
> This proposal represents a strategic diplomatic choice: rather than 
> the PostgreSQL core assuming all security responsibilities, it grants 
> security experts a *sovereign territory through extensions* where they 
> can perform at their best.
>
I wonder if instead of support a lot of extra hooks it will be better to 
provide extensible SMGR API:
https://www.postgresql.org/message-id/flat/CAPP%3DHha_wV1MV9yR70QZ5pk5dtNP%2BbOyBiFxPmrMKqnQeKMAwQ%40mail.gmail.com#ab0da3412525c7501ea17f3d4c602bbf
It seems to be much more straightforward, convenient and flexible 
mechanism than adding hooks, which can be used for many other purposes 
except transparent encryption.