Thread

Re: RFC: PostgreSQL Storage I/O Transformation Hooks

Konstantin Knizhnik <knizhnik@garret.ru> — 2025-12-28T17:17:40Z

On 28/12/2025 5:51 PM, Henson Choi wrote:
> Hi Konstantin,
>
> I understand the decorator pattern, and yes, it can work for some cases.
> But decorators can only intercept at the beginning and end of functions.
>
> Looking at the actual hook locations in md.c:
>
> - mdextend_pre_hook: after error checks, before file open → Decorator 
> possible
> - mdwrite_pre_hook: after assertions, before I/O loop → Decorator possible
> - mdread_post_hook: inside the segment loop → Decorator NOT possible
>
> The mdreadv() function, introduced in PostgreSQL 17 as part of the
> vectored I/O API, processes multiple blocks in a loop that respects
> segment boundaries. The decryption hook must be called inside this loop,
> after each segment's FileReadV() completes. A decorator wrapping mdreadv()
> from the outside cannot access this internal loop timing.
>
> With the SMGR decorator approach, the extension developer must:
> - Track upstream md.c changes
> - Replicate the internal loop logic to find the right decryption point
>
> With hooks, the extension developer only needs to:
> - Implement encrypt() and decrypt()
>
> Regarding encryption+compression: that's a valid use case for SMGR,
> but our primary concern is different. In South Korea, government
> regulations require the use of nationally-approved cryptographic
> algorithms (such as ARIA, SEED). This means organizations often cannot
> adopt foreign TDE solutions, regardless of their technical merit.
>
> We need a simple, stable hook interface that allows local security
> experts to integrate these required algorithms - experts who understand
> cryptography but not PostgreSQL storage internals.
>
> If both approaches can coexist, why not provide hooks for the simple
> case and SMGR for the complex case?
>
> Best regards,
> Henson Choi


Hi Henson,

Thank you for explanations.
I personally do not like hooks, I considered them as some kind of 
crutches which are needed to fix some problems with existed APIs:)
But them are quite popular in Postgres and really make it extensible.

The task of transparent data encryption is really very important for 
Postgres (if for some reasons it can not be done at file system level).
If we need to add more hooks to make it possible to add to Postgres, 
then dozen of yet another hooks may be acceptable...

I have not investigated it precisely, may be you are right that it is 
possible to implement transparent encryption using using decorator 
approach and custom SMGR. Frankly speaking I am quite upset how AIO was 
added to PG18. It introduces orthogonal hierarchy to SMGR and cause 
some  tight dependencies between this two modules which makes extension 
of any of them problematic if ever possible (i.e. if I want to add my 
storage manager and make AIO use it to access files system, rather than 
calling pread/pwrite directly). I am not sure that AIO can not be added 
through SMGR hierarchy (certainly by extending this interface), but it 
is certainly separate store having no relation to the topic of this 
discussion.

So I can assume that current coupling of AIO with SMGR makes it not 
possible to plugin transparent encryption rather than adding this hooks.
Still not quite sure that proposed set of hooks is absolutely necessary 
and sufficient...