Re: Extensible storage manager API - SMGR hook Redux

Hi!

We at Percona are very interested in this patch for our transparent data 
encryption extension. So we would love to collaborate with you, and 
anyone else interested, on making the SMGR extensible.

I have attached rebased and a bit cleaned up versions of Tristan's 
patches plus a couple of patches we have been working on in-house 
(mainly my colleague Zsolt). I also have some questions which I would 
like to discuss.

0001-0004

The same patches as Tristan posted but rebased and cleaned up a bit to 
better follow the code style. I also removed a couple of dead variables 
which seemed like left overs.

0005

Since we support having both encrypted and unencrypted relations we use 
the RelFileLocator to look up if a relation is encrypted. And to 
preserve that information when smgrcreate() creates a new relfile for a 
relation we pass along the old RelFileLocator.

For our use case it is possible that we could solve this in other ways. 
For example if we decide to go with configuring the SMGR per schema this 
will probably not be necessary at all.

0006

The patch introduces the concept of "chaining" SMGRs where we have tail 
(e.g. md or a theoretical Ceph SMGR) and modifier (e.g. TDE or the 
fsync_checker). Something like this would be useful for our case since 
it would be nice to be able to use the same encryption code for md and 
for some other potential replacement for md which uses some kind object 
storage for example.

As a bonus this allowed us to make the functions implementing md static.

It is currently controlled via a GUC, smgr_chain, but this will of 
course depend on how we decide to implement configuring which SMGR to use.

Questions

- What is up with the barrier when loading SMGRs? That does not seem 
necessary or am I missing something? I believe Andres also spotted this.

- How should we configure which SMGR to use for each relation? People 
have talked about doing it per tablespace or using hooks and we have a 
patch which uses a GUC for this. I have personally not researched these 
options enough to have an opinion yet.

- Is our idea about chaining SMGRs useful? In its current form or some 
variant inspired by it?

- We need to benchmark this to make sure we do not introduce too much 
overhead, especially for people who just want to use md. I saw for 
example that Andres had some complaint about extra indirection which we 
may have to address.

Andreas