Horton

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Bruce Momjian <bruce@momjian.us>
Cc: Greg Nancarrow <gregn4422@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, Heikki Linnakangas <hlinnaka@iki.fi>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-12-16T12:17:03Z
Lists: pgsql-hackers 
On Thu, Dec 16, 2021 at 12:15 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, Dec  2, 2021 at 07:19:50PM +0530, Dilip Kumar wrote:
> From the patch:
>
> > Currently, CREATE DATABASE forces a checkpoint, then copies all the files,
> > then forces another checkpoint. The comments in the createdb() function
> > explain the reasons for this. The attached patch fixes this problem by making
> > create database completely WAL logged so that we can avoid the checkpoints.
> >
> > This can also be useful for supporting the TDE. For example, if we need different
> > encryption for the source and the target database then we can not re-encrypt the
> > page data if we copy the whole directory.  But with this patch, we are copying
> > page by page so we have an opportunity to re-encrypt the page before copying that
> > to the target database.
>
> Uh, why is this true?  Why can't we just copy the heap/index files 8k at
> a time and reencrypt them during the file copy, rather than using shared
> buffers?

Hi Bruce,

Yeah, you are right that if we copy in 8k block then we can re-encrypt
the page, but in the current system, we are not copying block by
block.  So the main effort for this patch is not only for TDE but to
get rid of the checkpoint we are forced to do before and after create
database.  So my point is that in this patch since we are copying page
by page we get an opportunity to re-encrypt the page.  I agree that if
the re-encryption would have been the main goal of this patch then
true we can copy files in 8k blocks and re-encrypt those blocks, that
time even if we have to access some page data for re-encryption (like
nonce) then also we can do it, but that is not the main objective.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. When using the WAL-logged CREATE DATABASE strategy, bulk extend.

  2. Avoid using a fake relcache entry to own an SmgrRelation.

  3. Fix data-corruption hazard in WAL-logged CREATE DATABASE.

  4. initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.

  5. Simplify a needlessly-complicated regular expression.

  6. In 020_createdb.pl, change order of command-line arguments.

  7. Add new block-by-block strategy for CREATE DATABASE.

  8. Fix replay of create database records on standby

  9. Refactor code for reading and writing relation map files.

  10. Replace RelationOpenSmgr() with RelationGetSmgr().

  11. Refactor the fsync queue for wider use.