Horton

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

Justin Pryzby <pryzby@telsasoft.com>

View thread View on postgr.es
From: Justin Pryzby <pryzby@telsasoft.com>
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>, Andrew Dunstan <andrew@dunslane.net>, Dilip Kumar <dilipbalaut@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, pgsql-hackers@lists.postgresql.org
Date: 2022-02-13T02:17:46Z
Lists: pgsql-hackers 
On Sat, Feb 12, 2022 at 06:00:44PM -0800, Andres Freund wrote:
> Hi,
> 
> On 2022-02-11 16:19:12 -0500, Robert Haas wrote:
> > I somewhat hope we never end up with THREE strategies for creating a new
> > database, but now that I think about it, we might. Somebody might want to
> > use a fancy FS primitive that clones a directory at the FS level, or
> > something.
> 
> I think that'd be a great, and pretty easy to implement, feature. But it seems
> like it'd be mostly orthogonal to the "WAL log data" vs "checkpoint data"
> question? On the primary / single node system using "WAL log data" with "COW
> file copy" would work well.
> 
> I bet using COW file copies would speed up our own regression tests noticeably
> - on slower systems we spend a fair bit of time and space creating template0
> and postgres, with the bulk of the data never changing.
> 
> Template databases are also fairly commonly used by application developers to
> avoid the cost of rerunning all the setup DDL & initial data loading for
> different tests. Making that measurably cheaper would be a significant win.

+1

I ran into this last week and was still thinking about proposing it.

Would this help CI or any significant fraction of buildfarm ?
Or just tests run locally on supporting filesystems.

Note that pg_upgrade already supports copy/link/clone.  (Obviously, link
wouldn't do anything desirable for CREATE DATABASE).

-- 
Justin

Commits

  1. When using the WAL-logged CREATE DATABASE strategy, bulk extend.

  2. Avoid using a fake relcache entry to own an SmgrRelation.

  3. Fix data-corruption hazard in WAL-logged CREATE DATABASE.

  4. initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.

  5. Simplify a needlessly-complicated regular expression.

  6. In 020_createdb.pl, change order of command-line arguments.

  7. Add new block-by-block strategy for CREATE DATABASE.

  8. Fix replay of create database records on standby

  9. Refactor code for reading and writing relation map files.

  10. Replace RelationOpenSmgr() with RelationGetSmgr().

  11. Refactor the fsync queue for wider use.