Horton

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

Justin Pryzby <pryzby@telsasoft.com>

View thread View on postgr.es
From: Justin Pryzby <pryzby@telsasoft.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Andres Freund <andres@anarazel.de>, Ashutosh Sharma <ashu.coek88@gmail.com>, Maciek Sakrejda <m.sakrejda@gmail.com>, Bruce Momjian <bruce@momjian.us>, Alvaro Herrera <alvherre@alvh.no-ip.org>, Andrew Dunstan <andrew@dunslane.net>, Heikki Linnakangas <hlinnaka@iki.fi>, pgsql-hackers@lists.postgresql.org
Date: 2022-08-02T22:04:16Z
Lists: pgsql-hackers 
On Tue, Aug 02, 2022 at 05:46:34PM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > WARNING:  problem in alloc set PortalContext: req size > alloc size for chunk 0x7f99508911f0 in block 0x7f9950890800
> 
> This looks like nothing so much as the fallout from something scribbling
> past the end of an allocated palloc chunk, or perhaps writing on
> already-freed space.  Perhaps running the test case under valgrind
> would help to finger the culprit.

Yeah but my test case is so poor that it's a chore ...

(Sorry for that, but it took me 2 days to be able to reproduce the problem so I
sent it sooner rather than looking for a better way ... )

I got this interesting looking thing.

==11628== Invalid write of size 8
==11628==    at 0x1D12B3A: smgrsetowner (smgr.c:213)
==11628==    by 0x1C7C224: RelationGetSmgr (rel.h:572)
==11628==    by 0x1C7C224: RelationCopyStorageUsingBuffer (bufmgr.c:3725)
==11628==    by 0x1C7C7A6: CreateAndCopyRelationData (bufmgr.c:3817)
==11628==    by 0x14A4518: CreateDatabaseUsingWalLog (dbcommands.c:221)
==11628==    by 0x14AB009: createdb (dbcommands.c:1393)
==11628==    by 0x1D2B9AF: standard_ProcessUtility (utility.c:776)
==11628==    by 0x1D2C46A: ProcessUtility (utility.c:530)
==11628==    by 0x1D265F5: PortalRunUtility (pquery.c:1158)
==11628==    by 0x1D27089: PortalRunMulti (pquery.c:1315)
==11628==    by 0x1D27A7C: PortalRun (pquery.c:791)
==11628==    by 0x1D1E33D: exec_simple_query (postgres.c:1243)
==11628==    by 0x1D218BC: PostgresMain (postgres.c:4505)
==11628==  Address 0x1025bc18 is 2,712 bytes inside a block of size 8,192 free'd
==11628==    at 0x4033A3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==11628==    by 0x217D7C2: AllocSetReset (aset.c:608)
==11628==    by 0x219B57A: MemoryContextResetOnly (mcxt.c:181)
==11628==    by 0x217DBD5: AllocSetDelete (aset.c:654)
==11628==    by 0x219C1EC: MemoryContextDelete (mcxt.c:252)
==11628==    by 0x21A109F: PortalDrop (portalmem.c:596)
==11628==    by 0x21A269C: AtCleanup_Portals (portalmem.c:907)
==11628==    by 0x11FEAB1: CleanupTransaction (xact.c:2890)
==11628==    by 0x120A74C: AbortCurrentTransaction (xact.c:3328)
==11628==    by 0x1D2158C: PostgresMain (postgres.c:4232)
==11628==    by 0x1B15DB5: BackendRun (postmaster.c:4490)
==11628==    by 0x1B1D799: BackendStartup (postmaster.c:4218)
==11628==  Block was alloc'd at
==11628==    at 0x40327F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==11628==    by 0x217F0DC: AllocSetAlloc (aset.c:920)
==11628==    by 0x219E4D2: palloc (mcxt.c:1082)
==11628==    by 0x14A14BE: ScanSourceDatabasePgClassTuple (dbcommands.c:444)
==11628==    by 0x14A1CD8: ScanSourceDatabasePgClassPage (dbcommands.c:384)
==11628==    by 0x14A20BF: ScanSourceDatabasePgClass (dbcommands.c:322)
==11628==    by 0x14A4348: CreateDatabaseUsingWalLog (dbcommands.c:177)
==11628==    by 0x14AB009: createdb (dbcommands.c:1393)
==11628==    by 0x1D2B9AF: standard_ProcessUtility (utility.c:776)
==11628==    by 0x1D2C46A: ProcessUtility (utility.c:530)
==11628==    by 0x1D265F5: PortalRunUtility (pquery.c:1158)
==11628==    by 0x1D27089: PortalRunMulti (pquery.c:1315)

-- 
Justin

Commits

  1. When using the WAL-logged CREATE DATABASE strategy, bulk extend.

  2. Avoid using a fake relcache entry to own an SmgrRelation.

  3. Fix data-corruption hazard in WAL-logged CREATE DATABASE.

  4. initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.

  5. Simplify a needlessly-complicated regular expression.

  6. In 020_createdb.pl, change order of command-line arguments.

  7. Add new block-by-block strategy for CREATE DATABASE.

  8. Fix replay of create database records on standby

  9. Refactor code for reading and writing relation map files.

  10. Replace RelationOpenSmgr() with RelationGetSmgr().

  11. Refactor the fsync queue for wider use.