Horton

Re: [HACKERS] WAL logging problem in 9.4.3?

Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>

View thread View on postgr.es
From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: noah@leadboat.com
Cc: pgsql-hackers@postgresql.org, 9erthalion6@gmail.com, andrew.dunstan@2ndquadrant.com, hlinnaka@iki.fi, robertmhaas@gmail.com, michael@paquier.xyz
Date: 2019-05-23T07:10:35Z
Lists: pgsql-hackers

Attachments

Attached is a new version.

At Tue, 21 May 2019 21:29:48 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20190521.212948.34357392.horiguchi.kyotaro@lab.ntt.co.jp>

> At Mon, 20 May 2019 15:54:30 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20190520.155430.215084510.horiguchi.kyotaro@lab.ntt.co.jp>
> > > I suspect the design in the https://postgr.es/m/559FA0BA.3080808@iki.fi last
> > > paragraph will be simpler, not more complex.  In the implementation I'm
> > > envisioning, smgrDoPendingDeletes() would change name, perhaps to
> > > AtEOXact_Storage().  For every relfilenode it does not delete, it would ensure
> > > durability by syncing (for large nodes) or by WAL-logging each page (for small
> > > nodes).  RelationNeedsWAL() would return false whenever the applicable
> > > relfilenode appears in pendingDeletes.  Access methods would remove their
> > > smgrimmedsync() calls, but they would otherwise not change.  Would anyone like
> > > to try implementing that?
> > 
> > Following this direction, the attached PoC works *at least for*
> > the wal_optimization TAP tests, but doing pending flush not in
> > smgr but in relcache. This is extending skip-wal feature to
> > indexes. And makes the old 0002 patch on nbtree useless.
> 
> This is a tidier version of the patch.
> 
> - Passes regression tests including 018_wal_optimize.pl
> 
> - Move the substantial work to table/index AMs.
> 
>   Each AM can decide whether to support WAL skip or not.
>   Currently heap and nbtree support it.
> 
> - The timing of sync is moved from AtEOXact to PreCommit. This is
>   because heap_sync() needs xact state = INPROGRESS.
> 
> - matview and cluster is broken, since swapping to new
>   relfilenode doesn't change rd_newRelfilenodeSubid. I'll address
>   that.

cluster/matview are fixed.

A obstacle to fix them was the unreliability of
newRelfilenodeSubid.  As mentioned in the comment of
RelationData, newRelfilenodeSubid may dissapear by certain
sequence of commands.

In the attched v14, I added "rd_firstRelfilenodeSubid", which
stores the subtransaction id where the first relfilenode
replacementin the current transaction. It suivives any sequence
of commands, including one mentioned in CopyFrom's comment (which
I removed by this patch).

With the attached patch, on relations based on table/index AMs
that supports WAL-skipping, WAL-logging is eliminated if the
relation is created in the current transaction, or relfilenode is
replaced in the current transaction. At-commit file sync is
surely performed. (Only Heap and Btree support it.)

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Add perl2host call missing from a new test file.

  2. Skip WAL for new relfilenodes, under wal_level=minimal.

  3. Revert "Skip WAL for new relfilenodes, under wal_level=minimal."

  4. Back-patch log_newpage_range().

  5. During heap rebuild, lock any TOAST index until end of transaction.

  6. In log_newpage_range(), heed forkNum and page_std arguments.

  7. Back-patch src/test/recovery and PostgresNode from 9.6 to 9.5.

  8. Reduce pg_ctl's reaction time when waiting for postmaster start/stop.

  9. Accelerate end-of-transaction dropping of relations

  10. Redesign the planner's handling of index-descent cost estimation.

  11. Make TRUNCATE do truncate-in-place when processing a relation that was created