Horton

Re: [HACKERS] WAL logging problem in 9.4.3?

Kyotaro Horiguchi <horikyota.ntt@gmail.com>

View thread View on postgr.es
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: noah@leadboat.com
Cc: robertmhaas@gmail.com, pgsql-hackers@postgresql.org, 9erthalion6@gmail.com, andrew.dunstan@2ndquadrant.com, hlinnaka@iki.fi, michael@paquier.xyz
Date: 2019-11-26T12:37:52Z
Lists: pgsql-hackers

Attachments

At Sun, 24 Nov 2019 22:08:39 -0500, Noah Misch <noah@leadboat.com> wrote in 
> On Mon, Nov 25, 2019 at 11:08:54AM +0900, Kyotaro Horiguchi wrote:
> > At Sat, 23 Nov 2019 16:21:36 -0500, Noah Misch <noah@leadboat.com> wrote in 
> > > I noticed an additional defect:
> > > 
> > > BEGIN;
> > > CREATE TABLE t (c) AS SELECT 1;
> > > CHECKPOINT; -- write and fsync the table's one page
> > > TRUNCATE t; -- no WAL
> > > COMMIT; -- no FPI, just the commit record
> > > 
> > > If we crash after the COMMIT and before the next fsync or OS-elected sync of
> > > the table's file, the table will stay on disk with its pre-TRUNCATE content.
> > 
> > The TRUNCATE replaces relfilenode in the catalog
> 
> No, it does not.  Since the relation is new in the transaction, the TRUNCATE
> uses the heap_truncate_one_rel() strategy.
..
> The zero-pages case is not special.  Here's an example of the problem with a
> nonzero size:

I got it. That is, if the file has had blocks beyond the size at
commit, we should sync the file even if it is small enough. It nees to
track beore-trunction size as this patch used to have.

pendingSyncHash is resurrected to do truncate-size tracking. That
information cannot be stored in SMgrRelation, which will be dissapper
by invalidation, or Relation, which is not available in storage layer.
smgrDoPendingDeletes is needed to be called at aboft again to clean up
useless hash. I'm not sure the exact cause but
AssertPendingSyncs_RelationCache() fails at abort (so it is not called
at abort).

smgrDoPendingSyncs and RelFileNodeSkippingWAL() become simpler by
using the hash.

Is is not fully checked. I didn't merged and mesured performance yet,
but I post the status-quo patch for now.

- v25-0001-version-nm.patch

Noah's v24 patch.

- v25-0002-Revert-FlushRelationBuffersWithoutRelcache.patch

Remove useless function (added by this patch..).

- v25-0003-Improve-the-performance-of-relation-syncs.patch

Make smgrDoPendingSyncs scan shared buffer once.

v25-0004-Adjust-gistGetFakeLSN.patch

Amendment for gistGetFakeLSN. This uses GetXLogInsertRecPtr as long as
it is different from the previous call and emits dummy WAL if we need
a new LSN. Since other than switch_wal record cannot be empty so the
dummy WAL has an integer content for now.

v25-0005-Sync-files-shrinked-by-truncation.patch

Amendment for the truncation problem.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Add perl2host call missing from a new test file.

  2. Skip WAL for new relfilenodes, under wal_level=minimal.

  3. Revert "Skip WAL for new relfilenodes, under wal_level=minimal."

  4. Back-patch log_newpage_range().

  5. During heap rebuild, lock any TOAST index until end of transaction.

  6. In log_newpage_range(), heed forkNum and page_std arguments.

  7. Back-patch src/test/recovery and PostgresNode from 9.6 to 9.5.

  8. Reduce pg_ctl's reaction time when waiting for postmaster start/stop.

  9. Accelerate end-of-transaction dropping of relations

  10. Redesign the planner's handling of index-descent cost estimation.

  11. Make TRUNCATE do truncate-in-place when processing a relation that was created