Horton

Re: [HACKERS] WAL logging problem in 9.4.3?

Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>

View thread View on postgr.es
From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: noah@leadboat.com
Cc: pgsql-hackers@postgresql.org, 9erthalion6@gmail.com, andrew.dunstan@2ndquadrant.com, hlinnaka@iki.fi, robertmhaas@gmail.com, michael@paquier.xyz
Date: 2019-05-20T06:54:30Z
Lists: pgsql-hackers

Attachments

Hello.

At Thu, 16 May 2019 23:50:50 -0700, Noah Misch <noah@leadboat.com> wrote in <20190517065050.GA1298884@rfd.leadboat.com>
> On Tue, May 14, 2019 at 01:59:10PM +0900, Kyotaro HORIGUCHI wrote:
> > At Sun, 12 May 2019 17:37:05 -0700, Noah Misch <noah@leadboat.com> wrote in <20190513003705.GA1202614@rfd.leadboat.com>
> > > On Sun, Mar 31, 2019 at 03:31:58PM -0700, Noah Misch wrote:
> > > > On Sun, Mar 10, 2019 at 07:27:08PM -0700, Noah Misch wrote:
> > > > > I also liked the design in the https://postgr.es/m/559FA0BA.3080808@iki.fi
> > > > > last paragraph, and I suspect it would have been no harder to back-patch.  I
> > > > > wonder if it would have been simpler and better, but I'm not asking anyone to
> > > > > investigate that.
> > > > 
> > > > Now I am asking for that.  Would anyone like to try implementing that other
> > > > design, to see how much simpler it would be?
> > 
> > Yeah, I think it is a bit too-complex for the value. But I think
> > it is the best way as far as we keep reusing a file on
> > truncation of the whole file.
> 
> The design of v11-0006-Fix-WAL-skipping-feature.patch doesn't, in general,
> work for WAL records touching more than one buffer.  For heapam, that patch
> works around this problem by emitting XLOG_HEAP_INSERT or XLOG_HEAP_DELETE
> when we'd normally emit XLOG_HEAP_UPDATE.  As a result, post-crash-recovery
> heap page bits differ from the bits present when we don't crash.  Though I'm
> 85% confident this does not introduce a bug today, this is fragile.  That is
> the main complexity I wish to avoid.

Ok, I see your point. The same issue happens on index pages more
aggressively. I didn't allow wal-skipping on indexes for the
reason.

> I suspect the design in the https://postgr.es/m/559FA0BA.3080808@iki.fi last
> paragraph will be simpler, not more complex.  In the implementation I'm
> envisioning, smgrDoPendingDeletes() would change name, perhaps to
> AtEOXact_Storage().  For every relfilenode it does not delete, it would ensure
> durability by syncing (for large nodes) or by WAL-logging each page (for small
> nodes).  RelationNeedsWAL() would return false whenever the applicable
> relfilenode appears in pendingDeletes.  Access methods would remove their
> smgrimmedsync() calls, but they would otherwise not change.  Would anyone like
> to try implementing that?

Following this direction, the attached PoC works *at least for*
the wal_optimization TAP tests, but doing pending flush not in
smgr but in relcache. This is extending skip-wal feature to
indexes. And makes the old 0002 patch on nbtree useless.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Add perl2host call missing from a new test file.

  2. Skip WAL for new relfilenodes, under wal_level=minimal.

  3. Revert "Skip WAL for new relfilenodes, under wal_level=minimal."

  4. Back-patch log_newpage_range().

  5. During heap rebuild, lock any TOAST index until end of transaction.

  6. In log_newpage_range(), heed forkNum and page_std arguments.

  7. Back-patch src/test/recovery and PostgresNode from 9.6 to 9.5.

  8. Reduce pg_ctl's reaction time when waiting for postmaster start/stop.

  9. Accelerate end-of-transaction dropping of relations

  10. Redesign the planner's handling of index-descent cost estimation.

  11. Make TRUNCATE do truncate-in-place when processing a relation that was created