Horton

Re: [Patch] Optimize dropping of relation buffers using dlist

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: tsunakawa.takay@fujitsu.com, k.jamison@fujitsu.com, Amit Kapila <amit.kapila16@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Andres Freund <andres@anarazel.de>, Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2020-10-22T09:27:00Z
Lists: pgsql-hackers 
On Thu, Oct 22, 2020 at 9:50 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
> By the way, heap scan finds the size of target relation using
> smgrnblocks().  I'm not sure why we don't miss recently-extended pages
> on a heap-scan?  It seems to be possible that concurrent checkpoint
> fsyncs relation files inbetween the extension and scanning and the
> scanning gets smaller size than it really is.

Yeah.  That's a narrow window: fsync() returns an error after the file
shrinks and we immediately panic.  A version with a wider window: the
kernel tries to write in the background, gets an I/O error, shrinks
the file, but we don't know this and we continue running until the
next checkpoint calls fsync(), sees the error and panics.  Seq scans
between those two events fail to see recently committed data at the
end of the table.

Commits

  1. Fix size overflow in calculation introduced by commits d6ad34f3 and bea449c6.

  2. Optimize DropRelFileNodesAllBuffers() for recovery.

  3. Optimize DropRelFileNodeBuffers() for recovery.

  4. Cache smgrnblocks() results in recovery.

  5. Add a check to prevent overwriting valid data if smgrnblocks() gives a