Horton

Re: Handing off SLRU fsyncs to the checkpointer

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Jakub Wartak <Jakub.Wartak@tomtom.com>
Cc: "alvherre@2ndquadrant.com" <alvherre@2ndquadrant.com>, Robert Haas <robertmhaas@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2020-08-28T05:45:41Z
Lists: pgsql-hackers 
On Thu, Aug 27, 2020 at 8:48 PM Jakub Wartak <Jakub.Wartak@tomtom.com> wrote:
> >>     29.62%  postgres  [kernel.kallsyms]   [k] copy_user_enhanced_fast_string
> >>             ---copy_user_enhanced_fast_string
> >>                |--17.98%--copyin
> >> [..]
> >>                |          __pwrite_nocancel
> >>                |          FileWrite
> >>                |          mdwrite
> >>                |          FlushBuffer
> >>                |          ReadBuffer_common
> >>                |          ReadBufferWithoutRelcache
> >>                |          XLogReadBufferExtended
> >>                |          XLogReadBufferForRedoExtended
> >>                |           --17.57%--btree_xlog_insert
> >
> > To move these writes out of recovery's way, we should probably just
> > run the bgwriter process during crash recovery.  I'm going to look
> > into that.
>
> Sounds awesome.

I wrote a quick and dirty experimental patch to try that.  I can't see
any benefit from it on pgbench with default shared buffers, but maybe
it would do better with your append test due to locality, especially
if you can figure out how to tune bgwriter to pace itself optimally.
https://github.com/macdice/postgres/tree/bgwriter-in-crash-recovery

Commits

  1. Remove unused function prototypes.

  2. Defer flushing of SLRU files.

  3. Improve the vacuum error context phase information.

  4. Cache smgrnblocks() results in recovery.

  5. Refactor the fsync queue for wider use.

  6. Increase maximum number of clog buffers.

  7. Make the number of CLOG buffers adaptive, based on shared_buffers.

  8. Replace implementation of pg_log as a relation accessed through the