Horton
Menu

Re: WIP: WAL prefetch (another approach)

Thomas Munro <thomas.munro@gmail.com>

View thread
From: Thomas Munro <thomas.munro@gmail.com>
To: Stephen Frost <sfrost@snowman.net>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Dmitry Dolgov <9erthalion6@gmail.com>, David Steele <david@pgmasters.net>, Andres Freund <andres@anarazel.de>, Alvaro Herrera <alvherre@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2020-11-18T05:10:31Z
Lists: pgsql-hackers

Attachments

On Sat, Nov 14, 2020 at 4:13 AM Stephen Frost <sfrost@snowman.net> wrote:
> * Tomas Vondra (tomas.vondra@enterprisedb.com) wrote:
> > On 11/13/20 3:20 AM, Thomas Munro wrote:
> > > I'm not really sure what to do about achive restore scripts that
> > > block.  That seems to be fundamentally incompatible with what I'm
> > > doing here.
> >
> > IMHO we can't do much about that, except for documenting it - if the
> > prefetch can't work because of blocking restore script, someone has to
> > fix/improve the script. No way around that, I'm afraid.
>
> I'm a bit confused about what the issue here is- is the concern that a
> restore_command is specified that isn't allowed to run concurrently but
> this patch is intending to run more than one concurrently..?  There's
> another patch that I was looking at for doing pre-fetching of WAL
> segments, so if this is also doing that we should figure out which
> patch we want..

The problem is that the recovery loop tries to look further ahead in
between applying individual records, which causes the restore script
to run, and if that blocks, we won't apply records that we already
have, because we're waiting for the next WAL file to appear.  This
behaviour is on by default with my patch, so pg_standby will introduce
a weird replay delays.  We could think of some ways to fix that, with
meaningful return codes and periodic polling or something, I suppose,
but something feels a bit weird about it.

> I don't know that it's needed, but it feels likely that we could provide
> a better result if we consider making changes to the restore_command API
> (eg: have a way to say "please fetch this many segments ahead, and you
> can put them in this directory with these filenames" or something).  I
> would think we'd be able to continue supporting the existing API and
> accept that it might not be as performant.

Hmm.  Every time I try to think of a protocol change for the
restore_command API that would be acceptable, I go around the same
circle of thoughts about event flow and realise that what we really
need for this is ... a WAL receiver...

Here's a rebase over the recent commit "Get rid of the dedicated latch
for signaling the startup process." just to fix cfbot; no other
changes.

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix recovery_prefetch docs.

  2. Prefetch data referenced by the WAL, take II.

  3. Add circular WAL decoding buffer, take II.

  4. Fix generation of ./INSTALL for the distribution tarball

  5. Revert recovery prefetching feature.

  6. Sync guc.c and postgresql.conf.sample with the SGML docs.

  7. Add information of total data processed to replication slot stats.

  8. Doc: Review for "Optionally prefetch referenced data in recovery."

  9. Add circular WAL decoding buffer.

  10. Optionally prefetch referenced data in recovery.

  11. Remove read_page callback from XLogReader.

  12. Provide ReadRecentBuffer() to re-pin buffers by ID.

  13. Provide recovery_init_sync_method=syncfs.

  14. Mark factorial operator, and postfix operators in general, as deprecated.

  15. Rationalize GetWalRcv{Write,Flush}RecPtr().

  16. Support PrefetchBuffer() in recovery.

  17. Prevent hard failures of standbys caused by recycled WAL segments