Horton
Menu

Re: WIP: WAL prefetch (another approach)

Stephen Frost <sfrost@snowman.net>

View thread
From: Stephen Frost <sfrost@snowman.net>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: Thomas Munro <thomas.munro@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Dmitry Dolgov <9erthalion6@gmail.com>, David Steele <david@pgmasters.net>, Andres Freund <andres@anarazel.de>, Alvaro Herrera <alvherre@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2020-11-13T15:13:45Z
Lists: pgsql-hackers 
Greetings,

* Tomas Vondra (tomas.vondra@enterprisedb.com) wrote:
> On 11/13/20 3:20 AM, Thomas Munro wrote:
> > I'm not really sure what to do about achive restore scripts that
> > block.  That seems to be fundamentally incompatible with what I'm
> > doing here.
> 
> IMHO we can't do much about that, except for documenting it - if the
> prefetch can't work because of blocking restore script, someone has to
> fix/improve the script. No way around that, I'm afraid.

I'm a bit confused about what the issue here is- is the concern that a
restore_command is specified that isn't allowed to run concurrently but
this patch is intending to run more than one concurrently..?  There's
another patch that I was looking at for doing pre-fetching of WAL
segments, so if this is also doing that we should figure out which
patch we want..

I don't know that it's needed, but it feels likely that we could provide
a better result if we consider making changes to the restore_command API
(eg: have a way to say "please fetch this many segments ahead, and you
can put them in this directory with these filenames" or something).  I
would think we'd be able to continue supporting the existing API and
accept that it might not be as performant.

Thanks,

Stephen

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix recovery_prefetch docs.

  2. Prefetch data referenced by the WAL, take II.

  3. Add circular WAL decoding buffer, take II.

  4. Fix generation of ./INSTALL for the distribution tarball

  5. Revert recovery prefetching feature.

  6. Sync guc.c and postgresql.conf.sample with the SGML docs.

  7. Add information of total data processed to replication slot stats.

  8. Doc: Review for "Optionally prefetch referenced data in recovery."

  9. Add circular WAL decoding buffer.

  10. Optionally prefetch referenced data in recovery.

  11. Remove read_page callback from XLogReader.

  12. Provide ReadRecentBuffer() to re-pin buffers by ID.

  13. Provide recovery_init_sync_method=syncfs.

  14. Mark factorial operator, and postfix operators in general, as deprecated.

  15. Rationalize GetWalRcv{Write,Flush}RecPtr().

  16. Support PrefetchBuffer() in recovery.

  17. Prevent hard failures of standbys caused by recycled WAL segments