Horton
Menu

Re: WIP: WAL prefetch (another approach)

Thomas Munro <thomas.munro@gmail.com>

View thread
From: Thomas Munro <thomas.munro@gmail.com>
To: Stephen Frost <sfrost@snowman.net>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Dmitry Dolgov <9erthalion6@gmail.com>, David Steele <david@pgmasters.net>, Andres Freund <andres@anarazel.de>, Alvaro Herrera <alvherre@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-01-27T04:34:22Z
Lists: pgsql-hackers 
On Sat, Dec 5, 2020 at 7:27 AM Stephen Frost <sfrost@snowman.net> wrote:
> * Thomas Munro (thomas.munro@gmail.com) wrote:
> > I just noticed this thread proposing to retire pg_standby on that
> > basis:
> >
> > https://www.postgresql.org/message-id/flat/20201029024412.GP5380%40telsasoft.com
> >
> > I'd be happy to see that land, to fix this problem with my plan.  But
> > are there other people writing restore scripts that block that would
> > expect them to work on PG14?
>
> Ok, I think I finally get the concern that you're raising here-
> basically that if a restore command was written to sit around and wait
> for WAL segments to arrive, instead of just returning to PG and saying
> "WAL segment not found", that this would be a problem if we are running
> out ahead of the applying process and asking for WAL.
>
> The thing is- that's an outright broken restore command script in the
> first place.  If PG is in standby mode, we'll ask again if we get an
> error result indicating that the WAL file wasn't found.  The restore
> command documentation is quite clear on this point:
>
> The command will be asked for file names that are not present in the
> archive; it must return nonzero when so asked.
>
> There's no "it can wait around for the next file to show up if it wants
> to" in there- it *must* return nonzero when asked for files that don't
> exist.

Well the manual does actually describe how to write your own version
of pg_standby, referred to as a "waiting restore script":

https://www.postgresql.org/docs/13/log-shipping-alternative.html

I've now poked that other thread threatening to commit the removal of
pg_standby, and while I was there, also to remove the section on how
to write your own (it's possible that I missed some other reference to
the concept elsewhere, I'll need to take another look).

> So, I don't think that we really need to stress over this.  The fact
> that pg_standby offers options to have it wait instead of just returning
> a non-zero error-code and letting the loop that we already do in the
> core code seems like it's really just a legacy thing from before we were
> doing that and probably should have been ripped out long ago...  Even
> more reason to get rid of pg_standby tho, imv, we haven't been properly
> adjusting it when we've been making changes to the core code, it seems.

So far I haven't heard from anyone who thinks we should keep this old
facility (as useful as it was back then when it was the only way), so
I hope we can now quietly drop it.  It's not strictly an obstacle to
this recovery prefetching work, but it'd interact confusingly in hard
to describe ways, and it seems strange to perpetuate something that
many were already proposing to drop due to obsolescence.  Thanks for
the comments/sanity check.

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix recovery_prefetch docs.

  2. Prefetch data referenced by the WAL, take II.

  3. Add circular WAL decoding buffer, take II.

  4. Fix generation of ./INSTALL for the distribution tarball

  5. Revert recovery prefetching feature.

  6. Sync guc.c and postgresql.conf.sample with the SGML docs.

  7. Add information of total data processed to replication slot stats.

  8. Doc: Review for "Optionally prefetch referenced data in recovery."

  9. Add circular WAL decoding buffer.

  10. Optionally prefetch referenced data in recovery.

  11. Remove read_page callback from XLogReader.

  12. Provide ReadRecentBuffer() to re-pin buffers by ID.

  13. Provide recovery_init_sync_method=syncfs.

  14. Mark factorial operator, and postfix operators in general, as deprecated.

  15. Rationalize GetWalRcv{Write,Flush}RecPtr().

  16. Support PrefetchBuffer() in recovery.

  17. Prevent hard failures of standbys caused by recycled WAL segments