Re: WIP: WAL prefetch (another approach)

Thomas Munro <thomas.munro@gmail.com>

From: Thomas Munro <thomas.munro@gmail.com>

To: Tomas Vondra <tomas.vondra@2ndquadrant.com>

Cc: Stephen Frost <sfrost@snowman.net>, Dmitry Dolgov <9erthalion6@gmail.com>, David Steele <david@pgmasters.net>, Andres Freund <andres@anarazel.de>, Alvaro Herrera <alvherre@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>

Date: 2020-10-06T05:04:57Z

Lists: pgsql-hackers

Attachments

v12-0001-Add-pg_atomic_unlocked_add_fetch_XXX.patch (text/x-patch) patch v12-0001
v12-0002-Improve-information-about-received-WAL.patch (text/x-patch) patch v12-0002
v12-0003-Provide-XLogReadAhead-to-decode-future-WAL-recor.patch (text/x-patch) patch v12-0003
v12-0004-Prefetch-referenced-blocks-during-recovery.patch (text/x-patch) patch v12-0004
v12-0005-WIP-Avoid-extra-buffer-lookup-when-prefetching-W.patch (text/x-patch) patch v12-0005

On Thu, Sep 24, 2020 at 11:38 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Wed, Sep 9, 2020 at 11:16 AM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> > OK, thanks for looking into this. I guess I'll wait for an updated patch
> > before testing this further. The storage has limited capacity so I'd
> > have to either reduce the amount of data/WAL or juggle with the WAL
> > segments somehow. Doesn't seem worth it.
>
> Here's a new WIP version that works for archive-based recovery in my tests.

Rebased over recent merge conflicts in xlog.c.  I also removed a stray
debugging message.

One problem the current patch has is that if you use something like
pg_standby, that is, a restore command that waits for more data, then
it'll block waiting for WAL when it's trying to prefetch, which means
that replay is delayed.  I'm not sure what to think about that yet.

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix recovery_prefetch docs.
- dafae9707ab7 15.0 landed
Prefetch data referenced by the WAL, take II.
- 5dc0418fab28 15.0 landed
Add circular WAL decoding buffer, take II.
- 3f1ce973467a 15.0 landed
Fix generation of ./INSTALL for the distribution tarball
- 45aa88fe1d40 14.0 cited
Revert recovery prefetching feature.
- c2dc19342e05 14.0 landed
Sync guc.c and postgresql.conf.sample with the SGML docs.
- a55a98477b69 14.0 cited
Add information of total data processed to replication slot stats.
- f5fc2f5b23d1 14.0 cited
Doc: Review for "Optionally prefetch referenced data in recovery."
- dc88460c24ed 14.0 landed
Add circular WAL decoding buffer.
- f003d9f8721b 14.0 landed
Optionally prefetch referenced data in recovery.
- 1d257577e08d 14.0 landed
Remove read_page callback from XLogReader.
- 323cbe7c7ddc 14.0 cited
Provide ReadRecentBuffer() to re-pin buffers by ID.
- 2f27f8c51149 14.0 landed
Provide recovery_init_sync_method=syncfs.
- 61752afb2640 14.0 cited
Mark factorial operator, and postfix operators in general, as deprecated.
- 6ca547cf75ef 14.0 cited
Rationalize GetWalRcv{Write,Flush}RecPtr().
- d140f2f3e225 13.0 landed
Support PrefetchBuffer() in recovery.
- 3985b600f57d 13.0 landed
Prevent hard failures of standbys caused by recycled WAL segments
- 70b4f82a4b5c 11.0 cited