Re: WIP: WAL prefetch (another approach)

Thomas Munro <thomas.munro@gmail.com>

From: Thomas Munro <thomas.munro@gmail.com>

To: Tomas Vondra <tomas.vondra@2ndquadrant.com>

Cc: Stephen Frost <sfrost@snowman.net>, Dmitry Dolgov <9erthalion6@gmail.com>, David Steele <david@pgmasters.net>, Andres Freund <andres@anarazel.de>, Alvaro Herrera <alvherre@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>

Date: 2020-08-13T06:57:20Z

Lists: pgsql-hackers

Attachments

v10-0001-Add-pg_atomic_unlocked_add_fetch_XXX.patch (text/x-patch) patch v10-0001
v10-0002-Allow-XLogReadRecord-to-be-non-blocking.patch (text/x-patch) patch v10-0002
v10-0003-Prefetch-referenced-blocks-during-recovery.patch (text/x-patch) patch v10-0003

On Thu, Aug 6, 2020 at 10:47 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> On Thu, Aug 06, 2020 at 02:58:44PM +1200, Thomas Munro wrote:
> >On Tue, Aug 4, 2020 at 3:47 AM Tomas Vondra
> >> Any luck trying to reproduce thigs? Should I try again and collect some
> >> additional debug info?
> >
> >No luck.  I'm working on it now, and also trying to reduce the
> >overheads so that we're not doing extra work when it doesn't help.
>
> OK, I'll see if I can still reproduce it.

Since someone else ask me off-list, here's a rebase, with no
functional changes.  Soon I'll post a new improved version, but this
version just fixes the bitrot and hopefully turns cfbot green.

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix recovery_prefetch docs.
- dafae9707ab7 15.0 landed
Prefetch data referenced by the WAL, take II.
- 5dc0418fab28 15.0 landed
Add circular WAL decoding buffer, take II.
- 3f1ce973467a 15.0 landed
Fix generation of ./INSTALL for the distribution tarball
- 45aa88fe1d40 14.0 cited
Revert recovery prefetching feature.
- c2dc19342e05 14.0 landed
Sync guc.c and postgresql.conf.sample with the SGML docs.
- a55a98477b69 14.0 cited
Add information of total data processed to replication slot stats.
- f5fc2f5b23d1 14.0 cited
Doc: Review for "Optionally prefetch referenced data in recovery."
- dc88460c24ed 14.0 landed
Add circular WAL decoding buffer.
- f003d9f8721b 14.0 landed
Optionally prefetch referenced data in recovery.
- 1d257577e08d 14.0 landed
Remove read_page callback from XLogReader.
- 323cbe7c7ddc 14.0 cited
Provide ReadRecentBuffer() to re-pin buffers by ID.
- 2f27f8c51149 14.0 landed
Provide recovery_init_sync_method=syncfs.
- 61752afb2640 14.0 cited
Mark factorial operator, and postfix operators in general, as deprecated.
- 6ca547cf75ef 14.0 cited
Rationalize GetWalRcv{Write,Flush}RecPtr().
- d140f2f3e225 13.0 landed
Support PrefetchBuffer() in recovery.
- 3985b600f57d 13.0 landed
Prevent hard failures of standbys caused by recycled WAL segments
- 70b4f82a4b5c 11.0 cited