Horton
Menu

Re: WIP: WAL prefetch (another approach)

Tomas Vondra <tomas.vondra@2ndquadrant.com>

View thread
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Stephen Frost <sfrost@snowman.net>, Dmitry Dolgov <9erthalion6@gmail.com>, David Steele <david@pgmasters.net>, Andres Freund <andres@anarazel.de>, Alvaro Herrera <alvherre@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2020-10-07T00:58:38Z
Lists: pgsql-hackers 
On Thu, Sep 24, 2020 at 11:38:45AM +1200, Thomas Munro wrote:
>On Wed, Sep 9, 2020 at 11:16 AM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>> OK, thanks for looking into this. I guess I'll wait for an updated patch
>> before testing this further. The storage has limited capacity so I'd
>> have to either reduce the amount of data/WAL or juggle with the WAL
>> segments somehow. Doesn't seem worth it.
>
>Here's a new WIP version that works for archive-based recovery in my tests.
>
>The main change I have been working on is that there is now just a
>single XLogReaderState, so no more double-reading and double-decoding
>of the WAL.  It provides XLogReadRecord(), as before, but now you can
>also read further ahead with XLogReadAhead().  The user interface is
>much like before, except that the GUCs changed a bit.  They are now:
>
>  recovery_prefetch=on
>  recovery_prefetch_fpw=off
>  wal_decode_buffer_size=256kB
>  maintenance_io_concurrency=10
>
>I recommend setting maintenance_io_concurrency and
>wal_decode_buffer_size much higher than those defaults.
>

I think you've left the original GUC (replaced by the buffer size) in
the postgresql.conf.sample file. Confused me for a bit ;-)

I've done a bit of testing and so far it seems to work with WAL archive,
so I'll do more testing and benchmarking over the next couple days.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix recovery_prefetch docs.

  2. Prefetch data referenced by the WAL, take II.

  3. Add circular WAL decoding buffer, take II.

  4. Fix generation of ./INSTALL for the distribution tarball

  5. Revert recovery prefetching feature.

  6. Sync guc.c and postgresql.conf.sample with the SGML docs.

  7. Add information of total data processed to replication slot stats.

  8. Doc: Review for "Optionally prefetch referenced data in recovery."

  9. Add circular WAL decoding buffer.

  10. Optionally prefetch referenced data in recovery.

  11. Remove read_page callback from XLogReader.

  12. Provide ReadRecentBuffer() to re-pin buffers by ID.

  13. Provide recovery_init_sync_method=syncfs.

  14. Mark factorial operator, and postfix operators in general, as deprecated.

  15. Rationalize GetWalRcv{Write,Flush}RecPtr().

  16. Support PrefetchBuffer() in recovery.

  17. Prevent hard failures of standbys caused by recycled WAL segments