Horton
Menu

Re: WIP: WAL prefetch (another approach)

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>, Simon Riggs <simon.riggs@enterprisedb.com>, Julien Rouhaud <rjuju123@gmail.com>, Ashutosh Sharma <ashu.coek88@gmail.com>, Daniel Gustafsson <daniel@yesql.se>, Stephen Frost <sfrost@snowman.net>, Alvaro Herrera <alvherre@2ndquadrant.com>, Andres Freund <andres@anarazel.de>, David Steele <david@pgmasters.net>, Dmitry Dolgov <9erthalion6@gmail.com>, Jakub Wartak <Jakub.Wartak@tomtom.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Tomas Vondra <tomas.vondra@enterprisedb.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2022-04-25T18:31:48Z
Lists: pgsql-hackers 
Oh, one more bit of data: here's an excerpt from pg_waldump output after
the failed test:

rmgr: Btree       len (rec/tot):     72/    72, tx:        727, lsn: 0/01903BC8, prev 0/01903B70, desc: INSERT_LEAF off 111, blkref #0: rel 1663/16384/2673 blk 9
rmgr: Btree       len (rec/tot):     72/    72, tx:        727, lsn: 0/01903C10, prev 0/01903BC8, desc: INSERT_LEAF off 141, blkref #0: rel 1663/16384/2674 blk 7
rmgr: Standby     len (rec/tot):     42/    42, tx:        727, lsn: 0/01903C58, prev 0/01903C10, desc: LOCK xid 727 db 16384 rel 16391 
rmgr: Transaction len (rec/tot):    437/   437, tx:        727, lsn: 0/01903C88, prev 0/01903C58, desc: COMMIT 2022-04-25 20:16:03.374197 CEST; inval msgs: catcache 80 catcache 79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 snapshot 2608 relcache 16391
rmgr: Heap        len (rec/tot):     59/    59, tx:        728, lsn: 0/01903E40, prev 0/01903C88, desc: INSERT+INIT off 1 flags 0x00, blkref #0: rel 1663/16384/16391 blk 0
rmgr: Heap        len (rec/tot):     59/    59, tx:        728, lsn: 0/01903E80, prev 0/01903E40, desc: INSERT off 2 flags 0x00, blkref #0: rel 1663/16384/16391 blk 0
rmgr: Transaction len (rec/tot):     34/    34, tx:        728, lsn: 0/01903EC0, prev 0/01903E80, desc: COMMIT 2022-04-25 20:16:03.379323 CEST
rmgr: Heap        len (rec/tot):     59/    59, tx:        729, lsn: 0/01903EE8, prev 0/01903EC0, desc: INSERT off 3 flags 0x00, blkref #0: rel 1663/16384/16391 blk 0
rmgr: Heap        len (rec/tot):     59/    59, tx:        729, lsn: 0/01903F28, prev 0/01903EE8, desc: INSERT off 4 flags 0x00, blkref #0: rel 1663/16384/16391 blk 0
rmgr: Transaction len (rec/tot):     34/    34, tx:        729, lsn: 0/01903F68, prev 0/01903F28, desc: COMMIT 2022-04-25 20:16:03.381720 CEST

The error is complaining about not being able to read 0/01903E40,
which AFAICT is from the first "INSERT INTO sample_tbl" command,
which most certainly ought to be down to disk at this point.

Also, I modified the test script to see what WAL LSNs it thought
it was dealing with, and got

+\echo 'wal_lsn1 = ' :wal_lsn1
+wal_lsn1 =  0/1903E40
+\echo 'wal_lsn2 = ' :wal_lsn2
+wal_lsn2 =  0/1903EE8

confirming that idea of where 0/01903E40 is in the WAL history.
So this is sure looking like a bug somewhere in xlogreader.c,
not in pg_walinspect.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix recovery_prefetch docs.

  2. Prefetch data referenced by the WAL, take II.

  3. Add circular WAL decoding buffer, take II.

  4. Fix generation of ./INSTALL for the distribution tarball

  5. Revert recovery prefetching feature.

  6. Sync guc.c and postgresql.conf.sample with the SGML docs.

  7. Add information of total data processed to replication slot stats.

  8. Doc: Review for "Optionally prefetch referenced data in recovery."

  9. Add circular WAL decoding buffer.

  10. Optionally prefetch referenced data in recovery.

  11. Remove read_page callback from XLogReader.

  12. Provide ReadRecentBuffer() to re-pin buffers by ID.

  13. Provide recovery_init_sync_method=syncfs.

  14. Mark factorial operator, and postfix operators in general, as deprecated.

  15. Rationalize GetWalRcv{Write,Flush}RecPtr().

  16. Support PrefetchBuffer() in recovery.

  17. Prevent hard failures of standbys caused by recycled WAL segments