Horton

Re: Improving connection scalability: GetSnapshotData()

Ian Barwick <ian.barwick@2ndquadrant.com>

View thread View on postgr.es
From: Ian Barwick <ian.barwick@2ndquadrant.com>
To: Andres Freund <andres@anarazel.de>
Cc: Michael Paquier <michael@paquier.xyz>, Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>, Thomas Munro <thomas.munro@gmail.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, Daniel Gustafsson <daniel@yesql.se>, "Jonathan S. Katz" <jkatz@postgresql.org>, Alexander Korotkov <a.korotkov@postgrespro.ru>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Peter Geoghegan <pg@bowt.ie>, Bruce Momjian <bruce@momjian.us>, David Rowley <dgrowleyml@gmail.com>
Date: 2020-09-08T04:23:01Z
Lists: pgsql-hackers 
On 2020/09/08 13:11, Andres Freund wrote:
> Hi,
> 
> On 2020-09-08 13:03:01 +0900, Ian Barwick wrote:
(...)
>> I wonder if it's possible to increment "xactCompletionCount"
>> during replay along these lines:
>>
>>      *** a/src/backend/access/transam/xact.c
>>      --- b/src/backend/access/transam/xact.c
>>      *************** xact_redo_commit(xl_xact_parsed_commit *
>>      *** 5915,5920 ****
>>      --- 5915,5924 ----
>>               */
>>              if (XactCompletionApplyFeedback(parsed->xinfo))
>>                      XLogRequestWalReceiverReply();
>>      +
>>      +       LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
>>      +       ShmemVariableCache->xactCompletionCount++;
>>      +       LWLockRelease(ProcArrayLock);
>>        }
>>
>> which seems to work (though quite possibly I've overlooked something I don't
>> know that I don't know about and it will all break horribly somewhere,
>> etc. etc.).
> 
> We'd also need the same in a few more places. Probably worth looking at
> the list where we increment it on the primary (particularly we need to
> also increment it for aborts, and 2pc commit/aborts).

Yup.

> At first I was very confused as to why none of the existing tests have
> found this significant issue. But after thinking about it for a minute
> that's because they all use psql, and largely separate psql invocations
> for each query :(. Which means that there's no cached snapshot around...
> 
> Do you want to try to write a patch?

Sure, I'll give it a go as I have some time right now.


Regards

Ian Barwick

-- 
Ian Barwick                   https://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training & Services

Commits

  1. Try to unbreak 021_row_visibility.pl on mingw.

  2. Fix and test snapshot behavior on standby.

  3. Fix race condition in snapshot caching when 2PC is used.

  4. snapshot scalability: cache snapshots using a xact completion counter.

  5. Fix use of wrong index in ComputeXidHorizons().

  6. Make vacuum a bit more verbose to debug BF failure.

  7. snapshot scalability: Introduce dense array of in-progress xids.

  8. snapshot scalability: Move PGXACT->vacuumFlags to ProcGlobal->vacuumFlags.

  9. snapshot scalability: Move subxact info to ProcGlobal, remove PGXACT.

  10. snapshot scalability: Move PGXACT->xmin back to PGPROC.

  11. snapshot scalability: Don't compute global horizons while building snapshots.

  12. BRIN: Handle concurrent desummarization properly

  13. Track latest completed xid as a FullTransactionId.

  14. Rename VariableCacheData.nextFullXid to nextXid.

  15. snapshot scalability: Move delayChkpt from PGXACT to PGPROC.

  16. Report progress of CREATE INDEX operations