Horton

Re: Improving connection scalability: GetSnapshotData()

Alexander Korotkov <a.korotkov@postgrespro.ru>

View thread View on postgr.es
From: Alexander Korotkov <a.korotkov@postgrespro.ru>
To: Peter Geoghegan <pg@bowt.ie>
Cc: Andres Freund <andres@anarazel.de>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2020-03-29T18:24:32Z
Lists: pgsql-hackers 
On Sun, Mar 29, 2020 at 4:40 AM Peter Geoghegan <pg@bowt.ie> wrote:
> On Sun, Mar 1, 2020 at 12:36 AM Andres Freund <andres@anarazel.de> wrote:
> > The workload is a pgbench readonly, with pgbench -M prepared -c $conns
> > -j $conns -S -n for each client count.  This is on a machine with 2
> > Intel(R) Xeon(R) Platinum 8168, but virtualized.
> >
> > conns   tps master              tps pgxact-split
> >
> > 1       26842.492845            26524.194821
> > 10      246923.158682           249224.782661
> > 50      695956.539704           709833.746374
> > 100     1054727.043139          1903616.306028
> > 200     964795.282957           1949200.338012
> > 300     906029.377539           1927881.231478
> > 400     845696.690912           1911065.369776
> > 500     812295.222497           1926237.255856
> > 600     888030.104213           1903047.236273
> > 700     866896.532490           1886537.202142
> > 800     863407.341506           1883768.592610
> > 900     871386.608563           1874638.012128
> > 1000    887668.277133           1876402.391502
> > 1500    860051.361395           1815103.564241
> > 2000    890900.098657           1775435.271018
> > 3000    874184.980039           1653953.817997
> > 4000    845023.080703           1582582.316043
> > 5000    817100.195728           1512260.802371
> >
> > I think these are pretty nice results.
>
> This scalability improvement is clearly very significant. There is
> little question that this is a strategically important enhancement for
> the Postgres project in general. I hope that you will ultimately be
> able to commit the patchset before feature freeze.

+1, this is really very cool results.

Despite this patchset is expected to be clearly a big win on majority
of workloads, I think we still need to investigate different workloads
on different hardware to ensure there is no regression.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Commits

  1. Try to unbreak 021_row_visibility.pl on mingw.

  2. Fix and test snapshot behavior on standby.

  3. Fix race condition in snapshot caching when 2PC is used.

  4. snapshot scalability: cache snapshots using a xact completion counter.

  5. Fix use of wrong index in ComputeXidHorizons().

  6. Make vacuum a bit more verbose to debug BF failure.

  7. snapshot scalability: Introduce dense array of in-progress xids.

  8. snapshot scalability: Move PGXACT->vacuumFlags to ProcGlobal->vacuumFlags.

  9. snapshot scalability: Move subxact info to ProcGlobal, remove PGXACT.

  10. snapshot scalability: Move PGXACT->xmin back to PGPROC.

  11. snapshot scalability: Don't compute global horizons while building snapshots.

  12. BRIN: Handle concurrent desummarization properly

  13. Track latest completed xid as a FullTransactionId.

  14. Rename VariableCacheData.nextFullXid to nextXid.

  15. snapshot scalability: Move delayChkpt from PGXACT to PGPROC.

  16. Report progress of CREATE INDEX operations