Re: Further pg_upgrade analysis for many tables

On Sun, Jan 20, 2013 at 02:11:48PM -0500, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > ! 		 *	Using pg_restore --single-transaction is faster than other
> > ! 		 *	methods, like --jobs.
> 
> Is this still the case now that Jeff's AtEOXact patch is in?  The risk
> of locktable overflow with --single-transaction makes me think that
> pg_upgrade should avoid it unless there is a *really* strong performance
> case for it, and I fear your old measurements are now invalidated.

I had thought that the AtEOXact patch only helped single transactions
with many tables, but I now remember it mostly helps backends that have
accessed many tables.

With max_locks_per_transaction set high, I tested with the attached
patch that removes --single-transaction from pg_restore.  I saw a 4%
improvement by removing that option, and 15% at 64k.  (Test script
attached.)  I have applied the patch.  This is good news not just for
pg_upgrade but for other backends that access many tables.  

	             git     patch
	    1       11.06    11.03
	 1000       19.97    20.86
	 2000       28.50    27.61
	 4000       46.90    45.65
	 8000       79.38    80.68
	16000      153.33   147.13
	32000      317.40   302.96
	64000      782.94   659.52

FYI, this is better than the tests I did on the original patch that
showed --single-transaction was still a win then:

    http://www.postgresql.org/message-id/20121128202232.GA31741@momjian.us

> 	#tbls       git     -1    AtOEXAct  both
> 	    1      11.06   13.06   10.99   13.20
> 	 1000      21.71   22.92   22.20   22.51
> 	 2000      32.86   31.09   32.51   31.62
> 	 4000      55.22   49.96   52.50   49.99
> 	 8000     105.34   82.10   95.32   82.94
> 	16000     223.67  164.27  187.40  159.53
> 	32000     543.93  324.63  366.44  317.93
> 	64000    1697.14  791.82  767.32  752.57

Keep in mind this doesn't totally avoid the requirement to increase
max_locks_per_transaction.  There are cases at >6k where pg_dump runs
out of locks, but I don't see how we can improve that.  Hopefully users
have already seen pg_dump fail and have adjusted
max_locks_per_transaction.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +