Horton

Thread

Commits

  1. Make TupleTableSlots extensible, finish split of existing slot type.

  2. Inline hot path of slot_getsomeattrs().

  3. Compute information about EEOP_*_FETCHSOME at expression init time.

  4. Don't generate tuple deforming functions for virtual slots.

  5. Introduce notion of different types of slots (without implementing them).

  6. Verify that expected slot types match returned slot types.

  7. Rejigger materializing and fetching a HeapTuple from a slot.

  8. Rationalize expression context reset in ExecModifyTable().

  9. Don't require return slots for nodes without projection.

  10. Move EEOP_*_SYSVAR evaluation out of line.

  11. Use installcheck-parallel in pg_upgrade's testsuite.

  1. Why do pg_upgrade's test use the serial schedule?

    Andres Freund <andres@anarazel.de> — 2018-11-05T21:00:39Z

    Hi,

The test I'm most commonly waiting for when doing a parallel make check
is pg_upgrade. And to a significant degree that's because it uses the
serial installcheck rather than installcheck-parallel.

Is there a reason to not use installcheck-parallel?

serial:
19.42user 7.77system 1:53.23elapsed 24%CPU (0avgtext+0avgdata 111420maxresident)k
parallel:
20.76user 7.72system 1:27.05elapsed 32%CPU (0avgtext+0avgdata 112300maxresident)k

So, a saving of ~25s.


ISTM we also should disable fsyncs for the postmaster instances. Wins
another ~5s on my laptop, and I suspect it makes a larger difference on
some of the buildfarm animals.

Greetings,

Andres Freund

  2. Re: Why do pg_upgrade's test use the serial schedule?

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-11-05T21:10:28Z

    Andres Freund <andres@anarazel.de> writes:
> ISTM we also should disable fsyncs for the postmaster instances. Wins
> another ~5s on my laptop, and I suspect it makes a larger difference on
> some of the buildfarm animals.

Buildfarm did that long ago.

			regards, tom lane

  3. Re: Why do pg_upgrade's test use the serial schedule?

    Andres Freund <andres@anarazel.de> — 2018-11-05T21:20:57Z

    On 2018-11-05 16:10:28 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > ISTM we also should disable fsyncs for the postmaster instances. Wins
> > another ~5s on my laptop, and I suspect it makes a larger difference on
> > some of the buildfarm animals.
> 
> Buildfarm did that long ago.

I don't think it did for pg_upgrade's test.sh?

POSTMASTER_OPTS="-F -c listen_addresses=\"$LISTEN_ADDRESSES\" -k \"$PGHOST\""
...
"$oldbindir"/pg_ctl start -l "$logdir/postmaster1.log" -o "$POSTMASTER_OPTS" -w

Andres Freund

  4. Re: Why do pg_upgrade's test use the serial schedule?

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-11-05T21:32:20Z

    Andres Freund <andres@anarazel.de> writes:
> On 2018-11-05 16:10:28 -0500, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> ISTM we also should disable fsyncs for the postmaster instances. Wins
>>> another ~5s on my laptop, and I suspect it makes a larger difference on
>>> some of the buildfarm animals.

>> Buildfarm did that long ago.

> I don't think it did for pg_upgrade's test.sh?

It's injected via the TEMP_CONFIG env variable.

			regards, tom lane

  5. Re: Why do pg_upgrade's test use the serial schedule?

    Andres Freund <andres@anarazel.de> — 2018-11-05T22:00:30Z

    On 2018-11-05 16:32:20 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2018-11-05 16:10:28 -0500, Tom Lane wrote:
> >> Andres Freund <andres@anarazel.de> writes:
> >>> ISTM we also should disable fsyncs for the postmaster instances. Wins
> >>> another ~5s on my laptop, and I suspect it makes a larger difference on
> >>> some of the buildfarm animals.
> 
> >> Buildfarm did that long ago.
> 
> > I don't think it did for pg_upgrade's test.sh?
> 
> It's injected via the TEMP_CONFIG env variable.

Ah, thanks.  Is there any good reason to not instead have it in the
script? Doesn't strike me as great that it slows down normal regression
runs during development?  Injecting TEMP_CONFIG w/ fsync=off globally
for those (rather than in a single test wrapper), could counteract
explicit tests where fsync is wanted - beside the fact that one has to
do so manually.  Normal regression tests have long done so.

Greetings,

Andres Freund

  6. Re: Why do pg_upgrade's test use the serial schedule?

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-11-05T22:08:12Z

    Andres Freund <andres@anarazel.de> writes:
> On 2018-11-05 16:32:20 -0500, Tom Lane wrote:
>> It's injected via the TEMP_CONFIG env variable.

> Ah, thanks.  Is there any good reason to not instead have it in the
> script?

Why that script in particular?  If you don't want fsync, you likely
don't want it across the entire check-world run.  The TEMP_CONFIG
thing is a hack no doubt, but it gets the job done globally.  Also,
if you *do* want fsync, there's one place to turn it back on.  I'm
not a fan of individual tests deciding they know what to do.

			regards, tom lane

  7. Re: Why do pg_upgrade's test use the serial schedule?

    Andres Freund <andres@anarazel.de> — 2018-11-05T22:11:42Z

    On 2018-11-05 17:08:12 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2018-11-05 16:32:20 -0500, Tom Lane wrote:
> >> It's injected via the TEMP_CONFIG env variable.
> 
> > Ah, thanks.  Is there any good reason to not instead have it in the
> > script?
> 
> Why that script in particular?

Because just about everything else taking substantial time already
disables it. pg_regress forks off a postmaster with -F, the tap tests do
so for large portions via fsync=off in the config (c.f. PostgresNode.pm).

Greetings,

Andres Freund

  8. Re: Why do pg_upgrade's test use the serial schedule?

    Michael Paquier <michael@paquier.xyz> — 2018-11-06T00:24:57Z

    On Mon, Nov 05, 2018 at 02:11:42PM -0800, Andres Freund wrote:
> Because just about everything else taking substantial time already
> disables it. pg_regress forks off a postmaster with -F, the tap tests do
> so for large portions via fsync=off in the config (c.f. PostgresNode.pm).

And in most cases people will forget to set up TEMP_CONFIG when setting
up a new environment.  If it is possible to reduce the I/O traffic when
running check-world by default and if we are able to make it faster with
a parallel schedule, my take is to do by default instead of expecting
the user to set up that all the time.  That's less to think about.

So I am +1 and +1 for Andres' suggestions on this thread.
--
Michael

  9. Re: Why do pg_upgrade's test use the serial schedule?

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2018-11-07T07:16:04Z

    On 05/11/2018 22:00, Andres Freund wrote:
> The test I'm most commonly waiting for when doing a parallel make check
> is pg_upgrade. And to a significant degree that's because it uses the
> serial installcheck rather than installcheck-parallel.
> 
> Is there a reason to not use installcheck-parallel?

Probably worth trying.  Using serial was probably just the conservative
choice at the time.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  10. Re: Why do pg_upgrade's test use the serial schedule? (actual thread)

    Andres Freund <andres@anarazel.de> — 2018-11-17T00:45:25Z

    Hi,

I unfortunately copied the wrong message id into a bunch of commit
messages, and just noticed now. Messages pointing to this message's
parent via:
Discussion: https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
ought to have been
Discussion: https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de

Greetings,

Andres Freund