Horton

Thread

Commits

  1. Improve isolation tests infrastructure.

  1. Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-06T22:15:14Z

    Hi,

Right now check-world, a sensible thing to run before commits that
aren't very narrow, takes a long time.  To the point it seems to impact
development velocity enough that it's more sensible to just skip it, and
rely on the buildfarm.

That's not a great solution.

A large amount of the time is actually spent doing completely redundant
initdb, cluster start/stop work, and running tests serially that could
be run in parallel.

A single check-world on my machine takes over 20min.  That's just not
realistic to run without hurting development pace.

We can avoid a lot of redundant work (skip redundant initdb & cluster
start/stop), and we can quite easily parallelize others.

The problem is that doing so isn't something entirely trivially
scriptable, e.g. make installcheck-world doesn't run all tests, and it
doesn't do so in parallel.  Scripting it locally also has the issue that
it's very easy to not notice new stuff being added by others.

As an example of the speedups, here's the comparison for contrib:
make -C contrib:						2m21.056s
make -C contrib installcheck					0m30.672s
make -C contrib -j16 -s -Otarget installcheck USE_MODULE_DB=1	0m10.418s

that's not an entirely fair comparison however, because test_decoding
doesn't to installcheck, but the general principle holds.

This is obviously a large difference.

A lot of the slow tap tests could be run serially, recoverying a lot
more time.

There's also some tests simply taking way too long, e.g. the pg_dump
tests just do largely redundant tests for 30s.

I'm not quite sure what the best way to attack this is, but I think we
need to do something.

Greetings,

Andres Freund

  2. Re: Need a builtin way to run all tests faster manner

    Stephen Frost <sfrost@snowman.net> — 2017-03-06T22:30:11Z

    Andres,

* Andres Freund (andres@anarazel.de) wrote:
> There's also some tests simply taking way too long, e.g. the pg_dump
> tests just do largely redundant tests for 30s.

About half of the time in the pg_dump case, at least, appears to be in
010_dump_connstr.pl.  I've not looked into it, but based on the test
names it does look like some of those tests might be redundant to what
is already being covered in 002_pg_dump.pl.  Of course, the way the TAP
tests run, they require an initdb and startup of the postmaster, and
that's a somewhat fixed amount of time that any TAP test is going to
take.

I'm all for improving things certainly, though, well, while the pg_dump
tests do a lot, they also try to cover a good bit of the code in
pg_dump.c which is quite large.  I wouldn't want to reduce our code
coverage just to make the regression tests go faster.  The changes to
the pg_dump tests that I'm hoping to push soon will at least reduce the
output some, if not the length of time taken, while providing more code
coverage.

All that said, 30s out of 20m (which seems to more-or-less match what I
get locally too) makes me really wonder if that's the place that we need
to focus.  Is it really the longest running test we have and it's just
that we have tons of them that are doing initdb over and over again?

> I'm not quite sure what the best way to attack this is, but I think we
> need to do something.

I tend to agree with this, though I haven't got any great answers,
unfortunately.

Thanks!

Stephen

  3. Re: Need a builtin way to run all tests faster manner

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-07T00:45:27Z

    Stephen Frost <sfrost@snowman.net> writes:
> * Andres Freund (andres@anarazel.de) wrote:
>> I'm not quite sure what the best way to attack this is, but I think we
>> need to do something.

> I tend to agree with this, though I haven't got any great answers,
> unfortunately.

I don't want to reduce test coverage either.  I think the most painless
way to improve matters would just be to work harder on running tests in
parallel.  I think most devs these days do most of their work on 4- or
8-core machines, yet almost everything except the core regression tests
is depressingly serial.  I think we could likely get a 2x or better
reduction in total runtime without too much work just by attacking that.

			regards, tom lane

  4. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-07T00:53:30Z

    On 2017-03-06 19:45:27 -0500, Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Andres Freund (andres@anarazel.de) wrote:
> >> I'm not quite sure what the best way to attack this is, but I think we
> >> need to do something.
>
> > I tend to agree with this, though I haven't got any great answers,
> > unfortunately.
>
> I don't want to reduce test coverage either.  I think the most painless
> way to improve matters would just be to work harder on running tests in
> parallel.  I think most devs these days do most of their work on 4- or
> 8-core machines, yet almost everything except the core regression tests
> is depressingly serial.  I think we could likely get a 2x or better
> reduction in total runtime without too much work just by attacking that.

A lot more probably, based on my preliminary tests / my local test
script.

I'm just not quite sure what the best way is to make it easier to run
tests in parallel within the tree.

The best I can come up so far is a toplevel target that creates the temp
install, starts a cluster and then runs the 'installcheck-or-check'
target on all the subdirectories via recursion. Individual makefiles can
either use the pre-existing cluster (most of of contrib for example), or
use the temporary install and run their pre-existing check target using
that (the tap tests, test_decoding, ...).

Requires editing a bunch of Makefiles to take advantage.  But I don't
really see anything that doesn't require that.

- Andres

  5. Re: Need a builtin way to run all tests faster manner

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-03-07T12:36:51Z

    FWIW, +1 on improving matters here.

Andres Freund wrote:

> The best I can come up so far is a toplevel target that creates the temp
> install, starts a cluster and then runs the 'installcheck-or-check'
> target on all the subdirectories via recursion. Individual makefiles can
> either use the pre-existing cluster (most of of contrib for example), or
> use the temporary install and run their pre-existing check target using
> that (the tap tests, test_decoding, ...).

I think a toplevel installcheck-or-check target is a good first step
(though definitely lets find a better name).  Just being able to run all
tests without the need for 95% of pointless initdb's would be helpful
enough.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  6. Re: Need a builtin way to run all tests faster manner

    Simon Riggs <simon@2ndquadrant.com> — 2017-03-07T12:58:35Z

    On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

> FWIW, +1 on improving matters here.

+1 also.

I don't see what's wrong with relying on buildfarm though; testing is
exactly what its there for.

If we had a two-stage process, where committers can issue "trial
commits" as a way of seeing if the build farm likes things. If they
do, we can push to the main repo.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  7. Re: Need a builtin way to run all tests faster manner

    Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> — 2017-03-07T13:09:29Z

    Simon Riggs <simon@2ndquadrant.com> writes:

> On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>
>> FWIW, +1 on improving matters here.
>
> +1 also.
>
> I don't see what's wrong with relying on buildfarm though; testing is
> exactly what its there for.
>
> If we had a two-stage process, where committers can issue "trial
> commits" as a way of seeing if the build farm likes things. If they
> do, we can push to the main repo.

In perl we do this by having the smoke testers (build farm) pick up
branches with a specific prefix (smoke-me/ in our case, typically
smoke-me/<username>/<topic>), in addition to the blead (master) and
maint-x.y (release) branches.

-- 
"I use RMS as a guide in the same way that a boat captain would use
 a lighthouse.  It's good to know where it is, but you generally
 don't want to find yourself in the same spot." - Tollef Fog Heen

  8. Re: Need a builtin way to run all tests faster manner

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-03-07T15:24:54Z

    
On 03/07/2017 07:58 AM, Simon Riggs wrote:
> On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>
>> FWIW, +1 on improving matters here.
> +1 also.
>
> I don't see what's wrong with relying on buildfarm though; testing is
> exactly what its there for.
>
> If we had a two-stage process, where committers can issue "trial
> commits" as a way of seeing if the build farm likes things. If they
> do, we can push to the main repo.
>


I'm happy to work on this.  Not quite sure how it would work, but I'm
open to any suggestions.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  9. Re: Need a builtin way to run all tests faster manner

    Magnus Hagander <magnus@hagander.net> — 2017-03-08T03:52:04Z

    On Tue, Mar 7, 2017 at 7:24 AM, Andrew Dunstan <
andrew.dunstan@2ndquadrant.com> wrote:

>
>
> On 03/07/2017 07:58 AM, Simon Riggs wrote:
> > On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com>
> wrote:
> >
> >> FWIW, +1 on improving matters here.
> > +1 also.
> >
> > I don't see what's wrong with relying on buildfarm though; testing is
> > exactly what its there for.
> >
> > If we had a two-stage process, where committers can issue "trial
> > commits" as a way of seeing if the build farm likes things. If they
> > do, we can push to the main repo.
> >
>
>
> I'm happy to work on this.  Not quite sure how it would work, but I'm
> open to any suggestions.
>

Assuming the intention is a service for *committers only*, I suggest
setting up a separate (closed) git repository that committers can push to
and a separate set of BF animals could work from. We could just have it do
all branches in it, no need to filter that way as long as we keep it a
separate repo.

There have also on and off been discussions about building arbitrary
patches as they are sent to the mailinglists. Doing that without any
committer (or other trusted party) as a filter is a completely different
challenge of course, given that it basically amounts to downloading and
running random code off the internet.

But doing just the first would make it a lot easier, and probably still be
of good value.

An in-between could be to hook something off the CF app, but one important
question is how important covering many platforms is. Since we already have
good functionality for doing that in the buildfarm, it makes sense to
utilize that if we can.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

  10. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-08T04:23:29Z

    On 2017-03-07 20:58:35 +0800, Simon Riggs wrote:
> On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
> 
> > FWIW, +1 on improving matters here.
> 
> +1 also.
> 
> I don't see what's wrong with relying on buildfarm though; testing is
> exactly what its there for.
> 
> If we had a two-stage process, where committers can issue "trial
> commits" as a way of seeing if the build farm likes things. If they
> do, we can push to the main repo.

Personally that's not addressing my main concern, which is that the
latency of getting done with some patch/topic takes a long while. If I
have to wait for the buildfarm to check some preliminary patch, I still
have to afterwards work on pushing it to master.  And very likely my
local check would finish a lot faster than a bunch of buildfarm animals
- I have after all a plenty powerfull machine, lots of cores, fast ssd,
lots of memory, ...

So I really want faster end-to-end test, not less cpu time spent on my
own machine.

- Andres

  11. Re: Need a builtin way to run all tests faster manner

    Robert Haas <robertmhaas@gmail.com> — 2017-03-08T17:54:21Z

    On Tue, Mar 7, 2017 at 11:23 PM, Andres Freund <andres@anarazel.de> wrote:
> Personally that's not addressing my main concern, which is that the
> latency of getting done with some patch/topic takes a long while. If I
> have to wait for the buildfarm to check some preliminary patch, I still
> have to afterwards work on pushing it to master.  And very likely my
> local check would finish a lot faster than a bunch of buildfarm animals
> - I have after all a plenty powerfull machine, lots of cores, fast ssd,
> lots of memory, ...
>
> So I really want faster end-to-end test, not less cpu time spent on my
> own machine.

Yeah.  I think the buildfarm-for-test-commits or maybe
buildfarm-for-approved-branches-belonging-to-people-we-basically-trust
idea isn't a bad one, but it's not a substitute for $SUBJECT.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  12. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-08T21:32:49Z

    On 3/6/17 19:53, Andres Freund wrote:
> I'm just not quite sure what the best way is to make it easier to run
> tests in parallel within the tree.

make check-world -j2 seems to run fine for me.

With higher -j I appear to be running out of memory or disks space, so I
haven't checked that any further, but it seems possible.

You can also run prove with a -j option.

And we could parallelize some of the contrib/pl tests, e.g., plpython.

(The problem is that parallel make and parallel tests together might
explode a bit, so we might want some way to control which aspect we
parallelize.)

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  13. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-08T21:49:31Z

    On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote:
> On 3/6/17 19:53, Andres Freund wrote:
> > I'm just not quite sure what the best way is to make it easier to run
> > tests in parallel within the tree.
> 
> make check-world -j2 seems to run fine for me.

Hm, I at least used to get a lot of spurious failures with this. I
e.g. don't think the free port selection is race free.  Also that
doesn't solve the issue of the time spent in repetitive initdbs - but
that's mainly in contrib, and we probably solve that there by having a
special target in contrib/ building a cluster once.


> You can also run prove with a -j option.

Ah, interesting.


> (The problem is that parallel make and parallel tests together might
> explode a bit, so we might want some way to control which aspect we
> parallelize.)

Yea, I indeed see that. Probably could do a top-level prerequisite for
check on a check-world subset that includes docs?

- Andres

  14. Re: Need a builtin way to run all tests faster manner

    Michael Paquier <michael.paquier@gmail.com> — 2017-03-09T00:34:13Z

    On Thu, Mar 9, 2017 at 6:49 AM, Andres Freund <andres@anarazel.de> wrote:
> On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote:
>> On 3/6/17 19:53, Andres Freund wrote:
>> > I'm just not quite sure what the best way is to make it easier to run
>> > tests in parallel within the tree.
>>
>> make check-world -j2 seems to run fine for me.
>
> Hm, I at least used to get a lot of spurious failures with this. I
> e.g. don't think the free port selection is race free.

pg_regress is running each Postgres instance in a separate Unix socket
directory, so the port selection is not problem. PostgresNode.pm is
also careful about that.

> Also that
> doesn't solve the issue of the time spent in repetitive initdbs - but
> that's mainly in contrib, and we probably solve that there by having a
> special target in contrib/ building a cluster once.

Yeah, you can count the growing src/test/modules in that as well.
-- 
Michael

  15. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-10T06:59:53Z

    On 3/8/17 16:49, Andres Freund wrote:
>> make check-world -j2 seems to run fine for me.
> 
> Hm, I at least used to get a lot of spurious failures with this. I
> e.g. don't think the free port selection is race free.

I was also not sure about that, but as Michael has pointed out, that
doesn't matter anymore, because it now uses a private socket directory.

I have been pounding it a bit, and every so often the test_decoding
tests fail in mysterious ways, but otherwise it seems to work fine.  I'm
curious what you are seeing.

Combining make -j10 and prove -j4, I get the run time down to 2 minutes
and a bit, from 20+ minutes.  Using the -O option if you have GNU make
>=4 is also useful to get some more sane output.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  16. Re: Need a builtin way to run all tests faster manner

    Jim Nasby <jim.nasby@openscg.com> — 2017-03-10T08:27:00Z

    On 3/7/17 9:52 PM, Magnus Hagander wrote:
> There have also on and off been discussions about building arbitrary
> patches as they are sent to the mailinglists. Doing that without any
> committer (or other trusted party) as a filter is a completely different
> challenge of course, given that it basically amounts to downloading and
> running random code off the internet.

Perhaps https://travis-ci.org/ or something similar could be used for 
this. That avoids any issues about random code.
-- 
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com

  17. Re: Need a builtin way to run all tests faster manner

    Magnus Hagander <magnus@hagander.net> — 2017-03-10T15:52:47Z

    On Fri, Mar 10, 2017 at 3:27 AM, Jim Nasby <jim.nasby@openscg.com> wrote:

> On 3/7/17 9:52 PM, Magnus Hagander wrote:
>
>> There have also on and off been discussions about building arbitrary
>> patches as they are sent to the mailinglists. Doing that without any
>> committer (or other trusted party) as a filter is a completely different
>> challenge of course, given that it basically amounts to downloading and
>> running random code off the internet.
>>
>
> Perhaps https://travis-ci.org/ or something similar could be used for
> this. That avoids any issues about random code.


AFAIK travis-ci would require us to use github as our hoster for all those
things, and embrace that workflow, they don't support anything else.

There might be others that do, just not travis.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

  18. Re: Need a builtin way to run all tests faster manner

    Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> — 2017-03-10T16:29:32Z

    Magnus Hagander <magnus@hagander.net> writes:

> AFAIK travis-ci would require us to use github as our hoster for all those
> things, and embrace that workflow, they don't support anything else.
>
> There might be others that do, just not travis.

It merely requires the repository to exist on GitHub, and postgresql.git
is already mirrored to https://github.com/postgres/postgres.  If there
was a .travis.yml in the repo, people who fork it could easily enable
Travis-CI for their fork, even the official repo isn't hooked up.

- ilmari

-- 
- Twitter seems more influential [than blogs] in the 'gets reported in
  the mainstream press' sense at least.               - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
  to a mainstream media article.                      - Calle Dybedahl

  19. Re: Need a builtin way to run all tests faster manner

    Magnus Hagander <magnus@hagander.net> — 2017-03-10T18:02:08Z

    On Fri, Mar 10, 2017 at 11:29 AM, Dagfinn Ilmari Mannsåker <
ilmari@ilmari.org> wrote:

> Magnus Hagander <magnus@hagander.net> writes:
>
> > AFAIK travis-ci would require us to use github as our hoster for all
> those
> > things, and embrace that workflow, they don't support anything else.
> >
> > There might be others that do, just not travis.
>
> It merely requires the repository to exist on GitHub, and postgresql.git
> is already mirrored to https://github.com/postgres/postgres.  If there
> was a .travis.yml in the repo, people who fork it could easily enable
> Travis-CI for their fork, even the official repo isn't hooked up.
>

That's true. It would require a bunch of additional branches to make it
useful in core though, but it śeems like it could be a useful thing for
people to enable in their own personal forks/branches if they use github
for their largest feature development. Could be a good idea to for example
ave an example yml file somewhere on the wiki that people could put into
their branches.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

  20. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-10T19:09:09Z

    On 3/10/17 13:02, Magnus Hagander wrote:
> for their largest feature development. Could be a good idea to for
> example ave an example yml file somewhere on the wiki that people could
> put into their branches. 

https://github.com/petere/postgresql/blob/travis/.travis.yml

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  21. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-10T19:09:39Z

    On 3/10/17 03:27, Jim Nasby wrote:
> Perhaps https://travis-ci.org/ or something similar could be used for 
> this. That avoids any issues about random code.

That doesn't achieve any platform coverage, which is the main point here.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  22. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-10T19:43:52Z

    On 2017-03-10 01:59:53 -0500, Peter Eisentraut wrote:
> On 3/8/17 16:49, Andres Freund wrote:
> >> make check-world -j2 seems to run fine for me.
> > 
> > Hm, I at least used to get a lot of spurious failures with this. I
> > e.g. don't think the free port selection is race free.
> 
> I was also not sure about that, but as Michael has pointed out, that
> doesn't matter anymore, because it now uses a private socket directory.

Yea, I had forgotten about that bit.


> I have been pounding it a bit, and every so often the test_decoding
> tests fail in mysterious ways, but otherwise it seems to work fine.  I'm
> curious what you are seeing.

I do get regular issues, although the happen to not end up in visible
failures.  All the tests output their regression.diffs into the same
place - which means there'll every now be a failure to remove the file,
and if there were an actual failure it'd possibly end up being
attributed to the wrong test.  So I think we need to relocate that file
to be relative to the sql/ | specs/ | expected/ directories?

Curious about the test decoding thing - hadn't seen that here.

Greetings,

Andres Freund

  23. Re: Need a builtin way to run all tests faster manner

    Jim Nasby <jim.nasby@openscg.com> — 2017-03-10T19:53:12Z

    On 3/10/17 1:09 PM, Peter Eisentraut wrote:
> On 3/10/17 03:27, Jim Nasby wrote:
>> Perhaps https://travis-ci.org/ or something similar could be used for
>> this. That avoids any issues about random code.
>
> That doesn't achieve any platform coverage, which is the main point here.

I don't think platform coverage is the first thing to worry about with 
patches, nor with ongoing development.

The biggest win we'd get from something like Travis would be if the 
commitfest monitored for new patch files coming in for monitored threads 
and it created a new branch, applied the patches, and if they applied 
without error commit the branch and push to let Travis do it's thing. We 
wouldn't want that running in the main git repo, but it should be fine 
in a fork that's dedicated to that purpose.

If the travis build failed, commitfest could notify the author.

It could also rebase master into each branch on a daily basis so authors 
would know very quickly if something got committed that broke their patch.

Obviously that doesn't remove the need for manual testing or the 
buildfarm, but it would at least let everyone know that the patch passed 
a smoke test.
-- 
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com

  24. Re: Need a builtin way to run all tests faster manner

    Magnus Hagander <magnus@hagander.net> — 2017-03-10T20:05:04Z

    On Fri, Mar 10, 2017 at 2:53 PM, Jim Nasby <jim.nasby@openscg.com> wrote:

> On 3/10/17 1:09 PM, Peter Eisentraut wrote:
>
>> On 3/10/17 03:27, Jim Nasby wrote:
>>
>>> Perhaps https://travis-ci.org/ or something similar could be used for
>>> this. That avoids any issues about random code.
>>>
>>
>> That doesn't achieve any platform coverage, which is the main point here.
>>
>
> I don't think platform coverage is the first thing to worry about with
> patches, nor with ongoing development.
>

I think we are talking about solving two different problems...


>
> The biggest win we'd get from something like Travis would be if the
> commitfest monitored for new patch files coming in for monitored threads
> and it created a new branch, applied the patches, and if they applied
> without error commit the branch and push to let Travis do it's thing. We
> wouldn't want that running in the main git repo, but it should be fine in a
> fork that's dedicated to that purpose.
>

Travis specifically would not help us with this, due to the dependency on
gifhub, but something that knows how to run "patch ... && configure && make
&& make check" in a container would.

I'm unsure what would be easiest -- have something drive a "throwaway
github repo" off the data in the CF app and try to pull things from there,
or to just spawn containers and run it directly without travis.

The bigger issue with those is the usual -- how do you handle patches that
have dependencies on each other,because they're always going to show up as
broken individually. I guess we could tell people doing those to just push
a git branch on github and register that one in the CF app (which does have
some very basic support for tracking that, but I doubt anybody uses it
today).



> If the travis build failed, commitfest could notify the author.
>
> It could also rebase master into each branch on a daily basis so authors
> would know very quickly if something got committed that broke their patch.
>

It could at least verify that the patch still applies, yes.



> Obviously that doesn't remove the need for manual testing or the
> buildfarm, but it would at least let everyone know that the patch passed a
> smoke test


It's definitely something that would be useful, but as you say, it solves a
different problem.

//Magnus

  25. Re: Need a builtin way to run all tests faster manner

    Jim Nasby <jim.nasby@openscg.com> — 2017-03-10T20:11:07Z

    On 3/10/17 2:05 PM, Magnus Hagander wrote:
>
> Travis specifically would not help us with this, due to the dependency
> on gifhub, but something that knows how to run "patch ... && configure
> && make && make check" in a container would.

Who's updating https://github.com/postgres/postgres/ right now? 
Presumably that script would be the basis for this...

> I'm unsure what would be easiest -- have something drive a "throwaway
> github repo" off the data in the CF app and try to pull things from
> there, or to just spawn containers and run it directly without travis.

I'd be a bit nervous about creating our own container solution and 
opening that to automatically deploying patches. Travis (and other 
tools) already have that problem solved (or at least if they get hacked 
it's on them to clean up and not us :)

Plus it'd be a heck of a lot more work on our side to set all that stuff up.

> The bigger issue with those is the usual -- how do you handle patches
> that have dependencies on each other,because they're always going to
> show up as broken individually. I guess we could tell people doing those
> to just push a git branch on github and register that one in the CF app
> (which does have some very basic support for tracking that, but I doubt
> anybody uses it today).

If people use git format-patch it should JustWork(tm). Specifying a 
specific repo is another option.

Even if we can't make it work for really complicated patches it might 
still be a win.

>     If the travis build failed, commitfest could notify the author.
>
>     It could also rebase master into each branch on a daily basis so
>     authors would know very quickly if something got committed that
>     broke their patch.
>
>
> It could at least verify that the patch still applies, yes.

If the rebase was pushed to github and travis was setup, travis would 
then test the changes as well.
-- 
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com

  26. Re: Need a builtin way to run all tests faster manner

    Magnus Hagander <magnus@hagander.net> — 2017-03-10T20:18:27Z

    On Fri, Mar 10, 2017 at 3:11 PM, Jim Nasby <jim.nasby@openscg.com> wrote:

> On 3/10/17 2:05 PM, Magnus Hagander wrote:
>
>>
>> Travis specifically would not help us with this, due to the dependency
>> on gifhub, but something that knows how to run "patch ... && configure
>> && make && make check" in a container would.
>>
>
> Who's updating https://github.com/postgres/postgres/ right now?
> Presumably that script would be the basis for this...


Yes. It's a pure mirror script.

I'm pretty sure we *don't* want to push a branch for every single patch
that goes in a CF all into our master repository...

We could use a completely separate repo on github for it if we want, yes.
Then we just need to figure out how to get the patches there..


>
> I'm unsure what would be easiest -- have something drive a "throwaway
>> github repo" off the data in the CF app and try to pull things from
>> there, or to just spawn containers and run it directly without travis.
>>
>
> I'd be a bit nervous about creating our own container solution and opening
> that to automatically deploying patches. Travis (and other tools) already
> have that problem solved (or at least if they get hacked it's on them to
> clean up and not us :)
>
> Plus it'd be a heck of a lot more work on our side to set all that stuff
> up.


This is what I'm not so sure about. Setting up an empty container provided
we only ever need to test the one thing and only ever need the one platform
is not particularly complicated.

But if you can put together something that picks up the individual patches
out of the mail threads in the CF app and keeps branch-tips in a git repo
up-to-date with those, including feeding the results back into the app,
then go for it :)




> The bigger issue with those is the usual -- how do you handle patches
>> that have dependencies on each other,because they're always going to
>> show up as broken individually. I guess we could tell people doing those
>> to just push a git branch on github and register that one in the CF app
>> (which does have some very basic support for tracking that, but I doubt
>> anybody uses it today).
>>
>
> If people use git format-patch it should JustWork(tm). Specifying a
> specific repo is another option.
>

Right. But people don't use that all the time, and it's not currently a
requirement on patch submitters. and we've traditionally been of the
opinion that we don't want to put too much requirements on such things for
submitters.



> Even if we can't make it work for really complicated patches it might
> still be a win.


Definitely as long as we can keep some manual process for thos epatches.



>     If the travis build failed, commitfest could notify the author.
>>
>>     It could also rebase master into each branch on a daily basis so
>>     authors would know very quickly if something got committed that
>>     broke their patch.
>>
>>
>> It could at least verify that the patch still applies, yes.
>>
>
> If the rebase was pushed to github and travis was setup, travis would then
> test the changes as well.
>
>
Right.

//Magnus

  27. Re: Need a builtin way to run all tests faster manner

    Jim Nasby <jim.nasby@openscg.com> — 2017-03-10T21:45:20Z

    On 3/10/17 2:18 PM, Magnus Hagander wrote:
> But if you can put together something that picks up the individual
> patches out of the mail threads in the CF app and keeps branch-tips in a
> git repo up-to-date with those, including feeding the results back into
> the app, then go for it :)

Seems like an ideal project for someone not on -hackers... do we have a 
list of "How you can help Postgres besides hacking database code" anywhere?
-- 
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com

  28. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-10T23:57:00Z

    On 3/10/17 14:53, Jim Nasby wrote:
> The biggest win we'd get from something like Travis would be if the 
> commitfest monitored for new patch files coming in for monitored threads 
> and it created a new branch, applied the patches, and if they applied 
> without error commit the branch and push to let Travis do it's thing. We 
> wouldn't want that running in the main git repo, but it should be fine 
> in a fork that's dedicated to that purpose.

This has been discussed several times before, e.g.,

https://www.postgresql.org/message-id/54DD2413.8030201@gmx.net

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  29. Re: Need a builtin way to run all tests faster manner

    Jim Nasby <jim.nasby@openscg.com> — 2017-03-11T00:00:31Z

    On 3/10/17 5:57 PM, Peter Eisentraut wrote:
> On 3/10/17 14:53, Jim Nasby wrote:
>> The biggest win we'd get from something like Travis would be if the
>> commitfest monitored for new patch files coming in for monitored threads
>> and it created a new branch, applied the patches, and if they applied
>> without error commit the branch and push to let Travis do it's thing. We
>> wouldn't want that running in the main git repo, but it should be fine
>> in a fork that's dedicated to that purpose.
>
> This has been discussed several times before, e.g.,
>
> https://www.postgresql.org/message-id/54DD2413.8030201@gmx.net

Maybe instead of having the commitfest app try and divine patches from 
the list it should be able to send patches to the list from a specified 
git repo/branch. Anyone that provides that info would have tests run 
automagically, patches sent, etc. Anyone who doesn't can just keep using 
the old process.
-- 
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com

  30. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-11T00:04:39Z

    On 3/10/17 14:43, Andres Freund wrote:
> I do get regular issues, although the happen to not end up in visible
> failures.  All the tests output their regression.diffs into the same
> place - which means there'll every now be a failure to remove the file,
> and if there were an actual failure it'd possibly end up being
> attributed to the wrong test.  So I think we need to relocate that file
> to be relative to the sql/ | specs/ | expected/ directories?

I'm confused here.  Every test suite runs in a separate directory, and
for the tests in the same suite pg_regress will take care of running the
tests in parallel and producing the diffs.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  31. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-11T00:06:27Z

    On 3/10/17 19:00, Jim Nasby wrote:
> Maybe instead of having the commitfest app try and divine patches from 
> the list it should be able to send patches to the list from a specified 
> git repo/branch. Anyone that provides that info would have tests run 
> automagically, patches sent, etc. Anyone who doesn't can just keep using 
> the old process.

Those people who know what they're doing will presumably run all those
checks before they submit a patch.  It's those people who send in
patches that don't apply cleanly or fail the tests that would benefit
from this system.  But if they're that careless, then they also won't
take care to use this particular system correctly.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  32. Re: Need a builtin way to run all tests faster manner

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-11T02:10:16Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>>>> make check-world -j2 seems to run fine for me.

> I have been pounding it a bit, and every so often the test_decoding
> tests fail in mysterious ways, but otherwise it seems to work fine.  I'm
> curious what you are seeing.

For me, it's quite unreliable at make parallelism above -j2 or so.

I believe the core problem is that contrib/test_decoding's regresscheck
and isolationcheck targets each want to use ./tmp_check as their
--temp-instance.  make has no reason to believe it shouldn't run those
two sub-jobs in parallel, but when it does, we get two postmasters trying
to share the same directory.  This looks reasonably straightforward to
solve, but I'm not entirely familiar with the code here, and am not
sure what is the least ugly way to fix it.

			regards, tom lane

  33. Re: Need a builtin way to run all tests faster manner

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-11T17:05:23Z

    I wrote:
> I believe the core problem is that contrib/test_decoding's regresscheck
> and isolationcheck targets each want to use ./tmp_check as their
> --temp-instance.  make has no reason to believe it shouldn't run those
> two sub-jobs in parallel, but when it does, we get two postmasters trying
> to share the same directory.  This looks reasonably straightforward to
> solve, but I'm not entirely familiar with the code here, and am not
> sure what is the least ugly way to fix it.

Enlarging on that: if I cd into contrib/test_decoding and do
"make check -j4" or so, it reliably fails.  With the attached patch,
it passes.  This is a localized patch that only fixes things for
contrib/test_decoding; what I'm wondering is if it would be better to
establish a more widespread convention that $(pg_isolation_regress_check)
should use a different --temp-instance directory than $(pg_regress_check)
does.

			regards, tom lane

  34. Re: Need a builtin way to run all tests faster manner

    Jim Nasby <jim.nasby@openscg.com> — 2017-03-11T19:19:35Z

    On 3/10/17 6:06 PM, Peter Eisentraut wrote:
> On 3/10/17 19:00, Jim Nasby wrote:
>> Maybe instead of having the commitfest app try and divine patches from
>> the list it should be able to send patches to the list from a specified
>> git repo/branch. Anyone that provides that info would have tests run
>> automagically, patches sent, etc. Anyone who doesn't can just keep using
>> the old process.
>
> Those people who know what they're doing will presumably run all those
> checks before they submit a patch.  It's those people who send in
> patches that don't apply cleanly or fail the tests that would benefit
> from this system.  But if they're that careless, then they also won't
> take care to use this particular system correctly.

It's actually a lot harder to mess up providing a git repo link than 
manually submitting patches to the mailing list. For most patches, it's 
also a hell of a lot faster to just submit a repo URL rather than 
dealing with patch files. Having this also means that reviewers can 
focus more on what the patch is actually doing instead of mechanical 
crap best left to a machine.

Of course, *you* work on changes that are far more complex than any 
newbie will, and it wouldn't surprise me if such a feature wouldn't help 
you or other senior hackers at all. But AFAICT it wouldn't get in your 
way either. It would remove yet another burden for new hackers.

Anyway, this is well off topic for the original thread...
-- 
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com

  35. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-11T19:48:31Z

    On 2017-03-11 12:05:23 -0500, Tom Lane wrote:
> I wrote:
> > I believe the core problem is that contrib/test_decoding's regresscheck
> > and isolationcheck targets each want to use ./tmp_check as their
> > --temp-instance.  make has no reason to believe it shouldn't run those
> > two sub-jobs in parallel, but when it does, we get two postmasters trying
> > to share the same directory.  This looks reasonably straightforward to
> > solve, but I'm not entirely familiar with the code here, and am not
> > sure what is the least ugly way to fix it.
> 
> Enlarging on that: if I cd into contrib/test_decoding and do
> "make check -j4" or so, it reliably fails.

Yep, can reproduce here as well. Interesting that, with -j16, I could
survive several dozen runs from the toplevel locally.


> This is a localized patch that only fixes things for
> contrib/test_decoding; what I'm wondering is if it would be better to
> establish a more widespread convention that
> $(pg_isolation_regress_check) should use a different --temp-instance
> directory than $(pg_regress_check) does.

I think that'd be a good plan.  We probably should also keep --outputdir
seperate (which test_decoding/Makefile does, but
pg_isolation_regress_check doesn't)?

- Andres

  36. Re: Need a builtin way to run all tests faster manner

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-11T20:06:47Z

    Jim Nasby <jim.nasby@openscg.com> writes:
> It's actually a lot harder to mess up providing a git repo link than 
> manually submitting patches to the mailing list.

Yeah, we've heard that proposal before.  We're still not doing it though.
Insisting on patches being actually submitted to the mailing list is
important for archival and possibly legal reasons.  If someone sends
in a link to $random-repo, once that site goes away there's no way to
determine exactly what was submitted.

			regards, tom lane

  37. Re: Need a builtin way to run all tests faster manner

    Jim Nasby <jim.nasby@openscg.com> — 2017-03-11T20:13:09Z

    On 3/11/17 2:06 PM, Tom Lane wrote:
> Jim Nasby <jim.nasby@openscg.com> writes:
>> It's actually a lot harder to mess up providing a git repo link than
>> manually submitting patches to the mailing list.
> Yeah, we've heard that proposal before.  We're still not doing it though.
> Insisting on patches being actually submitted to the mailing list is
> important for archival and possibly legal reasons.  If someone sends
> in a link to $random-repo, once that site goes away there's no way to
> determine exactly what was submitted.

The full proposal was that the commitfest app have the ability to 
generate and post the patch for you, assuming that the smoke-test passes.
-- 
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com

  38. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-11T22:17:42Z

    On 2017-03-07 09:36:51 -0300, Alvaro Herrera wrote:
> FWIW, +1 on improving matters here.
>
> Andres Freund wrote:
>
> > The best I can come up so far is a toplevel target that creates the temp
> > install, starts a cluster and then runs the 'installcheck-or-check'
> > target on all the subdirectories via recursion. Individual makefiles can
> > either use the pre-existing cluster (most of of contrib for example), or
> > use the temporary install and run their pre-existing check target using
> > that (the tap tests, test_decoding, ...).
>
> I think a toplevel installcheck-or-check target is a good first step
> (though definitely lets find a better name).  Just being able to run all
> tests without the need for 95% of pointless initdb's would be helpful
> enough.

Here's a hacky proof-of-concept patch that implements a contrib/
'fastcheck' target.
timing of:

make -s -j01 -Otarget check				2m49.286s
make -s -j16 -Otarget check				0m44.120s
make -s -j01 -Otarget fastcheck				1m1.533s
make -s -j16 -Otarget fastcheck				0m24.385s
make -s -j01 -Otarget USE_MODULE_DB=1 \
	installcheck check-test_decoding-recurse	0m56.016s
make -s -j16 -Otarget USE_MODULE_DB=1 \
	installcheck check-test_decoding-recurse	0m23.608s

All of these should, excepting rerunning initdb/server start, have
equivalent test coverage.

'fastcheck' is obviously, if you look at it, a POC. We'd need:
- abstract this away somehow, it's not nice to copy the logic into a
  makefile
- tie contrib/'fastcheck' into check-world by overriding that target
- use the temp install we've created previously?
- more bulletproof server stop mechanism?

- Andres

  39. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-11T22:36:56Z

    On 2017-03-11 14:17:42 -0800, Andres Freund wrote:
> On 2017-03-07 09:36:51 -0300, Alvaro Herrera wrote:
> > FWIW, +1 on improving matters here.
> >
> > Andres Freund wrote:
> >
> > > The best I can come up so far is a toplevel target that creates the temp
> > > install, starts a cluster and then runs the 'installcheck-or-check'
> > > target on all the subdirectories via recursion. Individual makefiles can
> > > either use the pre-existing cluster (most of of contrib for example), or
> > > use the temporary install and run their pre-existing check target using
> > > that (the tap tests, test_decoding, ...).
> >
> > I think a toplevel installcheck-or-check target is a good first step
> > (though definitely lets find a better name).  Just being able to run all
> > tests without the need for 95% of pointless initdb's would be helpful
> > enough.
> 
> Here's a hacky proof-of-concept patch that implements a contrib/
> 'fastcheck' target.
> timing of:
> 
> make -s -j01 -Otarget check				2m49.286s
> make -s -j16 -Otarget check				0m44.120s
> make -s -j01 -Otarget fastcheck				1m1.533s
> make -s -j16 -Otarget fastcheck				0m24.385s
> make -s -j01 -Otarget USE_MODULE_DB=1 \
> 	installcheck check-test_decoding-recurse	0m56.016s
> make -s -j16 -Otarget USE_MODULE_DB=1 \
> 	installcheck check-test_decoding-recurse	0m23.608s

Ooops - all these timings are from a coverage enabled build - the times
are overall a bit smaller without that.

- Andres

  40. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-11T23:17:42Z

    On 2017-03-11 11:48:31 -0800, Andres Freund wrote:
> On 2017-03-11 12:05:23 -0500, Tom Lane wrote:
> > I wrote:
> > > I believe the core problem is that contrib/test_decoding's regresscheck
> > > and isolationcheck targets each want to use ./tmp_check as their
> > > --temp-instance.  make has no reason to believe it shouldn't run those
> > > two sub-jobs in parallel, but when it does, we get two postmasters trying
> > > to share the same directory.  This looks reasonably straightforward to
> > > solve, but I'm not entirely familiar with the code here, and am not
> > > sure what is the least ugly way to fix it.
> > 
> > Enlarging on that: if I cd into contrib/test_decoding and do
> > "make check -j4" or so, it reliably fails.
> 
> Yep, can reproduce here as well. Interesting that, with -j16, I could
> survive several dozen runs from the toplevel locally.
> 
> 
> > This is a localized patch that only fixes things for
> > contrib/test_decoding; what I'm wondering is if it would be better to
> > establish a more widespread convention that
> > $(pg_isolation_regress_check) should use a different --temp-instance
> > directory than $(pg_regress_check) does.
> 
> I think that'd be a good plan.  We probably should also keep --outputdir
> seperate (which test_decoding/Makefile does, but
> pg_isolation_regress_check doesn't)?

Here's a patch doing that (based on yours).  I'd be kind of inclined to
set --outputdir for !isolation tests too; possibly even move tmp_check
below output_iso/ output_regress/ or such - but that seems like it
potentially could cause some disagreement...

- Andres

  41. Re: Need a builtin way to run all tests faster manner

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-12T03:14:07Z

    Andres Freund <andres@anarazel.de> writes:
> On 2017-03-11 11:48:31 -0800, Andres Freund wrote:
>> I think that'd be a good plan.  We probably should also keep --outputdir
>> seperate (which test_decoding/Makefile does, but
>> pg_isolation_regress_check doesn't)?

> Here's a patch doing that (based on yours).  I'd be kind of inclined to
> set --outputdir for !isolation tests too; possibly even move tmp_check
> below output_iso/ output_regress/ or such - but that seems like it
> potentially could cause some disagreement...

This looks generally sane to me, although I'm not very happy about folding
the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
seems weird and unlike the way it's done for the regular regression test
case.  But probably Peter has a better-informed opinion.

			regards, tom lane

  42. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-13T02:58:33Z

    0;115;0c
On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2017-03-11 11:48:31 -0800, Andres Freund wrote:
> >> I think that'd be a good plan.  We probably should also keep --outputdir
> >> seperate (which test_decoding/Makefile does, but
> >> pg_isolation_regress_check doesn't)?
> 
> > Here's a patch doing that (based on yours).  I'd be kind of inclined to
> > set --outputdir for !isolation tests too; possibly even move tmp_check
> > below output_iso/ output_regress/ or such - but that seems like it
> > potentially could cause some disagreement...
> 
> This looks generally sane to me, although I'm not very happy about folding
> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
> seems weird and unlike the way it's done for the regular regression test
> case.

Yea, not super happy about that either - alternatively we could fold it
into pg_regress.  But given the way that prove_checks works, I thought
it'd not be too ugly comparatively.

- Andres

  43. Re: Need a builtin way to run all tests faster manner

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-13T04:35:06Z

    Andres Freund <andres@anarazel.de> writes:
> On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
>> This looks generally sane to me, although I'm not very happy about folding
>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
>> seems weird and unlike the way it's done for the regular regression test
>> case.

> Yea, not super happy about that either - alternatively we could fold it
> into pg_regress.

Yeah, teaching pg_regress to auto-create the --temp-instance directory
seems perfectly sane from here.

			regards, tom lane

  44. Re: Need a builtin way to run all tests faster manner

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-03-13T23:21:22Z

    
On 03/13/2017 12:35 AM, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
>> On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
>>> This looks generally sane to me, although I'm not very happy about folding
>>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
>>> seems weird and unlike the way it's done for the regular regression test
>>> case.
>> Yea, not super happy about that either - alternatively we could fold it
>> into pg_regress.
> Yeah, teaching pg_regress to auto-create the --temp-instance directory
> seems perfectly sane from here.


w.r.t. $subject, I thought it might be useful to get some recent stats
from the buildfarm. Results are below. The bin checks dwarf everything
else. Upgrade checks and isolation check are other items of significant
cost. Upgrade checks could be significantly shortened if we could avoid
rerunning the regression tests.

cheers

andrew


pgbfprod=> select s.branch, s.log_stage, count(*), avg(extract(epoch
from stage_duration)::numeric(15,2))::numeric(15,1),
stddev(extract(epoch from stage_duration)::numeric(15,2))::numeric(15,1)
from public.build_status_log s where sysname <> 'crake' and snapshot >
now() - interval '10 days'  and log_stage !~ 'start|stop' group by
s.branch, s.log_stage having count(*) > 20 and avg(extract(epoch from
stage_duration)::numeric(15,2)) > 20 order by log_stage, branch <>
'HEAD', branch desc
;
    branch     |                   log_stage                   | count
|  avg   | stddev
---------------+-----------------------------------------------+-------+--------+--------
 HEAD          | bin-check.log                                 |   388 |
1739.0 | 1758.5
 REL9_6_STABLE | bin-check.log                                 |    91 |
1430.5 | 1287.9
 REL9_5_STABLE | bin-check.log                                 |    87 |
1140.0 |  994.1
 REL9_4_STABLE | bin-check.log                                 |    72
|  751.0 |  666.8
 HEAD          | check.log                                     |  2305
|  263.1 | 1197.0
 REL9_6_STABLE | check.log                                     |   610
|  294.7 | 1369.6
 REL9_5_STABLE | check.log                                     |   627
|  170.1 |  819.6
 REL9_4_STABLE | check.log                                     |   512
|  140.4 |  535.3
 REL9_3_STABLE | check.log                                     |   449
|  112.0 |  446.0
 REL9_2_STABLE | check.log                                     |   406
|  109.2 |  380.9
 HEAD          | check-pg_upgrade.log                          |  1785
|  319.4 | 1310.5
 REL9_6_STABLE | check-pg_upgrade.log                          |   482
|  571.3 | 2811.0
 REL9_5_STABLE | check-pg_upgrade.log                          |   484
|  350.5 | 2160.3
 REL9_4_STABLE | check-pg_upgrade.log                          |   385
|  240.8 | 1278.9
 REL9_3_STABLE | check-pg_upgrade.log                          |   353
|  214.0 | 1188.3
 REL9_2_STABLE | check-pg_upgrade.log                          |   314
|  195.6 | 1016.6
 HEAD          | config.log                                    |  2216
|   84.5 |  101.5
 REL9_6_STABLE | config.log                                    |   576
|   90.0 |   90.8
 REL9_5_STABLE | config.log                                    |   584
|  114.0 |  358.8
 REL9_4_STABLE | config.log                                    |   495
|   84.5 |   85.3
 REL9_3_STABLE | config.log                                    |   431
|   97.9 |  100.7
 REL9_2_STABLE | config.log                                    |   391
|   93.1 |   94.6
 HEAD          | contrib-install-check-C.log                   |  2250
|  122.9 |  474.5
 REL9_6_STABLE | contrib-install-check-C.log                   |   606
|  124.6 |  410.3
 REL9_5_STABLE | contrib-install-check-C.log                   |   622
|   84.7 |  348.8
 REL9_4_STABLE | contrib-install-check-C.log                   |   508
|  105.9 |  434.1
 REL9_3_STABLE | contrib-install-check-C.log                   |   445
|   61.8 |  273.4
 REL9_2_STABLE | contrib-install-check-C.log                   |   403
|   54.3 |  205.7
 HEAD          | contrib-install-check-cs_CZ.UTF-8.log         |   184
|   25.7 |   11.8
 REL9_6_STABLE | contrib-install-check-cs_CZ.UTF-8.log         |    46
|   25.7 |   14.0
 REL9_5_STABLE | contrib-install-check-cs_CZ.UTF-8.log         |    42
|   23.8 |   16.6
 REL9_4_STABLE | contrib-install-check-cs_CZ.UTF-8.log         |    36
|   22.9 |   12.4
 HEAD          | contrib-install-check-en_US.8859-15.log       |    37
|  173.9 |   32.3
 HEAD          | contrib-install-check-en_US.ISO8859-1.log     |    33
|  244.7 |   35.7
 HEAD          | contrib-install-check-en_US.log               |   171
|   65.9 |  101.6
 REL9_6_STABLE | contrib-install-check-en_US.log               |    42
|   61.7 |   89.3
 REL9_5_STABLE | contrib-install-check-en_US.log               |    37
|   54.3 |   79.3
 REL9_4_STABLE | contrib-install-check-en_US.log               |    33
|   53.7 |   71.5
 REL9_3_STABLE | contrib-install-check-en_US.log               |    32
|   57.8 |   86.6
 REL9_2_STABLE | contrib-install-check-en_US.log               |    29
|   46.9 |   63.8
 HEAD          | contrib-install-check-en_US.utf8.log          |   142
|   29.2 |   31.4
 REL9_6_STABLE | contrib-install-check-en_US.utf8.log          |    30
|   40.2 |   43.7
 REL9_5_STABLE | contrib-install-check-en_US.utf8.log          |    29
|   41.8 |   51.6
 REL9_4_STABLE | contrib-install-check-en_US.utf8.log          |    25
|   56.6 |   75.8
 REL9_3_STABLE | contrib-install-check-en_US.utf8.log          |    24
|   35.7 |   40.4
 REL9_2_STABLE | contrib-install-check-en_US.utf8.log          |    21
|   31.3 |   36.1
 HEAD          | contrib-install-check-en_US.UTF-8.log         |   188
|   79.7 |   73.8
 REL9_6_STABLE | contrib-install-check-en_US.UTF-8.log         |    36
|   78.2 |   68.6
 REL9_5_STABLE | contrib-install-check-en_US.UTF-8.log         |    34
|   64.9 |   53.7
 REL9_4_STABLE | contrib-install-check-en_US.UTF-8.log         |    30
|   68.4 |   56.2
 REL9_3_STABLE | contrib-install-check-en_US.UTF-8.log         |    28
|   54.6 |   46.1
 REL9_2_STABLE | contrib-install-check-en_US.UTF-8.log         |    25
|   48.3 |   39.8
 HEAD          | contrib-install-check-POSIX.log               |    33
|  227.7 |   42.6
 HEAD          | ecpg-check.log                                |  2229
|   88.5 |  212.4
 REL9_6_STABLE | ecpg-check.log                                |   586
|   97.5 |  139.6
 REL9_5_STABLE | ecpg-check.log                                |   591
|   92.5 |  183.5
 REL9_4_STABLE | ecpg-check.log                                |   491
|   82.1 |  117.1
 REL9_3_STABLE | ecpg-check.log                                |   428
|   77.3 |  102.8
 REL9_2_STABLE | ecpg-check.log                                |   388
|   80.7 |   98.3
 REL9_6_STABLE | initdb-C.log                                  |   606
|   21.2 |   73.1
 HEAD          | initdb-en_US.8859-15.log                      |    37
|  241.8 |   24.8
 HEAD          | initdb-en_US.ISO8859-1.log                    |    33
|  251.6 |   39.1
 HEAD          | initdb-en_US.log                              |   171
|   54.3 |  103.8
 REL9_6_STABLE | initdb-en_US.log                              |    42
|   54.9 |   99.7
 REL9_5_STABLE | initdb-en_US.log                              |    37
|   61.2 |  112.5
 REL9_4_STABLE | initdb-en_US.log                              |    33
|   47.5 |   78.5
 REL9_3_STABLE | initdb-en_US.log                              |    32
|   46.0 |   74.6
 REL9_2_STABLE | initdb-en_US.log                              |    29
|   41.1 |   62.9
 HEAD          | initdb-POSIX.log                              |    33
|  251.1 |   31.6
 HEAD          | install-check-C.log                           |  2241
|  229.5 | 1227.9
 REL9_6_STABLE | install-check-C.log                           |   606
|  299.0 | 1601.9
 REL9_5_STABLE | install-check-C.log                           |   622
|  173.7 | 1147.2
 REL9_4_STABLE | install-check-C.log                           |   508
|  139.3 |  796.4
 REL9_3_STABLE | install-check-C.log                           |   446
|  104.7 |  713.3
 REL9_2_STABLE | install-check-C.log                           |   403
|   91.3 |  574.6
 HEAD          | install-check-cs_CZ.ISO-8859-2.log            |   137
|   37.5 |    1.5
 REL9_6_STABLE | install-check-cs_CZ.ISO-8859-2.log            |    33
|   30.8 |    1.5
 REL9_5_STABLE | install-check-cs_CZ.ISO-8859-2.log            |    29
|   31.0 |    1.7
 REL9_4_STABLE | install-check-cs_CZ.ISO-8859-2.log            |    25
|   26.2 |    0.4
 REL9_3_STABLE | install-check-cs_CZ.ISO-8859-2.log            |    24
|   24.9 |    1.2
 REL9_2_STABLE | install-check-cs_CZ.ISO-8859-2.log            |    21
|   22.6 |    0.7
 HEAD          | install-check-cs_CZ.UTF-8.log                 |   184
|   43.0 |    9.7
 REL9_6_STABLE | install-check-cs_CZ.UTF-8.log                 |    46
|   39.1 |   15.1
 REL9_5_STABLE | install-check-cs_CZ.UTF-8.log                 |    42
|   39.4 |   14.6
 REL9_4_STABLE | install-check-cs_CZ.UTF-8.log                 |    36
|   32.3 |    9.5
 REL9_3_STABLE | install-check-cs_CZ.UTF-8.log                 |    34
|   29.7 |    8.2
 REL9_2_STABLE | install-check-cs_CZ.UTF-8.log                 |    30
|   27.6 |    9.2
 HEAD          | install-check-cs_CZ.WIN-1250.log              |   137
|   37.3 |    1.2
 REL9_6_STABLE | install-check-cs_CZ.WIN-1250.log              |    33
|   30.5 |    1.3
 REL9_5_STABLE | install-check-cs_CZ.WIN-1250.log              |    29
|   31.0 |    1.7
 REL9_4_STABLE | install-check-cs_CZ.WIN-1250.log              |    25
|   26.3 |    0.7
 REL9_3_STABLE | install-check-cs_CZ.WIN-1250.log              |    24
|   24.9 |    1.4
 REL9_2_STABLE | install-check-cs_CZ.WIN-1250.log              |    21
|   22.3 |    0.6
 HEAD          | install-check-de_DE.utf8.log                  |   129
|   35.4 |    9.4
 REL9_6_STABLE | install-check-de_DE.utf8.log                  |    22
|   25.8 |    7.8
 REL9_5_STABLE | install-check-de_DE.utf8.log                  |    21
|   27.4 |    8.0
 HEAD          | install-check-en_US.8859-15.log               |    37
|  136.9 |   21.2
 HEAD          | install-check-en_US.ISO8859-1.log             |    33
|  319.0 |   29.4
 HEAD          | install-check-en_US.log                       |   171
|   65.7 |   58.7
 REL9_6_STABLE | install-check-en_US.log                       |    42
|   62.7 |   69.4
 REL9_5_STABLE | install-check-en_US.log                       |    37
|   59.4 |   56.2
 REL9_4_STABLE | install-check-en_US.log                       |    33
|   50.6 |   44.7
 REL9_3_STABLE | install-check-en_US.log                       |    32
|   50.2 |   46.8
 REL9_2_STABLE | install-check-en_US.log                       |    29
|   45.1 |   39.2
 HEAD          | install-check-en_US.utf8.log                  |   142
|   47.6 |   40.3
 REL9_6_STABLE | install-check-en_US.utf8.log                  |    30
|   58.1 |   54.4
 REL9_5_STABLE | install-check-en_US.utf8.log                  |    29
|   61.7 |   57.4
 REL9_4_STABLE | install-check-en_US.utf8.log                  |    25
|   57.2 |   59.9
 REL9_3_STABLE | install-check-en_US.utf8.log                  |    24
|   51.3 |   47.7
 REL9_2_STABLE | install-check-en_US.utf8.log                  |    21
|   45.2 |   41.9
 HEAD          | install-check-en_US.UTF-8.log                 |   188
|  153.5 |  162.7
 REL9_6_STABLE | install-check-en_US.UTF-8.log                 |    36
|  140.8 |  140.9
 REL9_5_STABLE | install-check-en_US.UTF-8.log                 |    34
|  145.7 |  144.3
 REL9_4_STABLE | install-check-en_US.UTF-8.log                 |    30
|  118.9 |  113.1
 REL9_3_STABLE | install-check-en_US.UTF-8.log                 |    28
|  107.0 |  101.8
 REL9_2_STABLE | install-check-en_US.UTF-8.log                 |    25
|   93.6 |   87.8
 HEAD          | install-check-POSIX.log                       |    33
|  189.5 |   76.5
 HEAD          | install-check-sk_SK.ISO-8859-2.log            |   137
|   37.4 |    1.2
 REL9_6_STABLE | install-check-sk_SK.ISO-8859-2.log            |    33
|   30.5 |    1.3
 REL9_5_STABLE | install-check-sk_SK.ISO-8859-2.log            |    29
|   30.9 |    1.5
 REL9_4_STABLE | install-check-sk_SK.ISO-8859-2.log            |    25
|   26.2 |    0.4
 REL9_3_STABLE | install-check-sk_SK.ISO-8859-2.log            |    24
|   24.8 |    1.2
 REL9_2_STABLE | install-check-sk_SK.ISO-8859-2.log            |    21
|   22.7 |    0.6
 HEAD          | install-check-sk_SK.UTF-8.log                 |   137
|   37.6 |    1.1
 REL9_6_STABLE | install-check-sk_SK.UTF-8.log                 |    33
|   31.0 |    1.8
 REL9_5_STABLE | install-check-sk_SK.UTF-8.log                 |    29
|   31.3 |    1.8
 REL9_4_STABLE | install-check-sk_SK.UTF-8.log                 |    25
|   26.2 |    0.4
 REL9_3_STABLE | install-check-sk_SK.UTF-8.log                 |    24
|   25.0 |    1.2
 REL9_2_STABLE | install-check-sk_SK.UTF-8.log                 |    21
|   22.7 |    1.2
 HEAD          | install-check-sk_SK.WIN-1250.log              |   137
|   37.4 |    1.2
 REL9_6_STABLE | install-check-sk_SK.WIN-1250.log              |    33
|   30.4 |    0.7
 REL9_5_STABLE | install-check-sk_SK.WIN-1250.log              |    29
|   31.0 |    1.7
 REL9_4_STABLE | install-check-sk_SK.WIN-1250.log              |    25
|   26.2 |    0.4
 REL9_3_STABLE | install-check-sk_SK.WIN-1250.log              |    24
|   25.0 |    1.2
 REL9_2_STABLE | install-check-sk_SK.WIN-1250.log              |    21
|   22.8 |    1.1
 HEAD          | isolation-check.log                           |  2250
|  175.6 |  457.5
 REL9_6_STABLE | isolation-check.log                           |   606
|  201.0 |  527.7
 REL9_5_STABLE | isolation-check.log                           |   622
|  172.3 |  846.7
 REL9_4_STABLE | isolation-check.log                           |   508
|  146.0 |  602.3
 REL9_3_STABLE | isolation-check.log                           |   445
|  114.7 |  504.4
 REL9_2_STABLE | isolation-check.log                           |   403
|   57.2 |  300.6
 HEAD          | make-contrib.log                              |  2176
|   43.7 |   67.6
 REL9_6_STABLE | make-contrib.log                              |   572
|   46.2 |   50.3
 REL9_5_STABLE | make-contrib.log                              |   579
|   44.9 |   54.2
 REL9_4_STABLE | make-contrib.log                              |   491
|   50.0 |   52.3
 REL9_3_STABLE | make-contrib.log                              |   428
|   48.6 |   51.4
 REL9_2_STABLE | make-contrib.log                              |   388
|   42.2 |   42.8
 HEAD          | make-dist.log                                 |    36
|   56.7 |    0.7
 HEAD          | make-doc.log                                  |    36
|  150.8 |    1.5
 REL9_2_STABLE | make-install.log                              |   403
|   22.0 |   29.8
 HEAD          | make.log                                      |  2328
|  363.1 |  581.0
 REL9_6_STABLE | make.log                                      |   610
|  369.6 |  400.9
 REL9_5_STABLE | make.log                                      |   627
|  392.8 |  556.8
 REL9_4_STABLE | make.log                                      |   512
|  335.3 |  329.4
 REL9_3_STABLE | make.log                                      |   449
|  320.0 |  311.0
 REL9_2_STABLE | make.log                                      |   406
|  299.8 |  276.3
 HEAD          | pl-install-check-en_US.8859-15.log            |    37
|   32.9 |    3.3
 HEAD          | pl-install-check-en_US.ISO8859-1.log          |    33
|   34.3 |    5.1
 HEAD          | pl-install-check-POSIX.log                    |    33
|   33.1 |    8.3
 REL9_2_STABLE | SCM-checkout.log                              |   406
|   33.2 |   87.8
 HEAD          | sepgsql-test.log                              |    63
|   20.7 |    6.8
 HEAD          | test-decoding-check.log                       |  1905
|  144.9 |  770.6
 REL9_6_STABLE | test-decoding-check.log                       |   493
|  226.3 | 1244.0
 REL9_5_STABLE | test-decoding-check.log                       |   495
|  147.1 | 1080.2
 REL9_4_STABLE | test-decoding-check.log                       |   419
|  168.1 | 1117.6
 HEAD          | testmodules-install-check-C.log               |  2233
|   22.6 |   42.3
 REL9_6_STABLE | testmodules-install-check-C.log               |   602
|   25.6 |   35.8
 HEAD          | testmodules-install-check-en_US.8859-15.log   |    37
|   57.8 |    5.2
 HEAD          | testmodules-install-check-en_US.ISO8859-1.log |    33
|   58.9 |   10.6
 HEAD          | testmodules-install-check-en_US.log           |   171
|   20.2 |   28.7
 REL9_6_STABLE | testmodules-install-check-en_US.utf8.log      |    30
|   30.4 |   46.9
 HEAD          | testmodules-install-check-en_US.UTF-8.log     |   188
|   22.8 |   19.4
 REL9_6_STABLE | testmodules-install-check-en_US.UTF-8.log     |    36
|   23.6 |   19.4
 REL9_5_STABLE | testmodules-install-check-en_US.UTF-8.log     |    34
|   20.4 |   16.4
 HEAD          | testmodules-install-check-POSIX.log           |    33
|   72.1 |   14.7
 HEAD          | typedefs.log                                  |    96
|   74.8 |   99.1
 REL9_6_STABLE | typedefs.log                                  |    28
|  106.3 |  119.7
 REL9_5_STABLE | typedefs.log                                  |    28
|  100.0 |  111.7
 REL9_4_STABLE | typedefs.log                                  |    24
|   88.1 |  100.7
 REL9_3_STABLE | typedefs.log                                  |    23
|   85.6 |   95.1
(180 rows)

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  45. Re: Need a builtin way to run all tests faster manner

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-03-13T23:26:20Z

    
On 03/13/2017 07:21 PM, Andrew Dunstan wrote:
>
> On 03/13/2017 12:35 AM, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
>>>> This looks generally sane to me, although I'm not very happy about folding
>>>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
>>>> seems weird and unlike the way it's done for the regular regression test
>>>> case.
>>> Yea, not super happy about that either - alternatively we could fold it
>>> into pg_regress.
>> Yeah, teaching pg_regress to auto-create the --temp-instance directory
>> seems perfectly sane from here.
>
> w.r.t. $subject, I thought it might be useful to get some recent stats
> from the buildfarm. Results are below. The bin checks dwarf everything
> else. Upgrade checks and isolation check are other items of significant
> cost. Upgrade checks could be significantly shortened if we could avoid
> rerunning the regression tests.


Sorry for line wrapping. Better formatted here:
<https://gist.github.com/adunstan/fbbdfe646427710233ecb67569665de9>

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  46. Re: Need a builtin way to run all tests faster manner

    Andres Freund <andres@anarazel.de> — 2017-03-14T20:31:08Z

    On 2017-03-13 00:35:06 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
> >> This looks generally sane to me, although I'm not very happy about folding
> >> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
> >> seems weird and unlike the way it's done for the regular regression test
> >> case.
> 
> > Yea, not super happy about that either - alternatively we could fold it
> > into pg_regress.
> 
> Yeah, teaching pg_regress to auto-create the --temp-instance directory
> seems perfectly sane from here.

I was thinking about outputdir, not temp-instance.  The latter is
already created:

		/* make the temp instance top directory */
		make_directory(temp_instance);

Attached is an updated patch that creates outputdir if necessary.  This
is possibly going to trigger a time-to-check-time-to-use coverity
warning, but the rest of pg_regress does if(!exists) mkdir() type logic,
so I did the same.

Besides the pg_regress change, the only thing I've changed is to remove
the in-line "$(MKDIR_P) output_iso && \".

- Andres

  47. Re: Need a builtin way to run all tests faster manner

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-03-15T17:26:35Z

    make check-world -O -j6 PROVE_FLAGS=-j6

2 min 34 seconds

Nice!

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  48. Re: Need a builtin way to run all tests faster manner

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-15T17:33:24Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> make check-world -O -j6 PROVE_FLAGS=-j6
> 2 min 34 seconds
> Nice!

One thing I've noticed with parallel check-world is that it's possible
for a sub-job to fail and for the resulting complaint from make to scroll
offscreen before everything stops, leaving all the visible output being
from another sub-job that completed just fine.  If you don't look VERY
closely, or check $?, you can easily be misled into thinking the tests
all passed.

I think it'd be a good idea to fix check-world so that there's a positive
printout at the end of a successful run, along the lines of the

        +@echo "All of PostgreSQL successfully made. Ready to install."

we have for the "all" target.

			regards, tom lane