Re: Improving the isolationtester: fewer failures, less delay

Hi,

On 2021-06-14 22:57:08 -0400, Tom Lane wrote:
> This is a followup to the conversation at [1], in which we speculated
> about constraining the isolationtester's behavior by annotating the
> specfiles, in order to eliminate common buildfarm failures such as [2]
> and reduce the need to use long delays to stabilize the test results.
> 
> I've spent a couple days hacking on this idea, and I think it has worked
> out really well.  On my machine, the time needed for "make installcheck"
> in src/test/isolation drops from ~93 seconds to ~26 seconds, as a result
> of removing all the multiple-second delays we used before.

Very cool stuff. All the reliability things aside, isolationtester
frequently is the slowest test in a parallel check world...


> Also, while I'm not fool enough to claim that this will reduce the
> rate of bogus failures to zero, I do think it addresses all the
> repeating failures we've seen lately.

And it should make it easier to fix some others and also to make it
easier to write some tests that were too hard to get to reliable today.


> This is still WIP to some extent, as I've not spent much time looking at
> specfiles other than the ones with big delays; there may be additional
> improvements possible in some places.  Also, I've not worried about
> whether the tests pass in serializable mode, since we have problems there
> already [4].  But this seemed like a good point at which to solicit
> feedback and see what the cfbot thinks of it.

Are there spec output changes / new failures, if you apply the patch,
but do not apply the changes to the spec files?


Will look at the patch itself in a bit.

Greetings,

Andres Freund