Re: Regression tests vs existing users in an installation

On Mon, Jul 18, 2016 at 11:13 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I don't particularly like your suggestion of spooky action at a
>> distance between force_parallel_mode and regression_test_mode.  That
>> just seems kooky.
>
> It's certainly a judgment call as to which way is cleaner, but I don't
> understand your objection.  There are plenty of ways in which multiple
> GUCs determine a given behavior already.  Also, breaking this behavior
> into two variables would let us document the user-useful behavior (do
> this to test parallel safety of functions) in a different place from the
> developer-useful behavior (do this to make EXPLAIN lie to you, which
> surely has no possible use except for regression testing).

There are certainly cases where the behavior that you get depends on
multiple GUCs.  For example, the vacuum cost limit stuff is like that,
and so are the cost factors which control the query planner.  But in
each of those cases, each GUC has a function that is fully orthogonal
to each other GUC.  That doesn't seem to be the case here.  You're
saying that force_parallel_mode will decide whether we get behavior A,
and regression_test_mode will decide whether we get behavior B, but if
you ask for both A and B you will also get an additional behavior C
which would not have been selected by either GUC taken alone.  Because
the different settings are now non-orthogonal, there's now no way to
get A and B without C, or A and C without B.

Moreover, I don't want to end up in a situation where
regression_test_mode=on enables a score of minor behavior changes that
can't be separated out and tested individually.  It's true that
checking the names of regression roles is such a very minor thing that
a generic name like regression_test_mode might be OK, with the idea
that anything else similarly minor that comes along later can be
folded into that as well.  But I fear that it will become a crutch: my
code makes the regression test fail.  Instead of writing better tests,
add another thing that's conditional on regression_test_mode!
Eventually, we'll have a bug that the regression tests fail to catch
because regression_test_mode papers over the problem.  We're some
distance from such a situation now, of course, but I bet the
temptation to be undisciplined about hooking more behavior into that
GUC will be almost irresistible for new developers, and in some cases
experienced ones, too.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company