Re: scalability bottlenecks with (many) partitions (and more)

Hi,

On 2025-03-04 16:30:34 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2025-03-04 19:58:38 +0100, Tomas Vondra wrote:
> >> I noticed sifaka started failing right after I pushed this:
> 
> > It's worth noting that
> > a) sifaka doesn't build with ldap support
> > b) the failure is in checkprep, not when running the tests
> > c) the buildfarm unfortunately doesn't archive install.log, so it's hard to
> >    know what actually went wrong
> 
> Yeah, I've been poking at that.  It's not at all clear why the
> animal is trying to run src/test/modules/ldap_password_func
> now when it didn't before.

It did do so before as well, afaict:
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=sifaka&dt=2025-03-04%2015%3A01%3A42&stg=module-ldap_password_func-check

It seems to me that the difference is that now checkprep is run, whereas
previously it wasn't.

Before:
/Library/Developer/CommandLineTools/usr/bin/make -C adt jsonpath_gram.h
make[3]: `jsonpath_gram.h' is up to date.
echo "# +++ tap check in src/test/modules/ldap_password_func +++" && rm -rf '/Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/ldap_password_func'/tmp_check && /bin/sh ../../../../config/install-sh -c -d '/Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/ldap_password_func'/tmp_check && cd . && TESTLOGDIR='/Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/ldap_password_func/tmp_check/log' TESTDATADIR='/Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/ldap_password_func/tmp_check' PATH="/Users/buildfarm/bf-data/HEAD/pgsql.build/tmp_install/Users/buildfarm/bf-data/HEAD/inst/bin:/Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/ldap_password_func:$PATH" DYLD_LIBRARY_PATH="/Users/buildfarm/bf-data/HEAD/pgsql.build/tmp_install/Users/buildfarm/bf-data/HEAD/inst/lib:$DYLD_LIBRARY_PATH" INITDB_TEMPLATE='/Users/buildfarm/bf-data/HEAD/pgsql.build'/tmp_install/initdb-template  PGPORT='65678' top_builddir='/Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/ldap_password_func/../../../..' PG_REGRESS='/Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/ldap_password_func/../../../../src/test/regress/pg_regress' share_contrib_dir='/Users/buildfarm/bf-data/HEAD/pgsql.build/tmp_install/Users/buildfarm/bf-data/HEAD/inst/share/postgresql/contrib' /usr/bin/prove -I ../../../../src/test/perl/ -I . --timer t/*.pl
# +++ tap check in src/test/modules/ldap_password_func +++
[10:08:59] t/001_mutated_bindpasswd.pl .. skipped: LDAP not supported by this build
[10:08:59]

Now:
/Library/Developer/CommandLineTools/usr/bin/make -C adt jsonpath_gram.h
make[3]: `jsonpath_gram.h' is up to date.
rm -rf '/Users/buildfarm/bf-data/HEAD/pgsql.build'/tmp_install
/bin/sh ../../../../config/install-sh -c -d '/Users/buildfarm/bf-data/HEAD/pgsql.build'/tmp_install/log
/Library/Developer/CommandLineTools/usr/bin/make -C '../../../..' DESTDIR='/Users/buildfarm/bf-data/HEAD/pgsql.build'/tmp_install install >'/Users/buildfarm/bf-data/HEAD/pgsql.build'/tmp_install/log/install.log 2>&1
/Library/Developer/CommandLineTools/usr/bin/make -j1  checkprep >>'/Users/buildfarm/bf-data/HEAD/pgsql.build'/tmp_install/log/install.log 2>&1
make: *** [temp-install] Error 2
log files for step module-ldap_password_funcCheck:


Note during a normal build ldap_password_func shouldn't be entered:
# Test runs an LDAP server, so only run if ldap is in PG_TEST_EXTRA
ifeq ($(with_ldap),yes)
ifneq (,$(filter ldap,$(PG_TEST_EXTRA)))
SUBDIRS += ldap_password_func
else
ALWAYS_SUBDIRS += ldap_password_func
endif
else
ALWAYS_SUBDIRS += ldap_password_func
endif


Which leads me to suspect that the difference might be related to
NO_TEMP_INSTALL not being set while it previously was. Which then triggers the
module being built, whereas it previously wasn't.

Of course relying on NO_TEMP_INSTALL preventing this from being built isn't
exactly reliable...


Greetings,

Andres Freund