Re: Race to build pg_isolation_regress in "make -j check-world"

On Mon, Nov 06, 2017 at 12:07:52AM -0800, Noah Misch wrote:
> I've been enjoying the speed of parallel check-world, but I get spurious
> failures from makefile race conditions.  Commit c66b438 fixed the simple ones.
> More tricky is this problem of multiple "make" processes entering
> src/test/regress concurrently, which causes failures like these:
> 
>   gcc: error: pg_regress.o: No such file or directory
>   make[4]: *** [pg_isolation_regress] Error 1
> 
>   /bin/sh: ../../../src/test/isolation/pg_isolation_regress: Permission denied
>   make -C test_extensions check
>   make[2]: *** [check] Error 126
>   make[2]: Leaving directory `/home/nm/src/pg/backbranch/10/src/test/isolation'
> 
>   /bin/sh: ../../../../src/test/isolation/pg_isolation_regress: Text file busy
>   make[3]: *** [isolationcheck] Error 126
>   make[3]: Leaving directory `/home/nm/src/pg/backbranch/10/src/test/modules/snapshot_too_old'
> 
> This is reproducible since commit 2038bf4 or earlier; "make -j check-world"
> had worse problems before that era.  A workaround is to issue "make -j; make
> -j -C src/test/isolation" before the check-world.

Commit de0aca6 fixed that problem, but I now see similar trouble from multiple
"make" processes running "make -C contrib/test_decoding install" concurrently.
This is a risk for any directory named in an EXTRA_INSTALL variable of more
than one makefile.  Under the right circumstances, this would affect
contrib/hstore and others in addition to contrib/test_decoding.  That brings
me back to the locking idea:

> The problem of multiple "make" processes in a directory (especially src/port)
> shows up elsewhere.  In a cleaned tree, "make -j -C src/bin" or "make -j
> installcheck-world" will do it.  For more-prominent use cases, src/Makefile
> prevents this with ".NOTPARALLEL:" and building first the directories that are
> frequent submake targets.  Perhaps we could fix the general problem with
> directory locking; targets that call "$(MAKE) -C FOO" would first sleep until
> FOO's lock is available.  That could be tricky to make robust.

If one is willing to assume that a lock-holding process never crashes, locking
in a shell script is simple: mkdir to lock, rmdir to unlock.  I don't want to
assume that.  The bakery algorithm provides convenient opportunities for
checking whether the last locker crashed; I have attached a shell script
demonstrating this approach.  Better ideas?  Otherwise, I'll look into
integrating this design into the makefiles.

Thanks,
nm