Horton

Thread

Commits

  1. Fix YA parallel-make hazard, this one in "make check" in plpython.

  2. Put back parallel-safety guards in plpython and src/test/regress/.

  3. Fix partial-build problems introduced by having more generated headers.

  1. submake-errcodes

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2018-04-10T09:52:43Z

    Hi,

I used to run

"
cd src/backend
make submake-errcodes
"

in the RPM spec file, but looks like it was removed recently. Is that replaced
with something else, or removed completely?

Regards,
-- 
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

  2. Re: submake-errcodes

    Christoph Berg <myon@debian.org> — 2018-04-10T10:02:32Z

    Re: Devrim Gündüz 2018-04-10 <1523353963.8169.26.camel@gunduz.org>
> I used to run
> 
> "
> cd src/backend
> make submake-errcodes
> "
> 
> in the RPM spec file, but looks like it was removed recently. Is that replaced
> with something else, or removed completely?

It is gone:
https://git.postgresql.org/pg/commitdiff/3b8f6e75f3c8c6d192621f21624cc8cee04ec3cb

... but that doesn't seem to work transparently either, in an out-of-tree build:

   debian/rules override_dh_auto_build-arch
make[1]: Verzeichnis „/home/cbe/projects/postgresql/pg/master“ wird betreten
/usr/bin/make -C build-py3/src/pl/plpython
make[2]: Verzeichnis „/home/cbe/projects/postgresql/pg/master/build-py3/src/pl/plpython“ wird betreten
/usr/bin/msgfmt -c -o po/cs.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/cs.po
/usr/bin/msgfmt -c -o po/de.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/de.po
/usr/bin/msgfmt -c -o po/es.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/es.po
/usr/bin/msgfmt -c -o po/fr.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/fr.po
/usr/bin/msgfmt -c -o po/it.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/it.po
/usr/bin/msgfmt -c -o po/ja.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/ja.po
/usr/bin/msgfmt -c -o po/ko.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/ko.po
/usr/bin/msgfmt -c -o po/pl.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/pl.po
/usr/bin/msgfmt -c -o po/pt_BR.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/pt_BR.po
/usr/bin/msgfmt -c -o po/ru.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/ru.po
/usr/bin/msgfmt -c -o po/sv.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/sv.po
/usr/bin/msgfmt -c -o po/zh_CN.mo /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/zh_CN.po
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/home/cbe/projects/postgresql/pg/master=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -fPIC -I. -I/home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython -I/usr/include/python3.6m -I../../../src/include -I/home/cbe/projects/postgresql/pg/master/build-py3/../src/include  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE   -c -o plpy_cursorobject.o /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/plpy_cursorobject.c
In file included from /home/cbe/projects/postgresql/pg/master/build-py3/../src/include/postgres.h:47:0,
                 from /home/cbe/projects/postgresql/pg/master/build-py3/../src/pl/plpython/plpy_cursorobject.c:7:
/home/cbe/projects/postgresql/pg/master/build-py3/../src/include/utils/elog.h:71:10: fatal error: utils/errcodes.h: Datei oder Verzeichnis nicht gefunden
 #include "utils/errcodes.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [<eingebaut>: plpy_cursorobject.o] Fehler 1
make[2]: Verzeichnis „/home/cbe/projects/postgresql/pg/master/build-py3/src/pl/plpython“ wird verlassen

Christoph

  3. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-10T14:01:16Z

    Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= <devrim@gunduz.org> writes:
> I used to run
> "
> cd src/backend
> make submake-errcodes
> "
> in the RPM spec file, but looks like it was removed recently. Is that replaced
> with something else, or removed completely?

You could replace it with submake-generated-headers, since that's more
general, but in principle you shouldn't need anything because that
target is invoked automatically as of yesterday.  What's the larger
context here --- why do you need any of this?

			regards, tom lane

  4. Re: submake-errcodes

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2018-04-10T17:41:31Z

    Hi,

On Tue, 2018-04-10 at 10:01 -0400, Tom Lane wrote:
> You could replace it with submake-generated-headers, since that's more
> general, but in principle you shouldn't need anything because that
> target is invoked automatically as of yesterday.  What's the larger
> context here --- why do you need any of this?

Good question -- IIRC we used it to build PL/Python. Just confirmed that
removing from v10 spec file does not break anything.

However, as Christoph wrote, builds against git master fail:

==========================================
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fPIC -I. -I. -I/usr/include/python3.6m -I../../../src/include  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o plpy_resultobject.o plpy_resultobject.c
In file included from ../../../src/include/postgres.h:47:0,
                 from plpy_cursorobject.c:7:
../../../src/include/utils/elog.h:71:10: fatal error: utils/errcodes.h: No such file or directory
 #include "utils/errcodes.h"
          ^~~~~~~~~~~~~~~~~~
In file included from ../../../src/include/postgres.h:47:0,
                 from plpy_procedure.c:7:
../../../src/include/utils/elog.h:71:10: fatal error: utils/errcodes.h: No such file or directory
 #include "utils/errcodes.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
make[1]: *** [<builtin>: plpy_procedure.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [<builtin>: plpy_cursorobject.o] Error 1
==========================================

Regards,

-- 
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

  5. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-10T17:56:37Z

    Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= <devrim@gunduz.org> writes:
> On Tue, 2018-04-10 at 10:01 -0400, Tom Lane wrote:
>> You could replace it with submake-generated-headers, since that's more
>> general, but in principle you shouldn't need anything because that
>> target is invoked automatically as of yesterday.  What's the larger
>> context here --- why do you need any of this?

> However, as Christoph wrote, builds against git master fail:

Hm ... you're cd'ing into src/pl/plpython and issuing "make all"?
That works for me.

... or, wait ... with -j it doesn't.  That's strange, will look.

			regards, tom lane

  6. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-10T19:04:22Z

    I wrote:
> Hm ... you're cd'ing into src/pl/plpython and issuing "make all"?
> That works for me.
> ... or, wait ... with -j it doesn't.  That's strange, will look.

So after a bit of digging, it seems that the locution

all: submake-generated-headers

doesn't result in ensuring that submake-generated-headers is complete
before we go to build the other targets required by "all"; it only
says that submake-generated-headers must be complete before we execute
the (empty) list of commands attached to the "all" target.

I'd tested high-j runs pretty carefully at top level, but it turns
out that that works because in both the toplevel GNUmakefile and
src/Makefile, all the interesting work happens in recursive sub-makes,
and we force the ordering of those properly with the dependencies on
the recursive make rules:

$(1)-$(2)-recurse: $(if $(filter all install, $(3)), submake-generated-headers) $(if $(filter check, $(3)), temp-install)
	$$(MAKE) -C $(2) $(3)

If you go to, eg, src/pl/plpython and issue "make -j", there's nothing
to prevent the builds of object files from happening before the header
build finishes.  There *was* something there before:

$(OBJS): | submake-generated-headers

but I took it out thinking it was no longer needed.

The short-term solution seems to be to put that back, but that's sort
of annoying because it means this isn't a bulletproof solution.  It
will only work for builds started in one of the directories that we
take the trouble to put this defense into, and I can't see doing that
everywhere.  Still, such things didn't work reliably before either
except in these few directories, so maybe it won't matter.

			regards, tom lane

  7. Re: submake-errcodes

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2018-04-10T20:09:33Z

    Tom Lane wrote:

> $(OBJS): | submake-generated-headers
> 
> but I took it out thinking it was no longer needed.
> 
> The short-term solution seems to be to put that back, but that's sort
> of annoying because it means this isn't a bulletproof solution.  It
> will only work for builds started in one of the directories that we
> take the trouble to put this defense into, and I can't see doing that
> everywhere.  Still, such things didn't work reliably before either
> except in these few directories, so maybe it won't matter.

Isn't it possible to put it in some common.mk file rather than each
individual Makefile?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  8. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-10T20:13:23Z

    Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> Tom Lane wrote:
>> $(OBJS): | submake-generated-headers
>> but I took it out thinking it was no longer needed.
>> 
>> The short-term solution seems to be to put that back, but that's sort
>> of annoying because it means this isn't a bulletproof solution.

> Isn't it possible to put it in some common.mk file rather than each
> individual Makefile?

I thought about putting the above into Makefile.global, but it'd only
work in cases where $(OBJS) gets set before including Makefile.global,
which turns out to be nearly noplace.  So we'd still end up touching
an awful lot of makefiles to make it work, for relatively little
practical benefit.

			regards, tom lane

  9. Re: submake-errcodes

    Christoph Berg <myon@debian.org> — 2018-04-11T08:39:54Z

    Re: Tom Lane 2018-04-10 <24426.1523387062@sss.pgh.pa.us>
> The short-term solution seems to be to put that back, but that's sort
> of annoying because it means this isn't a bulletproof solution.  It
> will only work for builds started in one of the directories that we
> take the trouble to put this defense into, and I can't see doing that
> everywhere.  Still, such things didn't work reliably before either
> except in these few directories, so maybe it won't matter.

Is that supposed to work now, without explicitly invoking sub-generated-headers?
It still doesn't work on current HEAD:

/home/cbe/projects/postgresql/pg/master/build-py3/../src/include/utils/elog.h:71:10: fatal error: utils/errcodes.h: Datei oder Verzeichnis nicht gefunden
 #include "utils/errcodes.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [<eingebaut>: plpy_cursorobject.o] Fehler 1
make[2]: Verzeichnis „/home/cbe/projects/postgresql/pg/master/build-py3/src/pl/plpython“ wird verlassen

The other directories we'd need it are contrib/{hstore,jsonb,ltree}_plpython.

Christoph

  10. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-11T13:38:34Z

    Christoph Berg <myon@debian.org> writes:
> It still doesn't work on current HEAD:

*What* still doesn't work on current HEAD?  I don't know what commands
you are running to get this.

			regards, tom lane

  11. Re: submake-errcodes

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2018-04-11T14:05:03Z

    Hi,

On Wed, 2018-04-11 at 09:38 -0400, Tom Lane wrote:
> *What* still doesn't work on current HEAD?  I don't know what commands
> you are running to get this.

I think my build and Christoph's builds fail because of the same reason again
(the same as yesterday):

==================================
+ cd src/pl/plpython
+ make all
<snip>

gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fPIC -I. -I. -I/usr/include/python3.6m -I../../../src/include  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o plpy_cursorobject.o plpy_cursorobject.c
In file included from ../../../src/include/postgres.h:47:0,
                 from plpy_cursorobject.c:7:
../../../src/include/utils/elog.h:71:10: fatal error: utils/errcodes.h: No such file or directory
 #include "utils/errcodes.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [<builtin>: plpy_cursorobject.o] Error 1
===========

Regards,
-- 
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

  12. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-11T14:16:24Z

    Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= <devrim@gunduz.org> writes:
> On Wed, 2018-04-11 at 09:38 -0400, Tom Lane wrote:
>> *What* still doesn't work on current HEAD?  I don't know what commands
>> you are running to get this.

> I think my build and Christoph's builds fail because of the same reason again
> (the same as yesterday):

> + cd src/pl/plpython
> + make all

That works for me:

$ git clean -dfx
$ ./configure ... --with-python and other stuff ...
$ cd src/pl/plpython
$ make all
make -C ../../../src/backend generated-headers
make[1]: Entering directory `/home/postgres/pgsql/src/backend'
make -C catalog distprep generated-header-symlinks
make[2]: Entering directory `/home/postgres/pgsql/src/backend/catalog'
...
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fPIC -I. -I. -I/usr/include/python2.6 -I../../../src/include  -D_GNU_SOURCE   -c -o plpy_util.o plpy_util.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fPIC -shared -o plpython2.so plpy_cursorobject.o plpy_elog.o plpy_exec.o plpy_main.o plpy_planobject.o plpy_plpymodule.o plpy_procedure.o plpy_resultobject.o plpy_spi.o plpy_subxactobject.o plpy_typeio.o plpy_util.o  -L../../../src/port -L../../../src/common    -Wl,--as-needed -Wl,-rpath,'/usr/lib64',--enable-new-dtags  -L/usr/lib64 -lpython2.6 -lpthread -ldl  -lutil -lm  

You sure you're on 31f1f0bb4fd642643994d35c35ecb5b929711a99 or later?
Which gmake version is this?

			regards, tom lane

  13. Re: submake-errcodes

    Michael Paquier <michael@paquier.xyz> — 2018-04-11T23:58:43Z

    On Wed, Apr 11, 2018 at 10:16:24AM -0400, Tom Lane wrote:
> That works for me:
> 
> $ git clean -dfx
> $ ./configure ... --with-python and other stuff ...
> $ cd src/pl/plpython
> $ make all
> make -C ../../../src/backend generated-headers
> make[1]: Entering directory `/home/postgres/pgsql/src/backend'
> make -C catalog distprep generated-header-symlinks
> make[2]: Entering directory `/home/postgres/pgsql/src/backend/catalog'
> ...
> gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fPIC -I. -I. -I/usr/include/python2.6 -I../../../src/include  -D_GNU_SOURCE   -c -o plpy_util.o plpy_util.c
> gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fPIC -shared -o plpython2.so plpy_cursorobject.o plpy_elog.o plpy_exec.o plpy_main.o plpy_planobject.o plpy_plpymodule.o plpy_procedure.o plpy_resultobject.o plpy_spi.o plpy_subxactobject.o plpy_typeio.o plpy_util.o  -L../../../src/port -L../../../src/common    -Wl,--as-needed -Wl,-rpath,'/usr/lib64',--enable-new-dtags  -L/usr/lib64 -lpython2.6 -lpthread -ldl  -lutil -lm  
> 
> You sure you're on 31f1f0bb4fd642643994d35c35ecb5b929711a99 or later?
> Which gmake version is this?

For what it's worth, all those combinations work for me when on
d1e90792:
./configure blah
cd src/pl/plpython/
make -j 4 [ all | install ]

But this does not work:
./configure blah
cd src/pl/plpython/
make -j 4 check

make[2]: *** [Makefile:89: keywords_srv.o] Error 1
make[2]: *** Deleting file 'keywords_srv.o'
make[2]: Leaving directory '/home/ioltas/git/postgres/src/common'
make[1]: *** [../../../src/Makefile.global:568: submake-libpgport] Error 2

Thanks,
--
Michael

  14. Re: submake-errcodes

    Christoph Berg <myon@debian.org> — 2018-04-12T07:01:21Z

    Re: Michael Paquier 2018-04-12 <20180411235843.GG32449@paquier.xyz>
> > You sure you're on 31f1f0bb4fd642643994d35c35ecb5b929711a99 or later?
> > Which gmake version is this?

GNU Make 4.2.1

> For what it's worth, all those combinations work for me when on
> d1e90792:
> ./configure blah
> cd src/pl/plpython/
> make -j 4 [ all | install ]
> 
> But this does not work:
> ./configure blah
> cd src/pl/plpython/
> make -j 4 check
> 
> make[2]: *** [Makefile:89: keywords_srv.o] Error 1
> make[2]: *** Deleting file 'keywords_srv.o'
> make[2]: Leaving directory '/home/ioltas/git/postgres/src/common'
> make[1]: *** [../../../src/Makefile.global:568: submake-libpgport] Error 2

Most of these work for me as well. The actual incantation via
debian/rules fails, but I couldn't really narrow down what the
difference is. Manually invoking the debian/rules targets:

$ debian/rules stamp/configure-build-py3
mkdir -p stamp build-py3
cd build-py3 && ../configure \
           --with-python \
           PYTHON=/usr/bin/python3 \
           --mandir=/usr/share/postgresql/11/man --docdir=/usr/share/doc/postgresql-doc-11 --sysconfdir=/etc/postgresql-common --datarootdir=/usr/share/ --datadir=/usr/share/postgresql/11 --bindir=/usr/lib/postgresql/11/bin --libdir=/usr/lib/x86_64-linux-gnu/ --libexecdir=/usr/lib/postgresql/ --includedir=/usr/include/postgresql/ --with-extra-version=" (Debian 11~~devel-1)" --enable-nls --enable-integer-datetimes --enable-thread-safety --enable-tap-tests --enable-debug  --disable-rpath --with-llvm --with-uuid=e2fs --with-gnu-ld --with-pgport=5432 --with-system-tzdata=/usr/share/zoneinfo --with-systemd  CFLAGS='-g -O2 -fdebug-prefix-map=/srv/projects/postgresql/pg/master=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer' LDFLAGS='-Wl,-z,relro -Wl,-z,now'
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
...
checking for PYTHON... /usr/bin/python3
configure: using python 3.6.5 (default, Apr  1 2018, 05:46:30)
checking for Python distutils module... yes
checking Python configuration directory... /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu
checking Python include directories... -I/usr/include/python3.6m
checking how to link an embedded Python application... -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -lpython3.6m -lpthread -ldl  -lutil -lm
...
preparing build tree... done
configure: creating ./config.status
config.status: creating GNUmakefile
config.status: creating src/Makefile.global
config.status: creating src/include/pg_config.h
config.status: creating src/include/pg_config_ext.h
config.status: creating src/interfaces/ecpg/include/ecpg_config.h
config.status: linking ../src/backend/port/tas/dummy.s to src/backend/port/tas.s
config.status: linking ../src/backend/port/dynloader/linux.c to src/backend/port/dynloader.c
config.status: linking ../src/backend/port/posix_sema.c to src/backend/port/pg_sema.c
config.status: linking ../src/backend/port/sysv_shmem.c to src/backend/port/pg_shmem.c
config.status: linking ../src/backend/port/dynloader/linux.h to src/include/dynloader.h
config.status: linking ../src/include/port/linux.h to src/include/pg_config_os.h
config.status: linking ../src/makefiles/Makefile.linux to src/Makefile.port
touch "stamp/configure-build-py3"

$ debian/rules stamp/build-py3
/usr/bin/make -C build-py3/src/pl/plpython
make[1]: Verzeichnis „/srv/projects/postgresql/pg/master/build-py3/src/pl/plpython“ wird betreten
/usr/bin/msgfmt -c -o po/cs.mo /srv/projects/postgresql/pg/master/build-py3/../src/pl/plpython/po/cs.po
... more msgfmt
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/srv/projects/postgresql/pg/master=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -fPIC -I. -I/srv/projects/postgresql/pg/master/build-py3/../src/pl/plpython -I/usr/include/python3.6m -I../../../src/include -I/srv/projects/postgresql/pg/master/build-py3/../src/include  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE   -c -o plpy_cursorobject.o /srv/projects/postgresql/pg/master/build-py3/../src/pl/plpython/plpy_cursorobject.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/srv/projects/postgresql/pg/master=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -fPIC -I. -I/srv/projects/postgresql/pg/master/build-py3/../src/pl/plpython -I/usr/include/python3.6m -I../../../src/include -I/srv/projects/postgresql/pg/master/build-py3/../src/include  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE   -c -o plpy_elog.o /srv/projects/postgresql/pg/master/build-py3/../src/pl/plpython/plpy_elog.c
In file included from /srv/projects/postgresql/pg/master/build-py3/../src/include/postgres.h:47:0,
                 from /srv/projects/postgresql/pg/master/build-py3/../src/pl/plpython/plpy_cursorobject.c:7:
/srv/projects/postgresql/pg/master/build-py3/../src/include/utils/elog.h:71:10: fatal error: utils/errcodes.h: Datei oder Verzeichnis nicht gefunden
 #include "utils/errcodes.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /srv/projects/postgresql/pg/master/build-py3/../src/include/postgres.h:47:0,
                 from /srv/projects/postgresql/pg/master/build-py3/../src/pl/plpython/plpy_elog.c:7:
/srv/projects/postgresql/pg/master/build-py3/../src/include/utils/elog.h:71:10: fatal error: utils/errcodes.h: Datei oder Verzeichnis nicht gefunden
 #include "utils/errcodes.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [<eingebaut>: plpy_cursorobject.o] Fehler 1
make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet....
make[1]: *** [<eingebaut>: plpy_elog.o] Fehler 1
make[1]: Verzeichnis „/srv/projects/postgresql/pg/master/build-py3/src/pl/plpython“ wird verlassen
make: *** [debian/rules:156: stamp/build-py3] Fehler 2


Invoking the commands manually works. debian/rules is
https://salsa.debian.org/postgresql/postgresql/blob/11/debian/rules

(Known bug: ltree_plpython is not yet built for python 3)

Christoph

  15. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-12T13:49:58Z

    Christoph Berg <myon@debian.org> writes:
> Most of these work for me as well. The actual incantation via
> debian/rules fails, but I couldn't really narrow down what the
> difference is. Manually invoking the debian/rules targets:

For a moment I thought that this might be the critical difference:
> preparing build tree... done
ie that you're doing a VPATH build.  But no, that case works as
expected too, for me.

The only conclusion I can come to is that you're using sources that
predate my recent makefile fixes, particularly 3b8f6e75f.  Please
double-check that.

			regards, tom lane

  16. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-12T14:36:08Z

    Michael Paquier <michael@paquier.xyz> writes:
> But this does not work:
> ./configure blah
> cd src/pl/plpython/
> make -j 4 check

Hm.  That shows yet another parallel-safety hazard, which can be resolved
like this:

diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index 653fe64..c17015b 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -131,7 +131,7 @@ installcheck: submake-pg-regress
 
 
 .PHONY: submake-pg-regress
-submake-pg-regress:
+submake-pg-regress: | submake-generated-headers
 	$(MAKE) -C $(top_builddir)/src/test/regress pg_regress$(X)
 
 clean distclean: clean-lib

to ensure that we finish the generated-headers work before launching
that child Make run.

I'm beginning to get dissatisfied with this approach of expecting the
topmost Make run to do the generated-headers work; it's bleeding into
more places than I'd hoped.  I don't see any really good alternative
though.  If we allow the child runs to try to do it, we're going to
have issues with parallel runs clobbering each others' output.  It's
somewhat surprising that that didn't occur more often before; the
only real difference since the bootstrap data restructuring is that
we have more generated headers than we used to.

I guess the good news is that the supported cases will be a whole lot
more bulletproof against high -j counts than they were before.  I never
used to dare going beyond -j8, because builds tended to fall over.

			regards, tom lane

  17. Re: submake-errcodes

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2018-04-12T15:15:32Z

    Hi Tom,

On Wed, 2018-04-11 at 10:16 -0400, Tom Lane wrote:
> You sure you're on 31f1f0bb4fd642643994d35c35ecb5b929711a99 or later?

To make sure, I am using latest git snapshot:

https://download.postgresql.org/pub/snapshot/dev/postgresql-snapshot.tar.bz2

and still same error. Easy to reproduce on F-27 box. 

> Which gmake version is this?

4.2.1

Regards,
-- 
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

  18. Re: submake-errcodes

    Andrew Gierth <andrew@tao11.riddles.org.uk> — 2018-04-12T17:57:55Z

    >>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:

 Tom> I'm beginning to get dissatisfied with this approach of expecting
 Tom> the topmost Make run to do the generated-headers work

Is it worth exploring the idea of changing to a non-recursive style of
makefile?

-- 
Andrew (irc:RhodiumToad)

  19. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-12T18:00:32Z

    Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
>  Tom> I'm beginning to get dissatisfied with this approach of expecting
>  Tom> the topmost Make run to do the generated-headers work

> Is it worth exploring the idea of changing to a non-recursive style of
> makefile?

Not post-feature-freeze, for sure.  Whether it's worth the work as a
long-term project, I dunno.

			regards, tom lane

  20. Re: submake-errcodes

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2018-04-12T18:07:38Z

    Andrew Gierth wrote:
> >>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
> 
>  Tom> I'm beginning to get dissatisfied with this approach of expecting
>  Tom> the topmost Make run to do the generated-headers work
> 
> Is it worth exploring the idea of changing to a non-recursive style of
> makefile?

I looked onto this a while ago.  It seems easily doable for the backend
code proper, modulo potential command-line length issues; but anything
involving shared libraries requires Makefile.shlib to be involved, which
seems painful to resolve.  Once I realized the shlib problem, I stopped
trying, but if you find some way around that it may be worthwhile.

I'm altogether ignorant on how might we use it for the headers problem,
mind.  I only tried to tackle the main executable.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  21. Re: submake-errcodes

    Jacob Champion <pchampion@pivotal.io> — 2018-04-12T18:22:45Z

    On Thu, Apr 12, 2018 at 11:00 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
>> Is it worth exploring the idea of changing to a non-recursive style of
>> makefile?
>
> Not post-feature-freeze, for sure.  Whether it's worth the work as a
> long-term project, I dunno.

I've been taking a look at this in my limited free time, so I might as
well publicly register my interest here. Moving to non-recursive Make
will probably be quite a bit of work/pain, but I also think it's
probably worth it in the end.

(We make heavy use of automated builds, and having a complete
dependency graph should speed those up considerably, in addition to
the maintenance benefits already implied here.)

--Jacob

  22. Re: submake-errcodes

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-12T18:36:00Z

    Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= <devrim@gunduz.org> writes:
> ... still same error. Easy to reproduce on F-27 box. 

I don't have F27 at hand, but I tried F26 and F28, and I can't reproduce
on either one.  I tried various combinations of python2 versus python3,
in-tree build versus VPATH from bare checkout versus VPATH from
distprep'd tree, and they all work for me.

Just to clarify, I'm experimenting with

$ git clean -dfx
$ ./configure ... --with-python [ PYTHON=/usr/bin/python3 ]
$ cd src/pl/plpython/
$ make -j25

and variants of that, and what I get as a result is a make trace
starting with

make -C ../../../src/backend generated-headers

If you're not seeing that, something's very wrong, and I do not
know what.

[ time passes ]

... or then again, maybe I do.  Is it possible that your build
recipe involves invoking our makefiles from an outer "make" run?
If so, maybe you need to explicitly set MAKELEVEL=0 when invoking
our build, to keep it from thinking it is a sub-make.  Not sure
about whether it'd be wise to reset MAKEFLAGS as well.

			regards, tom lane

  23. Re: submake-errcodes

    Andrew Gierth <andrew@tao11.riddles.org.uk> — 2018-04-12T18:37:21Z

    >>>>> "Alvaro" == Alvaro Herrera <alvherre@alvh.no-ip.org> writes:

 Alvaro> I'm altogether ignorant on how might we use it for the headers
 Alvaro> problem, mind. I only tried to tackle the main executable.

Solving the headers problem would seem to require making things
non-recursive at the topmost level rather than trying to do only a
partial conversion.

-- 
Andrew (irc:RhodiumToad)

  24. Re: submake-errcodes

    Andres Freund <andres@anarazel.de> — 2018-04-12T19:33:02Z

    On 2018-04-12 11:22:45 -0700, Jacob Champion wrote:
> On Thu, Apr 12, 2018 at 11:00 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
> >> Is it worth exploring the idea of changing to a non-recursive style of
> >> makefile?
> >
> > Not post-feature-freeze, for sure.  Whether it's worth the work as a
> > long-term project, I dunno.
> 
> I've been taking a look at this in my limited free time, so I might as
> well publicly register my interest here. Moving to non-recursive Make
> will probably be quite a bit of work/pain, but I also think it's
> probably worth it in the end.

Yea, it'd sure be nice.  Whether it's worth the pain or whether the time
is better spent moving to cmake or such, I'm not sure.

Greetings,

Andres Freund

  25. Re: submake-errcodes

    Christoph Berg <myon@debian.org> — 2018-04-12T20:27:18Z

    Re: Tom Lane 2018-04-12 <10354.1523558160@sss.pgh.pa.us>
> ... or then again, maybe I do.  Is it possible that your build
> recipe involves invoking our makefiles from an outer "make" run?
> If so, maybe you need to explicitly set MAKELEVEL=0 when invoking
> our build, to keep it from thinking it is a sub-make.  Not sure
> about whether it'd be wise to reset MAKEFLAGS as well.

I don't know about Devrim's case, but debian/rules is indeed a
makefile. Will look closer again once I'm back from pgconf.de, thanks
for the hint.

Christoph

  26. Re: submake-errcodes

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2018-04-15T20:36:28Z

    Hi Tom,

On Thu, 2018-04-12 at 14:36 -0400, Tom Lane wrote:
> I don't have F27 at hand, but I tried F26 and F28, and I can't reproduce
> on either one. 

I uploaded the SRPM for you:

https://gunduz.org/temp/postgresql11-11.0-20180415_1PGDG.f27.src.rpm

I built this SRPM using daily snapshot at:
https://download.postgresql.org/pub/snapshot/dev/

When you rebuild it, you'll see the error.

Regards,

-- 
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

  27. Re: submake-errcodes

    Christoph Berg <myon@debian.org> — 2018-04-16T07:24:27Z

    Re: To Tom Lane 2018-04-12 <20180412202717.GA32601@msg.df7cb.de>
> > ... or then again, maybe I do.  Is it possible that your build
> > recipe involves invoking our makefiles from an outer "make" run?
> > If so, maybe you need to explicitly set MAKELEVEL=0 when invoking
> > our build, to keep it from thinking it is a sub-make.  Not sure
> > about whether it'd be wise to reset MAKEFLAGS as well.
> 
> I don't know about Devrim's case, but debian/rules is indeed a
> makefile. Will look closer again once I'm back from pgconf.de, thanks
> for the hint.

Fwiw, setting MAKELEVEL=0 worked. Thanks!

https://salsa.debian.org/postgresql/postgresql/blob/11/debian/rules#L147-148

Christoph

  28. Re: submake-errcodes

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2018-04-16T08:49:31Z

    Hi,

On Mon, 2018-04-16 at 09:24 +0200, Christoph Berg wrote:
> Fwiw, setting MAKELEVEL=0 worked. Thanks!

Great, it solved my problem as well! Thanks Tom.

Regards,
-- 
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR