Horton

Thread

Commits

  1. configure: Expand -fvisibility checks to more compilers, test for -qvisibility

  2. aix: No need to use mkldexport when we want to export all symbols

  3. aix: when building with gcc, tell gcc we're building a shared library

  4. aix: Fix SHLIB_EXPORTS reference in VPATH builds

  5. Remove SUBSYS.o rule in common.mk, hasn't been used in a long time

  6. Remove rule to generate postgres.o, not needed for 20+ years

  7. Make the AIX case of Makefile.shlib safe for parallel make.

  1. static libpq (and other libraries) overwritten on aix

    Andres Freund <andres@anarazel.de> — 2022-08-17T19:01:54Z

    Hi,

I was hacking in making aix work with the meson patchset last night when I
noticed this delightful bit:

gmake -C src/interfaces/libpq
...

rm -f libpq.a
ar crs libpq.a fe-auth-scram.o fe-connect.o fe-exec.o fe-lobj.o fe-misc.o fe-print.o fe-protocol3.o fe-secure.o fe-trace.o legacy-pqsignal.o libpq-events.o pqexpbuffer.o fe-auth.o
touch libpq.a

( echo '#! libpq.so.5'; gawk '/^[^#]/ {printf "%s\n",$1}' /home/andres/src/postgres/build-ac/../src/interfaces/libpq/exports.txt ) >libpq.exp
gcc -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -O2 -pthread -D_REENTRANT -D_THREAD_SAFE  -o libpq.so.5 libpq.a -Wl,-bE:libpq.exp -L../../../src/port -L../../../src/common -lpgcommon_shlib -lpgport_shlib -Wl,-bbigtoc  -Wl,-blibpath:'/usr/local/pgsql/lib:/usr/lib:/lib'  -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -lm

rm -f libpq.a
ar crs libpq.a libpq.so.5


we first create a static library libpq.a as normal, but then we overwrite it
with the special aix way of packing up shared libraries, by packing them up in
a static library.  That part is correct, it's apparently the easiest way of
getting applications to link to shared libraries on AIX (I think the
-Wl,-bM:SRE is relevant for ensuring it'll be a dynamic link, rather than a
static one).

This likely has been going on for approximately forever.

Two questions:
1) Do we continue building static libraries for libpq etc?
2) Do we care about static libraries not suriving on AIX? There could also be
   a race in the buildrules leading to sometimes static libs sometimes shared
   libs winning, I think.

Greetings,

Andres Freund

  2. Re: static libpq (and other libraries) overwritten on aix

    Robert Haas <robertmhaas@gmail.com> — 2022-08-17T19:28:18Z

    On Wed, Aug 17, 2022 at 3:02 PM Andres Freund <andres@anarazel.de> wrote:
> 2) Do we care about static libraries not suriving on AIX? There could also be
>    a race in the buildrules leading to sometimes static libs sometimes shared
>    libs winning, I think.

Instead of overwriting the same file, can we not use different
filenames for different things?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  3. Re: static libpq (and other libraries) overwritten on aix

    Andres Freund <andres@anarazel.de> — 2022-08-17T20:08:56Z

    Hi,

On 2022-08-17 15:28:18 -0400, Robert Haas wrote:
> On Wed, Aug 17, 2022 at 3:02 PM Andres Freund <andres@anarazel.de> wrote:
> > 2) Do we care about static libraries not suriving on AIX? There could also be
> >    a race in the buildrules leading to sometimes static libs sometimes shared
> >    libs winning, I think.
>
> Instead of overwriting the same file, can we not use different
> filenames for different things?

Not easily, as far as I understand. The way one customarily links to shared
libraries on aix is to have an .a archive containing the shared library. That
way the -lpq picks up libpq.a, which then triggers the shared library to be
referenced.

E.g.
andres@gcc119:[/home/andres/src/postgres/build-ac]$ LIBPATH=$(pwd)/src/interfaces/libpq ldd src/bin/scripts/clusterdb
src/bin/scripts/clusterdb needs:
         /usr/lib/libc.a(shr_64.o)
         /usr/lib/libpthread.a(shr_xpg5_64.o)
         /usr/lib/libreadline.a(libreadline.so.6)
         /home/andres/src/postgres/build-ac/src/interfaces/libpq/libpq.a(libpq.so.5)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)
         /usr/lib/libcurses.a(shr42_64.o)
         /usr/lib/libpthreads.a(shr_xpg5_64.o)

Note the .a(libpq.so.5) bit.


Unfortunately that's exactly how one links to a static library as well.

So we'd have to change the name used as -l$this between linking to a shared
libpq and a static libpq.

Greetings,

Andres Freund

  4. Re: static libpq (and other libraries) overwritten on aix

    Noah Misch <noah@leadboat.com> — 2022-08-18T04:59:29Z

    The AIX sections of Makefile.shlib misuse the terms "static" and "shared".
Imagine s/static library/library ending in .a/ and s/shared library/library
ending in .so/.  That yields an accurate description of the makefile rules.

On Wed, Aug 17, 2022 at 12:01:54PM -0700, Andres Freund wrote:
> Two questions:
> 1) Do we continue building static libraries for libpq etc?

Essentially, we don't build static libpq today, and we should continue not
building it.  (The first-built libpq.a is static, but that file is an
implementation detail of the makefile rules.  The surviving libpq.a is a
normal AIX shared library.)

> 2) Do we care about static libraries not suriving on AIX?

No.

>    There could also be
>    a race in the buildrules leading to sometimes static libs sometimes shared
>    libs winning, I think.

Not since commit e8564ef, to my knowledge.


Along the lines of Robert's comment, it could be a nice code beautification to
use a different suffix for the short-lived .a file.  Perhaps _so_inputs.a.

I found this useful years ago:
https://web.archive.org/web/20151003130212/http://seriousbirder.com/blogs/aix-shared-and-static-libraries-explained/

  5. Re: static libpq (and other libraries) overwritten on aix

    Robert Haas <robertmhaas@gmail.com> — 2022-08-18T14:10:24Z

    On Thu, Aug 18, 2022 at 12:59 AM Noah Misch <noah@leadboat.com> wrote:
> Along the lines of Robert's comment, it could be a nice code beautification to
> use a different suffix for the short-lived .a file.  Perhaps _so_inputs.a.

Yeah, this is the kind of thing I was thinking about.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  6. Re: static libpq (and other libraries) overwritten on aix

    Andrew Dunstan <andrew@dunslane.net> — 2022-08-18T14:31:49Z

    On 2022-08-18 Th 10:10, Robert Haas wrote:
> On Thu, Aug 18, 2022 at 12:59 AM Noah Misch <noah@leadboat.com> wrote:
>> Along the lines of Robert's comment, it could be a nice code beautification to
>> use a different suffix for the short-lived .a file.  Perhaps _so_inputs.a.
> Yeah, this is the kind of thing I was thinking about.


+1 for that and clarifying Makefile.shlib.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  7. Re: static libpq (and other libraries) overwritten on aix

    Andres Freund <andres@anarazel.de> — 2022-08-18T16:03:57Z

    Hi,

On 2022-08-17 21:59:29 -0700, Noah Misch wrote:
> The AIX sections of Makefile.shlib misuse the terms "static" and "shared".
>
> Imagine s/static library/library ending in .a/ and s/shared library/library
> ending in .so/.  That yields an accurate description of the makefile rules.

I realize that aspect.

My point is that we currently, across most of the other platforms, support
building a "proper" static library, and install it too. But on AIX (and I
think mingw), we don't, but without an explicit comment about not doing so. In
fact, the all-static-lib target on those platforms will build a non-static
library, which seems not great.


> >    There could also be
> >    a race in the buildrules leading to sometimes static libs sometimes shared
> >    libs winning, I think.
>
> Not since commit e8564ef, to my knowledge.

I'd missed that the $(stlib): ... bit is not defined due to haslibarule being
defined...


> Along the lines of Robert's comment, it could be a nice code beautification to
> use a different suffix for the short-lived .a file.  Perhaps _so_inputs.a.

Agreed, it'd be an improvement.

Afaict we could just stop building the intermediary static lib. Afaict the
MKLDEXPORT path isn't needed for libraries without an exports.txt because the
linker defaults to exporting "most" symbols, and for symbols with an
exports.txt we don't need it either.

The only path that really needs MKLDEXPORT is postgres. Not really for the
export side either, but for the import side.

Greetings,

Andres Freund

  8. Re: static libpq (and other libraries) overwritten on aix

    Noah Misch <noah@leadboat.com> — 2022-08-19T05:56:43Z

    On Thu, Aug 18, 2022 at 09:03:57AM -0700, Andres Freund wrote:
> My point is that we currently, across most of the other platforms, support
> building a "proper" static library, and install it too. But on AIX (and I
> think mingw), we don't, but without an explicit comment about not doing so. In
> fact, the all-static-lib target on those platforms will build a non-static
> library, which seems not great.

Yep.  If someone had just pushed a correct patch to make AIX match our
GNU/Linux static linking assistance, I wouldn't be arguing to revert that
patch.  At the same time, if someone asks me to choose high-value projects for
20 people, doing more for static linking on AIX won't be on the list.

> On 2022-08-17 21:59:29 -0700, Noah Misch wrote:
> > Along the lines of Robert's comment, it could be a nice code beautification to
> > use a different suffix for the short-lived .a file.  Perhaps _so_inputs.a.
> 
> Agreed, it'd be an improvement.
> 
> Afaict we could just stop building the intermediary static lib. Afaict the
> MKLDEXPORT path isn't needed for libraries without an exports.txt because the
> linker defaults to exporting "most" symbols

If that works, great.

  9. Re: static libpq (and other libraries) overwritten on aix

    Andres Freund <andres@anarazel.de> — 2022-08-20T08:35:22Z

    Hi,

On 2022-08-18 22:56:43 -0700, Noah Misch wrote:
> > On 2022-08-17 21:59:29 -0700, Noah Misch wrote:
> > > Along the lines of Robert's comment, it could be a nice code beautification to
> > > use a different suffix for the short-lived .a file.  Perhaps _so_inputs.a.
> >
> > Agreed, it'd be an improvement.
> >
> > Afaict we could just stop building the intermediary static lib. Afaict the
> > MKLDEXPORT path isn't needed for libraries without an exports.txt because the
> > linker defaults to exporting "most" symbols
>
> If that works, great.

I looked at that. It's not too hard to make it work. But while doing so I
encountered some funny bits.

As far as I can tell the way we build shared libraries on aix with gcc isn't
correct:

Without -shared gcc won't know that it's building a shared library, which
afaict will prevent gcc from generating correct unwind info and we end up with
a statically linked copy of libgcc each time.

The naive thing of just adding -shared fails, but that's our fault:

ldd pgoutput.so
pgoutput.so needs:
Cannot find libgcc_s.a(shr.o)
         /usr/lib/libc.a(shr_64.o)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)

Makefile.aix has:
# -blibpath must contain ALL directories where we should look for libraries
libpath := $(shell echo $(subst -L,:,$(filter -L/%,$(LDFLAGS))) | sed -e's/ //g'):/usr/lib:/lib

but that's insufficient for gcc, because it won't find gcc's runtime lib. We
could force a build of the statically linked libgcc, but once it knows it's
generating with a shared library, a static libgcc unfortunately blows up the
size of the output considerably.

So I think we need something like

ifeq ($(GCC), yes)
libpath := $(libpath):$(dir $(shell gcc -print-libgcc-file-name))
endif

although deferring the computation of that would be nicer, but would require
some cleanup before.


With that libraries do shrink a bit. E.g. cube.so goes from 140k to 96k.


Afaict there's no reason to generate lib<name>.a for extension .so's, right?


We have plenty of detritus that's vaguely AIX related. The common.mk rule to
generate SUBSYS.o isn't used (mea culpa), and backend/Makefile's postgres.o
rule hasn't been used for well over 20 years.


I'll send in a patch series tomorrow, too tired for today.

Greetings,

Andres Freund

  10. Re: static libpq (and other libraries) overwritten on aix

    Noah Misch <noah@leadboat.com> — 2022-08-20T16:57:13Z

    On Sat, Aug 20, 2022 at 01:35:22AM -0700, Andres Freund wrote:
> Afaict there's no reason to generate lib<name>.a for extension .so's, right?

Right.  We install cube.so, not any *cube.a file.

  11. Re: static libpq (and other libraries) overwritten on aix

    Andres Freund <andres@anarazel.de> — 2022-08-20T17:42:13Z

    Hi,

On 2022-08-20 01:35:22 -0700, Andres Freund wrote:
> I'll send in a patch series tomorrow, too tired for today.

Here it goes.

0001 aix: Fix SHLIB_EXPORTS reference in VPATH builds

  That's mostly so I could even build. It's not quite right in the sense that
  we don't depend on the file, but that's a preexisting issue. Could be folded
  in with 0005, which fixes that aspect. Or it could be backpatched as the
  minimal fix.


0002 Remove SUBSYS.o rule in common.mk, hasn't been used in a long time
0003 Remove rule to generate postgres.o, not needed for 20+ years

  Both obvious, I think.


0004 aix: when building with gcc, tell gcc we're building a shared library

  That's the gcc -shared issue I explained in the email I'm replying to.

  We should probably consider building executables with -shared-libgcc too,
  that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But
  I've not done that here.


0005 aix: No need to use mkldexport when we want to export all symbols

  This makes the building of shared libraries a lot more similar to other
  platforms. Export files are only used when an exports.txt is present and
  there's no more intermediary static libraries.


0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility

  This isn't strictly speaking part of the same "thread" of work, but I don't
  want to touch aix more often than I have too... I'll post it in the other
  thread too.

  I did just test that this passes at least some tests on aix with xlc and
  solaris with sunpro.

Greetings,

Andres

  12. Re: static libpq (and other libraries) overwritten on aix

    Andres Freund <andres@anarazel.de> — 2022-08-25T03:43:04Z

    Hi,

On 2022-08-20 10:42:13 -0700, Andres Freund wrote:
> On 2022-08-20 01:35:22 -0700, Andres Freund wrote:
> > I'll send in a patch series tomorrow, too tired for today.
> 
> Here it goes.

> 0001 aix: Fix SHLIB_EXPORTS reference in VPATH builds
> 
>   That's mostly so I could even build. It's not quite right in the sense that
>   we don't depend on the file, but that's a preexisting issue. Could be folded
>   in with 0005, which fixes that aspect. Or it could be backpatched as the
>   minimal fix.
> 
> 
> 0002 Remove SUBSYS.o rule in common.mk, hasn't been used in a long time
> 0003 Remove rule to generate postgres.o, not needed for 20+ years
> 
>   Both obvious, I think.

Pushed these, given that they're all pretty trivial.



> 0004 aix: when building with gcc, tell gcc we're building a shared library
> 
>   That's the gcc -shared issue I explained in the email I'm replying to.
> 
>   We should probably consider building executables with -shared-libgcc too,
>   that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But
>   I've not done that here.
> 
> 
> 0005 aix: No need to use mkldexport when we want to export all symbols
> 
>   This makes the building of shared libraries a lot more similar to other
>   platforms. Export files are only used when an exports.txt is present and
>   there's no more intermediary static libraries.
> 
> 
> 0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility
> 
>   This isn't strictly speaking part of the same "thread" of work, but I don't
>   want to touch aix more often than I have too... I'll post it in the other
>   thread too.
> 
>   I did just test that this passes at least some tests on aix with xlc and
>   solaris with sunpro.

Any comments here?

Greetings,

Andres Freund

  13. Re: static libpq (and other libraries) overwritten on aix

    Noah Misch <noah@leadboat.com> — 2022-08-25T04:14:18Z

    On Wed, Aug 24, 2022 at 08:43:04PM -0700, Andres Freund wrote:
> On 2022-08-20 10:42:13 -0700, Andres Freund wrote:
> > 0004 aix: when building with gcc, tell gcc we're building a shared library
> > 
> >   That's the gcc -shared issue I explained in the email I'm replying to.
> > 
> >   We should probably consider building executables with -shared-libgcc too,
> >   that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But
> >   I've not done that here.
> > 
> > 
> > 0005 aix: No need to use mkldexport when we want to export all symbols
> > 
> >   This makes the building of shared libraries a lot more similar to other
> >   platforms. Export files are only used when an exports.txt is present and
> >   there's no more intermediary static libraries.
> > 
> > 
> > 0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility
> > 
> >   This isn't strictly speaking part of the same "thread" of work, but I don't
> >   want to touch aix more often than I have too... I'll post it in the other
> >   thread too.
> > 
> >   I did just test that this passes at least some tests on aix with xlc and
> >   solaris with sunpro.
> 
> Any comments here?

I don't know much about them, but they sound like the sort of thing that can't
cause subtle bugs.  If they build and test the first time, they're probably
valid.  You may as well push them.