Horton

Thread

Commits

  1. On macOS, use -isysroot in link steps as well as compile steps.

  1. [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    James Hilliard <james.hilliard1@gmail.com> — 2020-11-20T00:33:14Z

    It would appear weak symbol linking is not handled properly without
'isysroot' parameter passed to linker.

Fixes:
Undefined symbols for architecture x86_64:
  "___darwin_check_fd_set_overflow", referenced from:
      _ClientAuthentication in auth.o
      _pgstat_init in pgstat.o
      _ServerLoop in postmaster.o
ld: symbol(s) not found for architecture x86_64

See:
https://github.com/qt/qtwebengine-chromium/commit/d5c4b6230b7f915f6e044e230c0c575249938400
---
 configure           | 4 +++-
 configure.ac        | 4 +++-
 src/template/darwin | 2 ++
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index ace4ed5dec..92b089907e 100755
--- a/configure
+++ b/configure
@@ -19041,7 +19041,6 @@ if test "$with_readline" = yes; then
 else
   link_test_func=exit
 fi
-
 if test "$PORTNAME" = "darwin"; then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC supports -Wl,-dead_strip_dylibs" >&5
 $as_echo_n "checking whether $CC supports -Wl,-dead_strip_dylibs... " >&6; }
@@ -19194,6 +19193,9 @@ _ACEOF
 # we've finished all configure checks that depend on CPPFLAGS.
 if test x"$PG_SYSROOT" != x; then
   CPPFLAGS=`echo "$CPPFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+  if test "$PORTNAME" = "darwin"; then
+    LDFLAGS=`echo "$LDFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+  fi
 fi
 
 
diff --git a/configure.ac b/configure.ac
index 5b91c83fd0..a82cbbc147 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2326,7 +2326,6 @@ if test "$with_readline" = yes; then
 else
   link_test_func=exit
 fi
-
 if test "$PORTNAME" = "darwin"; then
   PGAC_PROG_CC_LDFLAGS_OPT([-Wl,-dead_strip_dylibs], $link_test_func)
 elif test "$PORTNAME" = "openbsd"; then
@@ -2362,6 +2361,9 @@ AC_SUBST(PG_VERSION_NUM)
 # we've finished all configure checks that depend on CPPFLAGS.
 if test x"$PG_SYSROOT" != x; then
   CPPFLAGS=`echo "$CPPFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+  if test "$PORTNAME" = "darwin"; then
+    LDFLAGS=`echo "$LDFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+  fi
 fi
 AC_SUBST(PG_SYSROOT)
 
diff --git a/src/template/darwin b/src/template/darwin
index f4d4e9d7cf..83d20a8cdd 100644
--- a/src/template/darwin
+++ b/src/template/darwin
@@ -11,6 +11,8 @@ fi
 if test x"$PG_SYSROOT" != x"" ; then
   if test -d "$PG_SYSROOT" ; then
     CPPFLAGS="-isysroot $PG_SYSROOT $CPPFLAGS"
+    # Prevent undefined symbols errors for weak linked symbols.
+    LDFLAGS="-isysroot $PG_SYSROOT $LDFLAGS"
   else
     PG_SYSROOT=""
   fi
-- 
2.29.2

  2. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    Tom Lane <tgl@sss.pgh.pa.us> — 2020-11-20T00:40:23Z

    James Hilliard <james.hilliard1@gmail.com> writes:
> It would appear weak symbol linking is not handled properly without
> 'isysroot' parameter passed to linker.

Nobody else has reported this problem, so maybe you should tell us
how to reproduce it before you suggest a build process change.

(And yes, we have buildfarm members testing Xcode > 11.4 ...)

			regards, tom lane

  3. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    James Hilliard <james.hilliard1@gmail.com> — 2020-11-20T00:45:29Z

    On Thu, Nov 19, 2020 at 5:40 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> James Hilliard <james.hilliard1@gmail.com> writes:
> > It would appear weak symbol linking is not handled properly without
> > 'isysroot' parameter passed to linker.
>
> Nobody else has reported this problem, so maybe you should tell us
> how to reproduce it before you suggest a build process change.
Hmm, maybe it's a more recent issue then, I took the version number
from the qt patch assuming it was the same issue, I hit it trying to build
master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7.
>
> (And yes, we have buildfarm members testing Xcode > 11.4 ...)
>
>                         regards, tom lane

  4. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    James Hilliard <james.hilliard1@gmail.com> — 2020-11-20T01:02:37Z

    I think I figured it out, it only happens on systems where the Xcode version is
newer than the OS it seems.

See:
https://forum.unity.com/threads/il2cpp-macstandalone-and-xcode-11-4.855187/

So looks like the failure happens due to the system sysroot or something like
that missing symbols from the Xcode sysroot, so I think this patch is
still correct
in that case as it forces the use of the Xcode sysroot for linking which should
be what we want as we want it to match sysroot passed to the compiler.

On Thu, Nov 19, 2020 at 5:45 PM James Hilliard
<james.hilliard1@gmail.com> wrote:
>
> On Thu, Nov 19, 2020 at 5:40 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > James Hilliard <james.hilliard1@gmail.com> writes:
> > > It would appear weak symbol linking is not handled properly without
> > > 'isysroot' parameter passed to linker.
> >
> > Nobody else has reported this problem, so maybe you should tell us
> > how to reproduce it before you suggest a build process change.
> Hmm, maybe it's a more recent issue then, I took the version number
> from the qt patch assuming it was the same issue, I hit it trying to build
> master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7.
> >
> > (And yes, we have buildfarm members testing Xcode > 11.4 ...)
> >
> >                         regards, tom lane

  5. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    Tom Lane <tgl@sss.pgh.pa.us> — 2020-11-20T01:04:49Z

    James Hilliard <james.hilliard1@gmail.com> writes:
> Hmm, maybe it's a more recent issue then, I took the version number
> from the qt patch assuming it was the same issue, I hit it trying to build
> master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7.

Hm, maybe you're using some unusual configure options or weird build
environment?

The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7)
and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel).
Maybe you're found some corner case in between those, but I guess it's
more likely due to a configuration choice.

			regards, tom lane

  6. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    James Hilliard <james.hilliard1@gmail.com> — 2020-11-20T01:55:18Z

    On Thu, Nov 19, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> James Hilliard <james.hilliard1@gmail.com> writes:
> > Hmm, maybe it's a more recent issue then, I took the version number
> > from the qt patch assuming it was the same issue, I hit it trying to build
> > master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7.
>
> Hm, maybe you're using some unusual configure options or weird build
> environment?
>
> The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7)
> and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel).
> Maybe you're found some corner case in between those, but I guess it's
> more likely due to a configuration choice.
I guess to verify one could try compiling with Xcode 12.2 on catalina 10.15.7.

This looks to be the function using the missing symbols, called by
FD_SET in postgres:

__header_always_inline int
__darwin_check_fd_set(int _a, const void *_b)
{
#ifdef __clang__
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wunguarded-availability-new"
#endif
    if ((uintptr_t)&__darwin_check_fd_set_overflow != (uintptr_t) 0) {
#if defined(_DARWIN_UNLIMITED_SELECT) || defined(_DARWIN_C_SOURCE)
        return __darwin_check_fd_set_overflow(_a, _b, 1);
#else
        return __darwin_check_fd_set_overflow(_a, _b, 0);
#endif
    } else {
        return 1;
    }
#ifdef __clang__
#pragma clang diagnostic pop
#endif
}

Which is located here:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include/sys/_types/_fd_def.h

The header file indicates the missing symbols are only available in 11.0 and up:
int __darwin_check_fd_set_overflow(int, const void *, int)
__API_AVAILABLE(macosx(11.0), ios(14.0), tvos(14.0), watchos(7.0));

So I think it's fairly clear what's happening now, the linker would normally use
the system libraries to determine that those missing symbols are
weakly linked, but since
the base OS is too old that fails unless we tell the linker where to
look(in the SDK).
>
>                         regards, tom lane

  7. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    Tom Lane <tgl@sss.pgh.pa.us> — 2020-11-20T02:20:11Z

    James Hilliard <james.hilliard1@gmail.com> writes:
> On Thu, Nov 19, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7)
>> and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel).
>> Maybe you're found some corner case in between those, but I guess it's
>> more likely due to a configuration choice.

> I guess to verify one could try compiling with Xcode 12.2 on catalina 10.15.7.

To check this, I updated my laptop (still on Catalina 10.15.7) to latest
Xcode, 12.2 (12B45b), from 12.0 --- and behold, I could duplicate it:

Undefined symbols for architecture x86_64:
  "___darwin_check_fd_set_overflow", referenced from:
      _ClientAuthentication in auth.o
      _pgstat_init in pgstat.o
      _ServerLoop in postmaster.o
ld: symbol(s) not found for architecture x86_64

However ... it then occurred to me to blow away my ccache and accache,
and after rebuilding from scratch, everything's fine.  So, are you
using ccache?

			regards, tom lane

  8. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    James Hilliard <james.hilliard1@gmail.com> — 2020-11-20T02:30:45Z

    On Thu, Nov 19, 2020 at 7:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> James Hilliard <james.hilliard1@gmail.com> writes:
> > On Thu, Nov 19, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7)
> >> and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel).
> >> Maybe you're found some corner case in between those, but I guess it's
> >> more likely due to a configuration choice.
>
> > I guess to verify one could try compiling with Xcode 12.2 on catalina 10.15.7.
>
> To check this, I updated my laptop (still on Catalina 10.15.7) to latest
> Xcode, 12.2 (12B45b), from 12.0 --- and behold, I could duplicate it:
>
> Undefined symbols for architecture x86_64:
>   "___darwin_check_fd_set_overflow", referenced from:
>       _ClientAuthentication in auth.o
>       _pgstat_init in pgstat.o
>       _ServerLoop in postmaster.o
> ld: symbol(s) not found for architecture x86_64
>
> However ... it then occurred to me to blow away my ccache and accache,
> and after rebuilding from scratch, everything's fine.  So, are you
> using ccache?
Don't think so, I first hit this issue on a clean clone of master and
I don't think
I even have ccache or accache(not familiar with this variant) installed at all.
Are you sure something else didn't get changed when you cleared the caches?
Is configure using the right SDK directory?
>
>                         regards, tom lane

  9. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    Tom Lane <tgl@sss.pgh.pa.us> — 2020-11-20T02:48:52Z

    I wrote:
> However ... it then occurred to me to blow away my ccache and accache,
> and after rebuilding from scratch, everything's fine.  So, are you
> using ccache?

Oh, scratch that, I fat-fingered the experiment somehow.  The issue
is still there.  Still, I'm hesitant to apply the fix you suggest,
because of the law of unintended consequences.  In particular, I'm
afraid that using -isysroot in the link step might result in executables
that are bound to that sysroot and will not work if it's not there
--- thus causing a problem for packagers trying to distribute PG
to non-developers.

We had a great deal of trouble in the initial experiments with
-isysroot, cf

https://www.postgresql.org/message-id/flat/20840.1537850987%40sss.pgh.pa.us

so I'm not that eager to mess with it.  I'm inclined to think that
this is more a case of "Apple broke ABI and they ought to fix that".

			regards, tom lane

  10. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    James Hilliard <james.hilliard1@gmail.com> — 2020-11-20T03:22:27Z

    On Thu, Nov 19, 2020 at 7:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> I wrote:
> > However ... it then occurred to me to blow away my ccache and accache,
> > and after rebuilding from scratch, everything's fine.  So, are you
> > using ccache?
>
> Oh, scratch that, I fat-fingered the experiment somehow.  The issue
> is still there.  Still, I'm hesitant to apply the fix you suggest,
> because of the law of unintended consequences.  In particular, I'm
> afraid that using -isysroot in the link step might result in executables
> that are bound to that sysroot and will not work if it's not there
> --- thus causing a problem for packagers trying to distribute PG
> to non-developers.

Since apple doesn't actually put real dylib's in the SDK path I think
we're fine:

$ find /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
-name "*.dylib"
$

Instead they use .tbd or text-based stub files there which appear to provide
enough info for the linker to determine that the undefined symbols are
weak linked. And since the linker doesn't actually link against the
.tbd file paths
in the way it would a dylib I don't think the SDK path could get used
accidentally.

>
> We had a great deal of trouble in the initial experiments with
> -isysroot, cf
>
> https://www.postgresql.org/message-id/flat/20840.1537850987%40sss.pgh.pa.us
>
> so I'm not that eager to mess with it.  I'm inclined to think that
> this is more a case of "Apple broke ABI and they ought to fix that".

This seems to be a longstanding issue, I expect they don't care much about
autotools compared with their normal Xcode project build system. So probably
better to just give this a try and see how it goes as it's likely to
keep popping up
otherwise.

>
>                         regards, tom lane

  11. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    Tom Lane <tgl@sss.pgh.pa.us> — 2020-11-20T04:02:13Z

    James Hilliard <james.hilliard1@gmail.com> writes:
> On Thu, Nov 19, 2020 at 7:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Oh, scratch that, I fat-fingered the experiment somehow.  The issue
>> is still there.  Still, I'm hesitant to apply the fix you suggest,
>> because of the law of unintended consequences.  In particular, I'm
>> afraid that using -isysroot in the link step might result in executables
>> that are bound to that sysroot and will not work if it's not there
>> --- thus causing a problem for packagers trying to distribute PG
>> to non-developers.

> Since apple doesn't actually put real dylib's in the SDK path I think
> we're fine:
> $ find /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
> -name "*.dylib"

True.  Also, while actual documentation on -isysroot seems to be damn
near nonexistent, all the example usages I can find on the net appear
to put it into both compile and link steps.  So maybe we're just doing
it wrong and happened to get away with that this long.

> This seems to be a longstanding issue, I expect they don't care much about
> autotools compared with their normal Xcode project build system. So probably
> better to just give this a try and see how it goes as it's likely to
> keep popping up otherwise.

Yeah, the one saving grace here is that, unlike the last time, we're not
hard up against a release deadline.  If we push this now we can hope to
get a fair amount of testing from PG developers before we have to ship
it.

			regards, tom lane

  12. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    James Hilliard <james.hilliard1@gmail.com> — 2020-11-20T04:13:00Z

    On Thu, Nov 19, 2020 at 9:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> James Hilliard <james.hilliard1@gmail.com> writes:
> > On Thu, Nov 19, 2020 at 7:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> Oh, scratch that, I fat-fingered the experiment somehow.  The issue
> >> is still there.  Still, I'm hesitant to apply the fix you suggest,
> >> because of the law of unintended consequences.  In particular, I'm
> >> afraid that using -isysroot in the link step might result in executables
> >> that are bound to that sysroot and will not work if it's not there
> >> --- thus causing a problem for packagers trying to distribute PG
> >> to non-developers.
>
> > Since apple doesn't actually put real dylib's in the SDK path I think
> > we're fine:
> > $ find /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
> > -name "*.dylib"
>
> True.  Also, while actual documentation on -isysroot seems to be damn
> near nonexistent, all the example usages I can find on the net appear
> to put it into both compile and link steps.  So maybe we're just doing
> it wrong and happened to get away with that this long.
Yeah, pretty sure that's what happened from what I'm seeing.
>
> > This seems to be a longstanding issue, I expect they don't care much about
> > autotools compared with their normal Xcode project build system. So probably
> > better to just give this a try and see how it goes as it's likely to
> > keep popping up otherwise.
>
> Yeah, the one saving grace here is that, unlike the last time, we're not
> hard up against a release deadline.  If we push this now we can hope to
> get a fair amount of testing from PG developers before we have to ship
> it.
>
>                         regards, tom lane

  13. Re: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.

    Tom Lane <tgl@sss.pgh.pa.us> — 2020-11-20T05:59:47Z

    James Hilliard <james.hilliard1@gmail.com> writes:
> On Thu, Nov 19, 2020 at 9:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> True.  Also, while actual documentation on -isysroot seems to be damn
>> near nonexistent, all the example usages I can find on the net appear
>> to put it into both compile and link steps.  So maybe we're just doing
>> it wrong and happened to get away with that this long.

> Yeah, pretty sure that's what happened from what I'm seeing.

My buildfarm critters seem to be OK with this, so pushed.

			regards, tom lane