Horton
Menu

Thread

  1. improving PL/Python builds on OS X

    Peter Eisentraut <peter_e@gmx.net> — 2012-12-21T17:45:13Z

    The PL/Python build on OS X is currently hardcoded to use the system
Python install.  If you try to override this when running configure, you
get a mysterious mix-and-match build.  If you want to build against your
own Python build, or MacPorts or Homebrew, you can't.

This is straightforward to fix.  In configure, besides checking Python
include and library paths, we can also check whether it's a framework
build and the right parameters for that.  The attached patch does that
and does the job for me.  Please test it.

One constraint, which is explained in the comment in
src/pl/plpython/Makefile is that in Python <2.5, there is no official
way to detect either framework builds or shared libpython builds, so we
can't support those versions on OS X, at least without more hardcoding
of things.  I'd rather phase some of that out, but if someone needs to
continue to use Python 2.4 or earlier on OS X, let me know.  (Or more
proper fix would be to split DLSUFFIX into two variables, but that seems
more work than it's worth right now.)

  2. Re: improving PL/Python builds on OS X

    Dave Page <dpage@pgadmin.org> — 2013-05-08T15:51:12Z

    On Fri, Dec 21, 2012 at 5:45 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> The PL/Python build on OS X is currently hardcoded to use the system
> Python install.  If you try to override this when running configure, you
> get a mysterious mix-and-match build.  If you want to build against your
> own Python build, or MacPorts or Homebrew, you can't.
>
> This is straightforward to fix.  In configure, besides checking Python
> include and library paths, we can also check whether it's a framework
> build and the right parameters for that.  The attached patch does that
> and does the job for me.  Please test it.
>
> One constraint, which is explained in the comment in
> src/pl/plpython/Makefile is that in Python <2.5, there is no official
> way to detect either framework builds or shared libpython builds, so we
> can't support those versions on OS X, at least without more hardcoding
> of things.  I'd rather phase some of that out, but if someone needs to
> continue to use Python 2.4 or earlier on OS X, let me know.  (Or more
> proper fix would be to split DLSUFFIX into two variables, but that seems
> more work than it's worth right now.)

This patch seems to have broken the build for our installers for 9.3.
Because we need a consistent build of the PL interpretors on all the
platforms we support, we use the ActiveState distributions of Perl,
Python and TCL (we can't rely on vendor supplied packages, because
their versions vary between different Linux distros and different OS X
versions). However, ActivePython doesn't include a shared library,
which this change seems to require.

Can that requirement be reverted?

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  3. Re: improving PL/Python builds on OS X

    Peter Eisentraut <peter_e@gmx.net> — 2013-05-08T16:34:29Z

    On 5/8/13 11:51 AM, Dave Page wrote:
> This patch seems to have broken the build for our installers for 9.3.
> Because we need a consistent build of the PL interpretors on all the
> platforms we support, we use the ActiveState distributions of Perl,
> Python and TCL (we can't rely on vendor supplied packages, because
> their versions vary between different Linux distros and different OS X
> versions). However, ActivePython doesn't include a shared library,
> which this change seems to require.
> 
> Can that requirement be reverted?

There was no change in this regard.  A shared library was always
required on OS X.

ActivePython does include a shared library.  I just tried it and it
builds fine.

  4. Re: improving PL/Python builds on OS X

    Dave Page <dpage@pgadmin.org> — 2013-05-08T17:24:33Z

    On Wed, May 8, 2013 at 5:34 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On 5/8/13 11:51 AM, Dave Page wrote:
>> This patch seems to have broken the build for our installers for 9.3.
>> Because we need a consistent build of the PL interpretors on all the
>> platforms we support, we use the ActiveState distributions of Perl,
>> Python and TCL (we can't rely on vendor supplied packages, because
>> their versions vary between different Linux distros and different OS X
>> versions). However, ActivePython doesn't include a shared library,
>> which this change seems to require.
>>
>> Can that requirement be reverted?
>
> There was no change in this regard.  A shared library was always
> required on OS X.

It's failing on Linux. Even worse, it configures fine and then builds
without error. There is a message spewed out by configure, but it
doesn't contain the words warning or error. Given that I explicitly
said I wanted Python support when I ran configure, it should certainly
fail with an error at configure time. We only noticed this was a
problem when the QA guys started diving in to more detailed tested, as
we don't watch for every message in the 50+ MB of logs our automated
build systems generate.

> ActivePython does include a shared library.  I just tried it and it
> builds fine.



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  5. Re: improving PL/Python builds on OS X

    Peter Eisentraut <peter_e@gmx.net> — 2013-05-09T01:26:32Z

    On Wed, 2013-05-08 at 18:24 +0100, Dave Page wrote:
> It's failing on Linux. Even worse, it configures fine and then builds
> without error. There is a message spewed out by configure, but it
> doesn't contain the words warning or error. Given that I explicitly
> said I wanted Python support when I ran configure, it should certainly
> fail with an error at configure time. We only noticed this was a
> problem when the QA guys started diving in to more detailed tested, as
> we don't watch for every message in the 50+ MB of logs our automated
> build systems generate.

It worked before because we used to allow linking shared libraries
against static libraries on some platforms.  But that was more or less a
lie because it doesn't work on 64-bit platforms.

ActiveState Python contains a static library with PIC files.  There is
no obvious way to detect that, which is why we don't support it
directly.  You can sort it out yourself by building with

    make shared_libpython=yes

In the long term, find a way to detect whether the library is usable.

  6. Re: improving PL/Python builds on OS X

    Dave Page <dpage@pgadmin.org> — 2013-05-09T07:09:22Z

    On Thu, May 9, 2013 at 2:26 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On Wed, 2013-05-08 at 18:24 +0100, Dave Page wrote:
>> It's failing on Linux. Even worse, it configures fine and then builds
>> without error. There is a message spewed out by configure, but it
>> doesn't contain the words warning or error. Given that I explicitly
>> said I wanted Python support when I ran configure, it should certainly
>> fail with an error at configure time. We only noticed this was a
>> problem when the QA guys started diving in to more detailed tested, as
>> we don't watch for every message in the 50+ MB of logs our automated
>> build systems generate.
>
> It worked before because we used to allow linking shared libraries
> against static libraries on some platforms.  But that was more or less a
> lie because it doesn't work on 64-bit platforms.
>
> ActiveState Python contains a static library with PIC files.  There is
> no obvious way to detect that, which is why we don't support it
> directly.  You can sort it out yourself by building with
>
>     make shared_libpython=yes

OK, thanks - I'll try that.

I assume you'll fix the configure script so it actually errors out if
plpython cannot be built, but is explicitly requested?

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  7. Re: improving PL/Python builds on OS X

    Dave Page <dpage@pgadmin.org> — 2013-05-09T07:25:27Z

    On Thu, May 9, 2013 at 8:09 AM, Dave Page <dpage@pgadmin.org> wrote:
> On Thu, May 9, 2013 at 2:26 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> On Wed, 2013-05-08 at 18:24 +0100, Dave Page wrote:
>>> It's failing on Linux. Even worse, it configures fine and then builds
>>> without error. There is a message spewed out by configure, but it
>>> doesn't contain the words warning or error. Given that I explicitly
>>> said I wanted Python support when I ran configure, it should certainly
>>> fail with an error at configure time. We only noticed this was a
>>> problem when the QA guys started diving in to more detailed tested, as
>>> we don't watch for every message in the 50+ MB of logs our automated
>>> build systems generate.
>>
>> It worked before because we used to allow linking shared libraries
>> against static libraries on some platforms.  But that was more or less a
>> lie because it doesn't work on 64-bit platforms.
>>
>> ActiveState Python contains a static library with PIC files.  There is
>> no obvious way to detect that, which is why we don't support it
>> directly.  You can sort it out yourself by building with
>>
>>     make shared_libpython=yes
>
> OK, thanks - I'll try that.
>
> I assume you'll fix the configure script so it actually errors out if
> plpython cannot be built, but is explicitly requested?

BTW - it's always worked fine for us on 64 bit machines with the past
few major releases of both PG and Python - are you saying that's pure
chance? For example:

postgres=# CREATE LANGUAGE plpython3u;
CREATE LANGUAGE
postgres=# CREATE FUNCTION pyversion() RETURNS text AS
postgres-# $$
postgres$# import sys
postgres$# return sys.version
postgres$# $$ LANGUAGE 'plpython3u';
CREATE FUNCTION
postgres=# SELECT pyversion();
                pyversion
-----------------------------------------
 3.2.2 (default, Sep  8 2011, 12:20:18) +
 [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)]
(1 row)

postgres=# SELECT version();
                                                    version
---------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC)
4.1.2 20080704 (Red Hat 4.1.2-52), 64-bit
(1 row)

postgres=#

(that's with fresh installations of ActivePython 3.2.2.3 and EDB's
PostgreSQL 9.2.4 package, on CentOS 6).

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  8. Re: improving PL/Python builds on OS X

    Peter Eisentraut <peter_e@gmx.net> — 2013-05-09T14:09:39Z

    On 5/9/13 3:09 AM, Dave Page wrote:
> I assume you'll fix the configure script so it actually errors out if
> plpython cannot be built, but is explicitly requested?

That is ancient behavior, which I'm not fond of, but now is not the time
to change that.

  9. Re: improving PL/Python builds on OS X

    Peter Eisentraut <peter_e@gmx.net> — 2013-05-09T14:11:09Z

    On 5/9/13 3:25 AM, Dave Page wrote:
> BTW - it's always worked fine for us on 64 bit machines with the past
> few major releases of both PG and Python - are you saying that's pure
> chance?

It works because ActiveState Python has PIC inside a static library.
But we have no straightforward way of knowing that (AFAIK), other than
observing whether the build result crashes or not.

  10. Re: improving PL/Python builds on OS X

    Dave Page <dpage@pgadmin.org> — 2013-05-09T14:52:08Z

    On Thu, May 9, 2013 at 3:09 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On 5/9/13 3:09 AM, Dave Page wrote:
>> I assume you'll fix the configure script so it actually errors out if
>> plpython cannot be built, but is explicitly requested?
>
> That is ancient behavior, which I'm not fond of, but now is not the time
> to change that.

Eh? Beta is precisely the time to find and fix bugs like this.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  11. Re: improving PL/Python builds on OS X

    Peter Eisentraut <peter_e@gmx.net> — 2013-05-09T21:06:57Z

    On 5/9/13 10:52 AM, Dave Page wrote:
> On Thu, May 9, 2013 at 3:09 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> On 5/9/13 3:09 AM, Dave Page wrote:
>>> I assume you'll fix the configure script so it actually errors out if
>>> plpython cannot be built, but is explicitly requested?
>>
>> That is ancient behavior, which I'm not fond of, but now is not the time
>> to change that.
> 
> Eh? Beta is precisely the time to find and fix bugs like this.

It's not a bug, it's working as designed (not by me).

It's certainly worth reexamining, but there are so many variations in
this area (and Perl and Tcl are set up the same way) that I don't want
to mess around in it now.

A better approach for now would be to look for a configuration item in
Python that tells us whether libpython contains PIC.



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

  12. Re: improving PL/Python builds on OS X

    Dave Page <dpage@pgadmin.org> — 2013-05-09T21:31:10Z

    On Thu, May 9, 2013 at 10:06 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On 5/9/13 10:52 AM, Dave Page wrote:
>> On Thu, May 9, 2013 at 3:09 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>> On 5/9/13 3:09 AM, Dave Page wrote:
>>>> I assume you'll fix the configure script so it actually errors out if
>>>> plpython cannot be built, but is explicitly requested?
>>>
>>> That is ancient behavior, which I'm not fond of, but now is not the time
>>> to change that.
>>
>> Eh? Beta is precisely the time to find and fix bugs like this.
>
> It's not a bug, it's working as designed (not by me).

Well, design bug then. It's clearly a nasty POLA violation at minimum.


--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

  13. Re: improving PL/Python builds on OS X

    Peter Eisentraut <peter_e@gmx.net> — 2013-05-10T01:47:46Z

    On Thu, 2013-05-09 at 10:11 -0400, Peter Eisentraut wrote:
> On 5/9/13 3:25 AM, Dave Page wrote:
> > BTW - it's always worked fine for us on 64 bit machines with the past
> > few major releases of both PG and Python - are you saying that's pure
> > chance?
> 
> It works because ActiveState Python has PIC inside a static library.
> But we have no straightforward way of knowing that (AFAIK), other than
> observing whether the build result crashes or not.

After further digging, it seems to me that their build is not a standard
build.  They must be patching in compiler options through the backdoor
somehow.  Their config/Makefile has

BASECFLAGS=      -fno-strict-aliasing -fPIC

meaning that they compile *everything* with those options.  But that's
not something that the standard Python configure script can produce.

Again, reliably detecting that might be difficult.



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

  14. Re: improving PL/Python builds on OS X

    Dave Page <dpage@pgadmin.org> — 2013-05-10T06:46:39Z

    On Fri, May 10, 2013 at 2:47 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On Thu, 2013-05-09 at 10:11 -0400, Peter Eisentraut wrote:
>> On 5/9/13 3:25 AM, Dave Page wrote:
>> > BTW - it's always worked fine for us on 64 bit machines with the past
>> > few major releases of both PG and Python - are you saying that's pure
>> > chance?
>>
>> It works because ActiveState Python has PIC inside a static library.
>> But we have no straightforward way of knowing that (AFAIK), other than
>> observing whether the build result crashes or not.
>
> After further digging, it seems to me that their build is not a standard
> build.  They must be patching in compiler options through the backdoor
> somehow.  Their config/Makefile has
>
> BASECFLAGS=      -fno-strict-aliasing -fPIC
>
> meaning that they compile *everything* with those options.  But that's
> not something that the standard Python configure script can produce.

Huh, interesting. I used to have a contact at ActiveState - I'll see
if I can get some more details.

> Again, reliably detecting that might be difficult.

Yeah, that's understandable. As long as I can force it though (and
your suggestion on that front seems to work fine), I'm happy.

Thanks.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers