Horton

Thread

Commits

  1. plpython: add missing plpython.h include to plpy_plpymodule.h

  2. plpython: Adjust docs after removal of Python 2 support.

  3. plpython: Code cleanup related to removal of Python 2 support.

  4. plpython: Remove regression test infrastructure for Python 2.

  5. plpython: Remove plpythonu, plpython2u and associated transform extensions.

  6. plpython: Reject Python 2 during build configuration.

  7. Add traceback information to PL/Python errors

  1. Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-10-31T18:45:48Z

    Hi,

The last release of python2 was 2.7.18 released 2020-04-20, with support
already having ended before that 2020-01-01.

To me it seems time to drop plpython2 support. Supporting plpython2
until ~7 years after python2 is EOL is already quite long... It'd be one
thing if it were completely painless, but imo it's not.

I was about to list better plpython2/3 support (including the regex
replacements for the regress tests) as a TODO for the meson proposal,
but it doesn't really seem worth investing in that...

Greetings,

Andres Freund

  2. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-10-31T18:49:22Z

    Andres Freund <andres@anarazel.de> writes:
> To me it seems time to drop plpython2 support. Supporting plpython2
> until ~7 years after python2 is EOL is already quite long... It'd be one
> thing if it were completely painless, but imo it's not.

7 years?  Oh, you're envisioning dropping plpython2 in HEAD but keeping
it alive in the back branches.  Yeah, it's a bit hard to justify
continuing support in HEAD.

			regards, tom lane

  3. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-10-31T19:05:42Z

    Hi,

On 2021-10-31 14:49:22 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > To me it seems time to drop plpython2 support. Supporting plpython2
> > until ~7 years after python2 is EOL is already quite long... It'd be one
> > thing if it were completely painless, but imo it's not.
> 
> 7 years?  Oh, you're envisioning dropping plpython2 in HEAD but keeping
> it alive in the back branches.

Yea. It's annoying to support python2, but I don't think it's severe
enough to consider dropping support in released branches. We might get
to that point for the newer released versions, but for now...


> Yeah, it's a bit hard to justify continuing support in HEAD.

Cool.

Greetings,

Andres Freund

  4. Re: Time to drop plpython2?

    Thomas Munro <thomas.munro@gmail.com> — 2021-10-31T20:00:49Z

    On Mon, Nov 1, 2021 at 8:05 AM Andres Freund <andres@anarazel.de> wrote:
> > Yeah, it's a bit hard to justify continuing support in HEAD.

+1, it's dropping out of distros, it'd be unsupportable without
unreasonable effort.

  5. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-11-01T05:53:01Z

    On 31.10.21 19:45, Andres Freund wrote:
> To me it seems time to drop plpython2 support. Supporting plpython2
> until ~7 years after python2 is EOL is already quite long... It'd be one
> thing if it were completely painless, but imo it's not.
> 
> I was about to list better plpython2/3 support (including the regex
> replacements for the regress tests) as a TODO for the meson proposal,
> but it doesn't really seem worth investing in that...

Usually, we have dropped support for older Python versions when 
continued support would be a burden for some ongoing development 
project.  This would certainly count as one of those, I think.

  6. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-11-04T18:58:54Z

    I see you have posted a patch for this in the meson thread 
(https://www.postgresql.org/message-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch). 
  Here is my review of that.

I would change the search order in configure from

PGAC_PATH_PROGS(PYTHON, [python python3 python2])

to

PGAC_PATH_PROGS(PYTHON, [python3 python])

This makes sure you don't accidentally pick up a "python" binary that 
points to Python 2.  This would otherwise immediately generate lots of 
build failures on older platforms that still have Python 2 around.

You left two FIXME sections in plpython.h.  AFAICT, we can make the 
substitutions corresponding to those #define's in the source code and 
remove those blocks.

src/pl/plpython/expected/README should be updated for the removed files.

Some documentation updates are needed.  I see possibly relevant text in 
the following files:

hstore.sgml
install-windows.sgml
installation.sgml
json.sgml
libpq.sgml
plpython.sgml
ref/comment.sgml (examples)
ref/create_transform.sgml (examples)
ref/drop_transform.sgml (examples)
standalone-profile.xsl

src/tools/msvc/ has lots of Python-related content.

More stuff to clean up:

contrib/hstore_plpython/.gitignore
contrib/jsonb_plpython/.gitignore
contrib/ltree_plpython/.gitignore
src/pl/plpython/.gitignore

Finally, morally related, there is some Python 2/3 compat code in 
contrib/unaccent/generate_unaccent_rules.py that could be removed. 
Also, arguably, change the shebang line in that script.

  7. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-11-04T19:54:43Z

    Hi,

On 2021-11-04 19:58:54 +0100, Peter Eisentraut wrote:
> I see you have posted a patch for this in the meson thread (https://www.postgresql.org/message-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch).

Yea, I was planning to post that here after a bit more polish. I mostly wanted
to get rid of the gross gross hack I did for the transmutation of the
regression test files.


> Here is my review of that.
> 
> I would change the search order in configure from
> 
> PGAC_PATH_PROGS(PYTHON, [python python3 python2])

> to
> 
> PGAC_PATH_PROGS(PYTHON, [python3 python])
> 
> This makes sure you don't accidentally pick up a "python" binary that points
> to Python 2.  This would otherwise immediately generate lots of build
> failures on older platforms that still have Python 2 around.

Yes, we should do that, at least for now. I did see build failures that
required me to specify the python version to avoid issues around it.


> You left two FIXME sections in plpython.h.  AFAICT, we can make the
> substitutions corresponding to those #define's in the source code and remove
> those blocks.

Yea, it shouldn't be hard. Just required more time than I had to send it out
before Nov 1st ;)

With meson I'd do a version: '>= 3' or such, to filter out a bare 'python'
being python2, but I don't think there's an equally trivial way to do that
with autoconf.


> Finally, morally related, there is some Python 2/3 compat code in
> contrib/unaccent/generate_unaccent_rules.py that could be removed. Also,
> arguably, change the shebang line in that script.

Hm. So far the python used for plpython and python for code generation etc is
independent. I don't know if plpython actually can be cross-compiled, but if
so, they'd have to be independent.  Otherwise I'd say we should just invoke
contrib/unaccent/generate_unaccent_rules.py with a python chosen by
configure/meson, rather than relying on a shebang that can't be adjusted
without modifying source code.


Another thing I wondered about is what we want to do with the extension
names. Do we want to leave it named plpython3u? Do we want to have a plpython
that depends on plpython3u?

I'd be inclined to just keep it at plpython3u for now, but there's an argument
that going for plpython would be better long term: Presumably there will be
python 4 at some point - but I'd expect that to not be a breaking release,
given the disaster that python 3 is. Making a non-versioned name better?

Greetings,

Andres Freund

  8. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-11-04T20:26:39Z

    Andres Freund <andres@anarazel.de> writes:
> Another thing I wondered about is what we want to do with the extension
> names. Do we want to leave it named plpython3u? Do we want to have a plpython
> that depends on plpython3u?

I think we want to keep plpython3u.  Maybe we can point plpythonu
at that, but I'm concerned about the potential for user confusion.
In particular, I think there's a nonzero probability that someone
will choose to maintain plpython2u/plpythonu outside of core,
just because they still don't want to migrate their Python code.

> I'd be inclined to just keep it at plpython3u for now, but there's an argument
> that going for plpython would be better long term: Presumably there will be
> python 4 at some point - but I'd expect that to not be a breaking release,
> given the disaster that python 3 is. Making a non-versioned name better?

Meh.  If there is a python 4, I'd expect it to be named that way precisely
because it *is* a breaking release.  Why would we set ourselves up for
a repeat of this mess?

			regards, tom lane

  9. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-11-10T10:56:10Z

    On 04.11.21 20:54, Andres Freund wrote:
>> Finally, morally related, there is some Python 2/3 compat code in
>> contrib/unaccent/generate_unaccent_rules.py that could be removed. Also,
>> arguably, change the shebang line in that script.
> 
> Hm. So far the python used for plpython and python for code generation etc is
> independent. I don't know if plpython actually can be cross-compiled, but if
> so, they'd have to be independent.  Otherwise I'd say we should just invoke
> contrib/unaccent/generate_unaccent_rules.py with a python chosen by
> configure/meson, rather than relying on a shebang that can't be adjusted
> without modifying source code.

We don't rely on the shebang for running this (see Makefile).  I just 
see some trend of people changing shebang lines as a sort of signal, 
"this script has abandoned Python 2".  It's not very important.

> Another thing I wondered about is what we want to do with the extension
> names. Do we want to leave it named plpython3u? Do we want to have a plpython
> that depends on plpython3u?

I would tend to mirror what the Python community does with 
/usr/bin/python.  Right now, most people are removing /usr/bin/python 
and just provide /usr/bin/python3.  This would be analogous to what your 
patch does.  Some time in the future, they may add /usr/bin/python back, 
and then we could do the same, meaning add a plpythonu that depends on 
plpython3u.

  10. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-11-15T02:24:31Z

    ... btw, there's a fairly critical gating factor for any plan to drop
python2 support: the buildfarm.  I just counted, and there are exactly
as many members running python 2.x as 3.x (49 apiece), not counting
Windows machines that aren't running configure.  We can't commit
something that's going to make half the buildfarm go red.

(It's likely that some fraction of them do already have python3 installed,
in which case the search order change Peter recommended would be enough to
fix it.  But I'm sure not all do.)

This ties into the business about converting the build system to meson,
as that also requires python 3 --- with, IIUC, a higher minimum version
than we might otherwise need.  I'm disinclined to cause two separate
flag days for buildfarm owners, so what I now think is we ought to put
this idea on the shelf until we've finished that conversion or decided
we're not gonna do it.  We need to identify exactly what needs to be
installed before we start pestering the owners.

			regards, tom lane

  11. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-11-15T17:06:32Z

    Hi,

On 2021-11-14 21:24:31 -0500, Tom Lane wrote:
> ... btw, there's a fairly critical gating factor for any plan to drop
> python2 support: the buildfarm.  I just counted, and there are exactly
> as many members running python 2.x as 3.x (49 apiece), not counting
> Windows machines that aren't running configure.  We can't commit
> something that's going to make half the buildfarm go red.
> 
> (It's likely that some fraction of them do already have python3 installed,
> in which case the search order change Peter recommended would be enough to
> fix it.  But I'm sure not all do.)

How about committing the order change alone? That seems like something
warranted completely in isolation? Afterwards we can see how many run what and
go from there?


> This ties into the business about converting the build system to meson,
> as that also requires python 3 --- with, IIUC, a higher minimum version
> than we might otherwise need.  I'm disinclined to cause two separate
> flag days for buildfarm owners, so what I now think is we ought to put
> this idea on the shelf until we've finished that conversion or decided
> we're not gonna do it.  We need to identify exactly what needs to be
> installed before we start pestering the owners.

Yea, that's true.

Greetings,

Andres Freund

  12. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-11-15T17:19:51Z

    Andres Freund <andres@anarazel.de> writes:
> On 2021-11-14 21:24:31 -0500, Tom Lane wrote:
>> (It's likely that some fraction of them do already have python3 installed,
>> in which case the search order change Peter recommended would be enough to
>> fix it.  But I'm sure not all do.)

> How about committing the order change alone? That seems like something
> warranted completely in isolation? Afterwards we can see how many run what and
> go from there?

I don't think that's warranted.  The existing design is that we let
the user say which python is "python", and I do not think we should
change that in advance of actually dropping python2 support.

I was wondering about simply probing to see if python3 exists (and if
so, what version it is exactly), as an additional configure test that
doesn't hook into anything.  That would give us some information without
suddenly changing what is being tested.

			regards, tom lane

  13. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-11-15T18:12:46Z

    Hi,

On 2021-11-15 12:19:51 -0500, Tom Lane wrote:
> I don't think that's warranted.  The existing design is that we let
> the user say which python is "python", and I do not think we should
> change that in advance of actually dropping python2 support.

Hm. I think it'd be ok, given that python 2 is well past EOL. But I also see
your point.


> I was wondering about simply probing to see if python3 exists (and if
> so, what version it is exactly), as an additional configure test that
> doesn't hook into anything.  That would give us some information without
> suddenly changing what is being tested.

But this is probably a good compromise. Were you thinking of doing a proper
autoconf test or just putting something like python3 --version || true in
configure?

I guess it'd be easiest to interpret if we output the current PYTHON version
and, iff < 3.0, also output the 'python3' version?

Greetings,

Andres Freund

  14. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-11-15T18:18:22Z

    Andres Freund <andres@anarazel.de> writes:
> On 2021-11-15 12:19:51 -0500, Tom Lane wrote:
>> I was wondering about simply probing to see if python3 exists (and if
>> so, what version it is exactly), as an additional configure test that
>> doesn't hook into anything.  That would give us some information without
>> suddenly changing what is being tested.

> But this is probably a good compromise. Were you thinking of doing a proper
> autoconf test or just putting something like python3 --version || true in
> configure?

Hm, I had in mind an actual configure test; but since it's just a
temporary exploratory measure, shortcuts are fine.  However, I'm
not sure that what you suggest would result in capturing anything
in config.log.

> I guess it'd be easiest to interpret if we output the current PYTHON version
> and, iff < 3.0, also output the 'python3' version?

We'll already know if PYTHON is set, at least so far as the buildfarm
is concerned.

			regards, tom lane

  15. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-11-15T18:52:19Z

    On 15.11.21 19:18, Tom Lane wrote:
> Andres Freund<andres@anarazel.de>  writes:
>> On 2021-11-15 12:19:51 -0500, Tom Lane wrote:
>>> I was wondering about simply probing to see if python3 exists (and if
>>> so, what version it is exactly), as an additional configure test that
>>> doesn't hook into anything.  That would give us some information without
>>> suddenly changing what is being tested.
>> But this is probably a good compromise. Were you thinking of doing a proper
>> autoconf test or just putting something like python3 --version || true in
>> configure?
> Hm, I had in mind an actual configure test; but since it's just a
> temporary exploratory measure, shortcuts are fine.  However, I'm
> not sure that what you suggest would result in capturing anything
> in config.log.

I'm not sure this is really going to end up moving things forward.

I think we should just write to the build farm owners, we plan to drop 
python2 support in, say, 60 days, please update your setup to use 
python3 or disable python support.

If we add this test first, then all we're going to learn is probably 
that 60% of those who are currently using python2 don't have python3 
installed, and then we're still going to have to send that above email.

  16. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-11-15T19:26:55Z

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> If we add this test first, then all we're going to learn is probably 
> that 60% of those who are currently using python2 don't have python3 
> installed, and then we're still going to have to send that above email.

I don't know what fraction don't have python3 installed, and it doesn't
matter much, since it's unlikely that it's either 0% or 100%.  What
I hoped to learn was, of those who *do* have some python3 installed,
which version it is.  That might inform our thoughts about where to
set the minimum python3 version.

Relevant data points:

* Our docs claim the minimum 3.x version for pl/python itself is 3.1,
but this is unbacked by any testing; the oldest 3.x in the buildfarm
is 3.4.3 (three such animals).

* Meson only promises support back to python 3.6, but if that's
accurate it's going to be a problem for us, because there are lots
of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
AIX 7.2 for starters).  I've been planning to do some testing and
see if meson will run under python 3.4 or 3.5.

But this probably belongs on the meson thread, since that looks like
it will be a much stronger constraint than pl/python is.

			regards, tom lane

  17. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-11-15T20:00:19Z

    Hi,

Continuing the discussion from https://www.postgresql.org/message-id/2146739.1637004415%40sss.pgh.pa.us

On 2021-11-15 14:26:55 -0500, Tom Lane wrote:
> * Meson only promises support back to python 3.6, but if that's
> accurate it's going to be a problem for us, because there are lots
> of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
> AIX 7.2 for starters).  I've been planning to do some testing and
> see if meson will run under python 3.4 or 3.5.

Slightly older versions, which do work to build postgres with the proposed
patchset, run on python3 3.5. I don't think it's likely we could make the
versions that only required 3.4 work reasonably.


Is RHEL7 really an issue? I only have Centos 7 around, but that has python
3.6.

I don't know much about AIX, but according to https://repology.org/project/python/versions
the AIX toolbox has 3.7.11. I don't know enough about AIX to know whether
there's other sources of python3, bison, etc that are common. On the AIX
system I have access to they all seem to be symlinked to /opt/freeware, which
I understand is that toolbox stuff?

Greetings,

Andres Freund

  18. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-11-15T20:30:02Z

    Andres Freund <andres@anarazel.de> writes:
> On 2021-11-15 14:26:55 -0500, Tom Lane wrote:
>> * Meson only promises support back to python 3.6, but if that's
>> accurate it's going to be a problem for us, because there are lots
>> of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
>> AIX 7.2 for starters).  I've been planning to do some testing and
>> see if meson will run under python 3.4 or 3.5.

> Slightly older versions, which do work to build postgres with the proposed
> patchset, run on python3 3.5. I don't think it's likely we could make the
> versions that only required 3.4 work reasonably.

OK, thanks for the datapoint.

> Is RHEL7 really an issue? I only have Centos 7 around, but that has python
> 3.6.

The info I checked said that RHEL7 originally shipped with 3.3.
I'm not sure that Red Hat would've outright replaced that, but they
do have a notion of add-on "software collections", and I'm certain
that they would have provided newer pythons via that mechanism.
So it should only be a big issue for someone who didn't want to install
an add-on collection.  I suppose the overlap of that group with the
people who will want to put PG 15+ on that platform is probably nil.

> I don't know much about AIX, but according to https://repology.org/project/python/versions
> the AIX toolbox has 3.7.11. I don't know enough about AIX to know whether
> there's other sources of python3, bison, etc that are common. On the AIX
> system I have access to they all seem to be symlinked to /opt/freeware, which
> I understand is that toolbox stuff?

Hmm, I was basing that on (a) what I can see installed on gcc119,
which is 3.5.1, and (b) AIX 7.2's 2015 release date, which matches up
pretty well with python 3.5.  As with RHEL, it's entirely likely that
IBM has made newer releases available as add-ons, but there's no
guarantee that any given installation would have that.

Solaris is a bit worse, since they shipped 11.3 a shade earlier,
with python 3.4.3.  No idea about add-on conventions on that
platform.

There's room to argue that all three of these will be out of
primary support before PG15 ships, so maybe we don't need to
worry about whether we can build with their default toolsets.
Still, it's a tradeoff I'd rather not make.

			regards, tom lane

  19. Re: Time to drop plpython2?

    Robert Haas <robertmhaas@gmail.com> — 2021-11-15T20:36:11Z

    On Mon, Nov 15, 2021 at 3:30 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> The info I checked said that RHEL7 originally shipped with 3.3.
> I'm not sure that Red Hat would've outright replaced that, but they
> do have a notion of add-on "software collections", and I'm certain
> that they would have provided newer pythons via that mechanism.
> So it should only be a big issue for someone who didn't want to install
> an add-on collection.  I suppose the overlap of that group with the
> people who will want to put PG 15+ on that platform is probably nil.

It'd only be an issue if they want to compile from source, right?
We're not speaking of changing the runtime prerequisites, IIUC.

I think it's really important that we continue to run on all of the
supported Linux distributions and even some recently-out-of-support
ones if they are popular. My experience is that many people stay on
whatever version of RHEL they've got for quite a long time. I think
we'll be doing ourselves a disservice if such people find that they
can't easily upgrade to newer versions of PostgreSQL without
installing a bunch of new software and/or upgrading software they've
already got on the machine.

On the other hand, the class of users that I'm thinking about does not
typically install anything from source, so they are not directly
impacted by build prerequisites.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  20. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-11-15T21:04:05Z

    Hi,

On 2021-11-15 15:30:02 -0500, Tom Lane wrote:
> > Is RHEL7 really an issue? I only have Centos 7 around, but that has python
> > 3.6.
>
> The info I checked said that RHEL7 originally shipped with 3.3.
> I'm not sure that Red Hat would've outright replaced that, but they
> do have a notion of add-on "software collections", and I'm certain
> that they would have provided newer pythons via that mechanism.
> So it should only be a big issue for someone who didn't want to install
> an add-on collection.  I suppose the overlap of that group with the
> people who will want to put PG 15+ on that platform is probably nil.

3.6 appears to be in the 'update' repository, which I think is separate from
the add-on collection? It appears to be enabled by default in centos7, not
sure how much that means.

Name        : python3
Version     : 3.6.8
Release     : 18.el7
Architecture: x86_64
Size        : 39896
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Group       : Unspecified
URL         : https://www.python.org/
Repository  : updates
Summary     : Interpreter of the Python programming language
Source      : python3-3.6.8-18.el7.src.rpm


> Hmm, I was basing that on (a) what I can see installed on gcc119,
> which is 3.5.1, and (b) AIX 7.2's 2015 release date, which matches up
> pretty well with python 3.5.  As with RHEL, it's entirely likely that
> IBM has made newer releases available as add-ons, but there's no
> guarantee that any given installation would have that.

I think AIX itself only ships with python 2.7, and all python3 versions are
from toolbox. But I don't actually know.


> Solaris is a bit worse, since they shipped 11.3 a shade earlier,
> with python 3.4.3.  No idea about add-on conventions on that
> platform.

Based solely on some person in meson's irc channel, it appears that they added
a newer python3 sometime in 11.4's lifetime:

alanc:
Solaris 11.4 currently ships Python 3.7 & 3.9
as the maintainer of the meson package for Solaris, I deal with the
lack of newer python versions on older Solaris releases by only packaging
meson for Solaris 11.4, and telling anyone who wants it for older releases to
upgrade their OS, or build all the dependencies themselves
...
or it's 11.4.0 from 2018, and not updated with the latest updates (which are
only available for support customers right now unfortunately) - 3.7 was added
in 11.4.12
https://docs.oracle.com/cd/E37838_01/html/E99082/bundledfreeware.html
...


> There's room to argue that all three of these will be out of
> primary support before PG15 ships, so maybe we don't need to
> worry about whether we can build with their default toolsets.
> Still, it's a tradeoff I'd rather not make.

There's two other paths worth mentioning:
- muon is a WIP alternative implementation of meson in plain C99. Doesn't yet
  have enough feature coverage.
- pyinstaller generates an executable from the meson python code, and supports
  solaris and aix

Greetings,

Andres Freund

  21. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2021-11-15T21:07:37Z

    Hi, 

On November 15, 2021 12:36:11 PM PST, Robert Haas <robertmhaas@gmail.com> wrote:
>It'd only be an issue if they want to compile from source, right?
>We're not speaking of changing the runtime prerequisites, IIUC.

Correct.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

  22. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-11-15T21:12:12Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Nov 15, 2021 at 3:30 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The info I checked said that RHEL7 originally shipped with 3.3.

> It'd only be an issue if they want to compile from source, right?
> We're not speaking of changing the runtime prerequisites, IIUC.

I'm not sure.  Does it make sense to document that pl/python has
a different Python version requirement than the build system does?
If we do, who exactly is going to be testing that such a combination
works?  Will it even be possible to compile pl/python against Python
headers/libs of a different Python generation than meson is running
under?

ISTM we'd be a lot better off saying "the minimum Python version is
3.something", full stop, and then making sure that that minimum is
represented in the buildfarm.  But it's not quite clear yet what
"something" needs to be.

> I think it's really important that we continue to run on all of the
> supported Linux distributions and even some recently-out-of-support
> ones if they are popular.

I agree completely, which is why I'm raising the point.

			regards, tom lane

  23. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-11-15T22:45:52Z

    On 15.11.21 20:26, Tom Lane wrote:
> * Our docs claim the minimum 3.x version for pl/python itself is 3.1,
> but this is unbacked by any testing; the oldest 3.x in the buildfarm
> is 3.4.3 (three such animals).

I confirmed locally that 3.2.6 still works with PL/Python.  I expect 
that Python 3.1 also still works, except that I can't build it at the 
moment.  Python 3.0.* was sort of a dud, so it's not really interesting 
to consider.

Partially because of the need to keep supporting Python 2, there hasn't 
been any work done in PL/Python to make use of any newer APIs, so the 
Python 3 minimum version is frozen in time at the moment.

  24. Re: Time to drop plpython2?

    Robert Haas <robertmhaas@gmail.com> — 2021-11-16T15:16:04Z

    On Mon, Nov 15, 2021 at 4:12 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > It'd only be an issue if they want to compile from source, right?
> > We're not speaking of changing the runtime prerequisites, IIUC.
>
> I'm not sure.  Does it make sense to document that pl/python has
> a different Python version requirement than the build system does?
> If we do, who exactly is going to be testing that such a combination
> works?  Will it even be possible to compile pl/python against Python
> headers/libs of a different Python generation than meson is running
> under?

Hmm, that's true. I hadn't considered the fact that anyone who is
packaging PostgreSQL probably also wants to build plpython. However,
it's possible that a side-by-side install of a newer python version
could be used for the build system while building against the system
python for plpython. That might or might not be too exotic a
configuration for someone to consider.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  25. Re: Time to drop plpython2?

    Andrew Dunstan <andrew@dunslane.net> — 2021-11-16T16:26:41Z

    On 11/14/21 21:24, Tom Lane wrote:
> ... btw, there's a fairly critical gating factor for any plan to drop
> python2 support: the buildfarm.  I just counted, and there are exactly
> as many members running python 2.x as 3.x (49 apiece), not counting
> Windows machines that aren't running configure.  We can't commit
> something that's going to make half the buildfarm go red.
>
> (It's likely that some fraction of them do already have python3 installed,
> in which case the search order change Peter recommended would be enough to
> fix it.  But I'm sure not all do.)
>
> This ties into the business about converting the build system to meson,
> as that also requires python 3 --- with, IIUC, a higher minimum version
> than we might otherwise need.  I'm disinclined to cause two separate
> flag days for buildfarm owners, so what I now think is we ought to put
> this idea on the shelf until we've finished that conversion or decided
> we're not gonna do it.  We need to identify exactly what needs to be
> installed before we start pestering the owners.
>
> 			


crake has been using 2.7, but has 3.9.7 installed. I tried switching to
that but ran into an issue with upgrading things from 9.5 on. It would
have been all the way back to 9.2 but the plpython tests drop the
extension even though the transform tests in contrib don't, and neither
do the plperl tests drop plperlu. I'm rather inclined to say we should
stop doing that, or at least be consistent about it.

Having rebuilt all the branches from 9.5 and up the cross version
upgrade tests are now passing on crake.

My other machine with an old python instance is bowerbird. It has python
3.4 installed but not used, alongside 2.7 which is udsed. I will install
the latest and see if that can be made to work.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  26. Re: Time to drop plpython2?

    Andrew Dunstan <andrew@dunslane.net> — 2021-11-18T21:33:13Z

    On 11/16/21 11:26, Andrew Dunstan wrote:
>
> My other machine with an old python instance is bowerbird. It has python
> 3.4 installed but not used, alongside 2.7 which is udsed. I will install
> the latest and see if that can be made to work.
>
>

bowerbird is now building with python 3.10


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  27. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-01-11T12:59:32Z

    On 15.11.21 19:52, Peter Eisentraut wrote:
> I think we should just write to the build farm owners, we plan to drop 
> python2 support in, say, 60 days, please update your setup to use 
> python3 or disable python support.

This discussion stalled.  I think we should do *something* for 
PostgreSQL 15.

I suspect at this point, the Meson move isn't going to happen for 
PostgreSQL 15.  Even if the code were ready, which it is not, then this 
kind of thing would surely be something more sensible to put into PG16 
early rather than PG15 late.  So I'm setting aside any concerns about 
which Python 3.x is the appropriate minimum level.

There is the discussion [0], which involves raising the minimum Python 
version from 2.6 to 2.7, which would affect some build farm members. 
But if we're going to drop 2.x anyway for PG15, say, then we don't need 
to spend time organizing a 2.6 to 2.7 transition.  (Although 
backpatching might be required.)

Also, I was recently doing some bug fixing work in PL/Python, and just 
getting a Python 2 installation is not straightforward these days on the 
sort of OS that might be on a development machine.  So there is also 
that driver for dropping Python 2.

I think what I wrote in the message I'm responding to is the best way 
forward, although maybe with less than 60 days now.

Thoughts?


[0]: 
https://www.postgresql.org/message-id/flat/c74add3c-09c4-a9dd-1a03-a846e5b2fc52%40enterprisedb.com

  28. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-01-11T16:06:25Z

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 15.11.21 19:52, Peter Eisentraut wrote:
>> I think we should just write to the build farm owners, we plan to drop 
>> python2 support in, say, 60 days, please update your setup to use 
>> python3 or disable python support.

> This discussion stalled.  I think we should do *something* for 
> PostgreSQL 15.

Agreed.  We want plpython2 out of there for v15, or we'll be adding
another year to the length of time we're on the hook to support
python2 in the back branches --- which, as you say, is getting
ever more painful to do.

> I suspect at this point, the Meson move isn't going to happen for 
> PostgreSQL 15.

Also agreed.  Nonetheless, we need to make a recommendation to the
buildfarm owners about what's the minimum python3 version we intend
to support going forward.  Do we want to just set it at 3.6, with
the expectation that the meson move will happen before too long?

(I'm not going to accept being fuzzy on this point, because I need
to know what to install on prairiedog and gaur.  I intend to install
whatever our minimum supported version is, not only to keep us honest
on that being actually supported, but because I anticipate that
newer python versions will be correspondingly harder to build on
ancient platforms.)

			regards, tom lane

  29. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-01-12T07:39:03Z

    On 11.01.22 17:06, Tom Lane wrote:
> Nonetheless, we need to make a recommendation to the
> buildfarm owners about what's the minimum python3 version we intend
> to support going forward.  Do we want to just set it at 3.6, with
> the expectation that the meson move will happen before too long?

Well, the minimum supported version has always been the oldest version 
that actually works.  I don't think we ever said, we support >= X, even 
though < X still actually works, about any dependency.

I don't care much to tie this to Meson right now.  Meson might well move 
to 3.8 next week and ruin this whole scheme.

I'm okay with issuing some sort of recommendation for what is reasonable 
to test, and 3.5 or 3.6 seems like a good cutoff, considering what LTS 
OS currently ship.  But I'm not sure if that is the same as "support".

  30. Re: Time to drop plpython2?

    Robert Haas <robertmhaas@gmail.com> — 2022-01-12T18:42:25Z

    On Wed, Jan 12, 2022 at 2:39 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
> Well, the minimum supported version has always been the oldest version
> that actually works.  I don't think we ever said, we support >= X, even
> though < X still actually works, about any dependency.

I think that we sometimes say that versions < X are unsupported if we
are unable to test whether or not they work. In other words, I think
the relevant question is whether we are able to demonstrate that it
works, not whether it actually does work.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  31. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-01-12T18:49:15Z

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 11.01.22 17:06, Tom Lane wrote:
>> Nonetheless, we need to make a recommendation to the
>> buildfarm owners about what's the minimum python3 version we intend
>> to support going forward.

> Well, the minimum supported version has always been the oldest version 
> that actually works.  I don't think we ever said, we support >= X, even 
> though < X still actually works, about any dependency.

The concern I have is how do we know what "actually works", if we're
not testing it?  installation.sgml currently promises python2 >= 2.6,
and we know that that works because we have 2.6 in the buildfarm.
It also promises python3 >= 3.1, but we have no buildfarm animals
testing anything older than 3.4.3, so I don't think that promise
is worth the electrons it's written on.  Furthermore, if the meson
conversion forces people to update their python3 to something newer,
there will probably be no testing of plpython against anything older
than what meson requires.

> I don't care much to tie this to Meson right now.  Meson might well move 
> to 3.8 next week and ruin this whole scheme.

Wouldn't be a problem unless our build scripts require that newer
version of meson.  Andres mentioned earlier that we should be able
to run with some older meson versions that only require python 3.5
or so, so I'm hoping we can end up with directives like "use meson
X or later and python 3.5 or later".

> I'm okay with issuing some sort of recommendation for what is reasonable 
> to test, and 3.5 or 3.6 seems like a good cutoff, considering what LTS 
> OS currently ship.  But I'm not sure if that is the same as "support".

Well, I'll see about putting 3.5 on my dinosaurs, and hope I don't
have to do it over.

Anyway, getting back to the point: I think we should notify the
owners ASAP and set a 30-day deadline.  We should try to get this
done before the March CF starts, so it's too late for a 60-day
grace period.  In any case, the worst-case scenario for an owner
is to disable --with-python until they have time to do an upgrade,
so it doesn't seem like a month is a big problem.

			regards, tom lane

  32. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-01-13T15:02:14Z

    On 12.01.22 19:49, Tom Lane wrote:
> Anyway, getting back to the point: I think we should notify the
> owners ASAP and set a 30-day deadline.  We should try to get this
> done before the March CF starts, so it's too late for a 60-day
> grace period.  In any case, the worst-case scenario for an owner
> is to disable --with-python until they have time to do an upgrade,
> so it doesn't seem like a month is a big problem.

Sure, let's do that.  I don't have a buildfarm animal these days, so I'm 
not on that list, so it would be great if you could do that

  33. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-01-13T17:00:27Z

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 12.01.22 19:49, Tom Lane wrote:
>> Anyway, getting back to the point: I think we should notify the
>> owners ASAP and set a 30-day deadline.

> Sure, let's do that.  I don't have a buildfarm animal these days, so I'm 
> not on that list, so it would be great if you could do that

Done.  I told them "mid February", so we can plan on say the 15th
as the target date for pushing a patch.

I realized BTW that the meson business is not relevant for prairiedog
or gaur.  Those animals will die regardless of python version because
they can't build ninja (for lack of <spawn.h>).  So I think maybe
I'll install python 3.1 and see if that compatibility claim is really
true ;-)

			regards, tom lane

  34. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-01-13T17:47:40Z

    Hi,


On 2022-01-11 13:59:32 +0100, Peter Eisentraut wrote:
> I suspect at this point, the Meson move isn't going to happen for PostgreSQL
> 15.

Yes - I IIRC even noted early in the thread that I don't think it's realistic
to finish it in time for 15. There's just a good amount of portability hacking
left to be done, and a good chunk of cleanup. I have a new colleague working
on automating testing it on more platforms.

Getting a few of the prerequisite patches in would make it more likely to
succeed in 16 though.


> Even if the code were ready, which it is not, then this kind of thing
> would surely be something more sensible to put into PG16 early rather than
> PG15 late.

Right. There's also a lot of discussion around transition paths etc that need
to be happening first. A lot of other projects that moved ran with multiple
buildsystems for ~1 release, so things could mopped up without blocking
everyone.


> I think what I wrote in the message I'm responding to is the best way
> forward, although maybe with less than 60 days now.

Already happened, but I'm a belated +1 ;)

Greetings,

Andres Freund

  35. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-14T19:18:58Z

    I wrote:
> Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
>> On 12.01.22 19:49, Tom Lane wrote:
> Anyway, getting back to the point: I think we should notify the
> owners ASAP and set a 30-day deadline.

>> Sure, let's do that.  I don't have a buildfarm animal these days, so I'm 
>> not on that list, so it would be great if you could do that

> Done.  I told them "mid February", so we can plan on say the 15th
> as the target date for pushing a patch.

Well, it's mid-February.  Do we have a python2-removal patch
that's ready to go?

			regards, tom lane

  36. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-14T20:42:40Z

    Hi,

On 2022-02-14 14:18:58 -0500, Tom Lane wrote:
> Well, it's mid-February.  Do we have a python2-removal patch
> that's ready to go?

I can refresh mine. Iit might be good to first reapply
f201da39edc - "Make configure prefer python3 to plain python."
for a few days?

Greetings,

Andres Freund

  37. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-14T20:48:12Z

    Andres Freund <andres@anarazel.de> writes:
> On 2022-02-14 14:18:58 -0500, Tom Lane wrote:
>> Well, it's mid-February.  Do we have a python2-removal patch
>> that's ready to go?

> I can refresh mine. Iit might be good to first reapply
> f201da39edc - "Make configure prefer python3 to plain python."
> for a few days?

We could I guess, but does it really buy anything?  I'm sure that
some of the buildfarm still hasn't updated their Python installation,
but it'll be about the same failure we'd get from the final patch.

			regards, tom lane

  38. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-15T03:00:20Z

    Hi, 

On February 14, 2022 12:48:12 PM PST, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>Andres Freund <andres@anarazel.de> writes:
>> On 2022-02-14 14:18:58 -0500, Tom Lane wrote:
>>> Well, it's mid-February.  Do we have a python2-removal patch
>>> that's ready to go?
>
>> I can refresh mine. Iit might be good to first reapply
>> f201da39edc - "Make configure prefer python3 to plain python."
>> for a few days?
>
>We could I guess, but does it really buy anything?  I'm sure that
>some of the buildfarm still hasn't updated their Python installation,
>but it'll be about the same failure we'd get from the final patch.

I guess what I was actually wondering about - but didn't write - was whether it's worth rejecting python 2 with just  configure / msvc perl changes initially.

The proper removal of python 2 support includes a bunch of buildsystem and code changes. Seems like it could be useful to have a snapshot of the buildfarm state after rejecting python 2, separate from the more verbose changes.

Andres


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

  39. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-15T03:57:37Z

    Andres Freund <andres@anarazel.de> writes:
> On February 14, 2022 12:48:12 PM PST, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> We could I guess, but does it really buy anything?  I'm sure that
>> some of the buildfarm still hasn't updated their Python installation,
>> but it'll be about the same failure we'd get from the final patch.

> I guess what I was actually wondering about - but didn't write - was whether it's worth rejecting python 2 with just  configure / msvc perl changes initially.

Ah, I see.  +1, that would let us start nagging the laggard buildfarm
owners right away.  But I do think we want the rest of the cleanup
done pretty soon --- especially simplification of the plpython
regression test arrangements.

			regards, tom lane

  40. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-15T06:01:31Z

    Hi,

On 2022-02-14 22:57:37 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On February 14, 2022 12:48:12 PM PST, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> We could I guess, but does it really buy anything?  I'm sure that
> >> some of the buildfarm still hasn't updated their Python installation,
> >> but it'll be about the same failure we'd get from the final patch.
> 
> > I guess what I was actually wondering about - but didn't write - was whether it's worth rejecting python 2 with just  configure / msvc perl changes initially.
> 
> Ah, I see.  +1, that would let us start nagging the laggard buildfarm
> owners right away.

Cool.


> But I do think we want the rest of the cleanup done pretty soon ---
> especially simplification of the plpython regression test arrangements.

That part I have mostly ready (worked as part of a rebase of the meson tree
sometime last weekend). It's trawling through the docs, the msvc build scripts
and some plpython C code that's a bit more work...

Attached the state as I have in the meson tree. Will split out the configure
test tomorrow. Might or might get through the msvc scripts and docs as well.

Greetings,

Andres Freund

  41. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-15T21:40:16Z

    Hi,

On 2022-02-14 22:01:31 -0800, Andres Freund wrote:
> Attached the state as I have in the meson tree. Will split out the configure
> test tomorrow. Might or might get through the msvc scripts and docs as well.

Attached are four patches:
1: plpython: Reject Python 2 during build configuration.
2: plpython: Remove plpythonu, plpython2u extensions.
3: plpython: Remove regression test infrastructure for Python 2.
4: WIP: plpython: Code cleanup related to removal of Python 2 support.

I think we could apply 1) now? I've adjusted installation.sgml, but there's
plenty other references to python 2 left.

The configure already only checked for major version 3, but we document
requiring python 3.2. Afaics there was no version check at all in the msvc
build before this - I've now added one, but also just for 3, for now.

I think 2) is also ok, but I'd rather commit that a buildfarm cycle after 1).

I tested the msvc portion of 3) via CI. There's still some python specific code
in vcregress.pl:plcheck(), but I think that's OK for now. We could change the
python 3 regression tests to not create the extension themselves to get rid of
that, but that doesn't really seem like an advantage?

4) needs a bit more work as noted by two FIXMEs and review comments in
https://postgr.es/m/3fc1211d-960b-4b2f-3e96-a6099db847fc%40enterprisedb.com

There's a good chunk of docs work left.

Greetings,

Andres Freund

  42. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-02-16T14:43:19Z

    On 15.02.22 22:40, Andres Freund wrote:
> 1: plpython: Reject Python 2 during build configuration.

There is a bit of redundancy in the new wording in installation.sgml:

"The minimum required version is <productname>Python</productname> 3.2 
or later."

There is also a small bit in install-windows.sgml that would be worth 
updating:

<programlisting>
$config->{python} = 'c:\python26';
</programlisting>

Other than that, this looks fine to me, and we should move forward with 
this now.

> 2: plpython: Remove plpythonu, plpython2u extensions.
> 3: plpython: Remove regression test infrastructure for Python 2.

The removal of the Python 2 contrib extension variants ended up in patch 
3, which seems incorrect.

  43. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-16T19:58:50Z

    Hi,

On 2022-02-16 15:43:19 +0100, Peter Eisentraut wrote:
> On 15.02.22 22:40, Andres Freund wrote:
> > 1: plpython: Reject Python 2 during build configuration.
> 
> There is a bit of redundancy in the new wording in installation.sgml:
> 
> "The minimum required version is <productname>Python</productname> 3.2 or
> later."

I stared for a bit, and I just don't see the redundancy?


> There is also a small bit in install-windows.sgml that would be worth
> updating:
> 
> <programlisting>
> $config->{python} = 'c:\python26';
> </programlisting>

Yea. I saw that but ended up with not updating it, because it already was out
of date ;). But I guess I'll just put 310 there or something.


> Other than that, this looks fine to me, and we should move forward with this
> now.

Cool, will apply 1) later today.


> > 2: plpython: Remove plpythonu, plpython2u extensions.
> > 3: plpython: Remove regression test infrastructure for Python 2.
> 
> The removal of the Python 2 contrib extension variants ended up in patch 3,
> which seems incorrect.

Uh, huh. That was a mistake, not sure what happened...

Greetings,

Andres Freund

  44. Re: Time to drop plpython2?

    Chapman Flack <chap@anastigmatix.net> — 2022-02-16T20:05:36Z

    On 02/16/22 14:58, Andres Freund wrote:
>> "The minimum required version is <productname>Python</productname> 3.2 or
>> later."
> 
> I stared for a bit, and I just don't see the redundancy?

"minimum ... or later" maybe?

Regards,
-Chap

  45. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-16T20:31:16Z

    On 2022-02-16 15:05:36 -0500, Chapman Flack wrote:
> On 02/16/22 14:58, Andres Freund wrote:
> >> "The minimum required version is <productname>Python</productname> 3.2 or
> >> later."
> > 
> > I stared for a bit, and I just don't see the redundancy?
> 
> "minimum ... or later" maybe?

Ah. Thanks.

  46. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-17T06:52:08Z

    On 2022-02-16 11:58:50 -0800, Andres Freund wrote:
> Cool, will apply 1) later today.

Done. Curious how red the BF will turn out to be. Let's hope it's not
too bad.

  47. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-17T07:14:46Z

    Hi,

On 2022-02-16 22:52:08 -0800, Andres Freund wrote:
> On 2022-02-16 11:58:50 -0800, Andres Freund wrote:
> > Cool, will apply 1) later today.
> 
> Done. Curious how red the BF will turn out to be. Let's hope it's not
> too bad.

I've pinged the owners of the animals failing so far:
- snakefly, massasauga
- jay, trilobite, hippopotamus, avocet
- myna, butterflyfish
- rhinoceros

Greetings,

Andres Freund

  48. Re: Time to drop plpython2?

    Michael Paquier <michael@paquier.xyz> — 2022-02-17T07:52:49Z

    On Wed, Feb 16, 2022 at 11:14:46PM -0800, Andres Freund wrote:
> I've pinged the owners of the animals failing so far:
> - myna, butterflyfish

These two are managed by a colleague, and I have an access to them.
They should get back to green quickly now, if I did not mess up..
--
Michael

  49. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-17T18:08:55Z

    Hi,

On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
> >
> > Done. Curious how red the BF will turn out to be. Let's hope it's not
> > too bad.
>
> I've pinged the owners of the animals failing so far:

Now also pinged:
- curculio
- guaibasaurus
- snapper
- gadwall, takin


> - snakefly, massasauga

Fixed.


> - jay, trilobite, hippopotamus, avocet

Nothing yet.


> - myna, butterflyfish

Fixed, as noted by Micheal on this thread.


> - rhinoceros

Joe replied that he is afk, looking into it tomorrow.


Greetings,

Andres Freund

  50. Re: Time to drop plpython2?

    Mikael Kjellström <mikael.kjellstrom@mksoft.nu> — 2022-02-18T07:33:38Z

    
On 2022-02-17 19:08, Andres Freund wrote:
>> I've pinged the owners of the animals failing so far:
> 
> Now also pinged:
> - curculio

Should be fixed by now.

I did install the python3-package but the binary was called:

/usr/local/bin/python3.5

for some reason so configure didn't pick it up.

Fixed it by creating a symlink to:

/usr/local/bin/python3

/Mikael

  51. Re: Time to drop plpython2?

    Joe Conway <mail@joeconway.com> — 2022-02-18T19:19:49Z

    On 2/17/22 13:08, Andres Freund wrote:
> On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
>> > Done. Curious how red the BF will turn out to be. Let's hope it's not
>> > too bad.

>> - rhinoceros
> 
> Joe replied that he is afk, looking into it tomorrow.

I installed python3 packages (initially forgetting the devel package -- 
d'oh!) and changed build-farm.conf thusly:

8<-------------------
***************
*** 185,190 ****
--- 185,193 ----

         build_env => {

+               # specify python 3
+               PYTHON => 'python3',
+
                 # use a dedicated cache for the build farm. this should 
give us
                 # very high hit rates and slightly faster cache searching.
                 #
8<-------------------

That seems to have worked.

But the question is, is that the correct/recommended method?

Joe
-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

  52. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-18T19:37:16Z

    Hi,

On 2022-02-18 14:19:49 -0500, Joe Conway wrote:
> On 2/17/22 13:08, Andres Freund wrote:
> > On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
> > > > Done. Curious how red the BF will turn out to be. Let's hope it's not
> > > > too bad.
> 
> > > - rhinoceros
> > 
> > Joe replied that he is afk, looking into it tomorrow.
> 
> I installed python3 packages (initially forgetting the devel package --
> d'oh!) and changed build-farm.conf thusly:
> 
> 8<-------------------
> ***************
> *** 185,190 ****
> --- 185,193 ----
> 
>         build_env => {
> 
> +               # specify python 3
> +               PYTHON => 'python3',
> +
>                 # use a dedicated cache for the build farm. this should give
> us
>                 # very high hit rates and slightly faster cache searching.
>                 #
> 8<-------------------
> 
> That seems to have worked.
> 
> But the question is, is that the correct/recommended method?

If python3 is in PATH, then you shouldn't need that part.

Greetings,

Andres Freund

  53. Re: Time to drop plpython2?

    Joe Conway <mail@joeconway.com> — 2022-02-18T19:46:39Z

    On 2/18/22 14:37, Andres Freund wrote:
> Hi,
> 
> On 2022-02-18 14:19:49 -0500, Joe Conway wrote:
>> On 2/17/22 13:08, Andres Freund wrote:
>> > On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
>> > > > Done. Curious how red the BF will turn out to be. Let's hope it's not
>> > > > too bad.
>> 
>> > > - rhinoceros
>> > 
>> > Joe replied that he is afk, looking into it tomorrow.
>> 
>> I installed python3 packages (initially forgetting the devel package --
>> d'oh!) and changed build-farm.conf thusly:
>> 
>> 8<-------------------
>> ***************
>> *** 185,190 ****
>> --- 185,193 ----
>> 
>>         build_env => {
>> 
>> +               # specify python 3
>> +               PYTHON => 'python3',
>> +
>>                 # use a dedicated cache for the build farm. this should give
>> us
>>                 # very high hit rates and slightly faster cache searching.
>>                 #
>> 8<-------------------
>> 
>> That seems to have worked.
>> 
>> But the question is, is that the correct/recommended method?
> 
> If python3 is in PATH, then you shouldn't need that part.

Not quite -- python3 is definitely in the PATH:

8<-------------------
$ which python3
/usr/bin/python3
8<-------------------

And I gather that merely installing python3 RPMs on a RHEL-esque 7.X 
system does not replace the python symlink:

8<-------------------
$ yum whatprovides /usr/bin/python

python-2.7.5-89.el7.x86_64 : An interpreted, interactive, 
object-oriented programming language
Repo        : base
Matched from:
Filename    : /usr/bin/python

$ ll /usr/bin/python
lrwxrwxrwx. 1 root root 7 Mar 13  2021 /usr/bin/python -> python2
8<-------------------


Joe
-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

  54. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-18T20:25:33Z

    Hi,

On 2022-02-18 14:46:39 -0500, Joe Conway wrote:
> On 2/18/22 14:37, Andres Freund wrote:
> > > That seems to have worked.
> > > 
> > > But the question is, is that the correct/recommended method?
> > 
> > If python3 is in PATH, then you shouldn't need that part.
> 
> Not quite -- python3 is definitely in the PATH:
> 
> 8<-------------------
> $ which python3
> /usr/bin/python3
> 8<-------------------
> 
> And I gather that merely installing python3 RPMs on a RHEL-esque 7.X system
> does not replace the python symlink:
> 
> 8<-------------------
> $ yum whatprovides /usr/bin/python
> 
> python-2.7.5-89.el7.x86_64 : An interpreted, interactive, object-oriented
> programming language
> Repo        : base
> Matched from:
> Filename    : /usr/bin/python
> 
> $ ll /usr/bin/python
> lrwxrwxrwx. 1 root root 7 Mar 13  2021 /usr/bin/python -> python2
> 8<-------------------

Yea, that all looks fine. What's the problem if you don't specify the
PYTHON=python3? We try python3, python in that order by default, so it should
pick up the same python3 you specify explicitly?

Or maybe I'm just misunderstanding what you're asking...

Greetings,

Andres Freund

  55. Re: Time to drop plpython2?

    Joe Conway <mail@joeconway.com> — 2022-02-18T20:35:37Z

    On 2/18/22 15:25, Andres Freund wrote:
> On 2022-02-18 14:46:39 -0500, Joe Conway wrote:
>> $ ll /usr/bin/python
>> lrwxrwxrwx. 1 root root 7 Mar 13  2021 /usr/bin/python -> python2
>> 8<-------------------
> 
> Yea, that all looks fine. What's the problem if you don't specify the
> PYTHON=python3? We try python3, python in that order by default, so it should
> pick up the same python3 you specify explicitly?

Initially I just installed the python3 RPMs and when I tried running 
manually it was still error'ing on configure due to finding python2.

Even after adding EXPORT PYTHON=python3 to my ~/.bash_profile I was 
seeing the same.

By adding PYTHON => 'python3' to build-farm.conf I saw that the error 
changed to indicate missing python3-devel package. Once I installed 
that, everything went green.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

  56. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-18T20:53:38Z

    Hi,

On 2022-02-18 15:35:37 -0500, Joe Conway wrote:
> Initially I just installed the python3 RPMs and when I tried running
> manually it was still error'ing on configure due to finding python2.

> Even after adding EXPORT PYTHON=python3 to my ~/.bash_profile I was seeing
> the same.
>
> By adding PYTHON => 'python3' to build-farm.conf I saw that the error
> changed to indicate missing python3-devel package. Once I installed that,
> everything went green.

Hm. It definitely did test python3, earlier today:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2022-02-18%2016%3A52%3A13

checking for python3... no
checking for python... /usr/bin/python


the next run then saw:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2022-02-18%2017%3A50%3A09
checking for PYTHON... python3
configure: using python 3.6.8 (default, Nov 16 2020, 16:55:22)
checking for Python sysconfig module... yes
checking Python configuration directory... /usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
checking Python include directory... -I/usr/include/python3.6m

but then failed because the python headers weren't available:
checking for Python.h... no
configure: error: header file <Python.h> is required for Python


Note that this did *not* yet use PYTHON => 'python3' in build_env, but has it
in the environment starting the buildfarm client.


the next run succeeded, with 'PYTHON' => 'python3' in build env. But
presumably this just was because you installed the python3-devel package?

Greetings,

Andres Freund

  57. Re: Time to drop plpython2?

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-02-18T21:03:34Z

    On 2022-Feb-17, Andres Freund wrote:

> Now also pinged:
> - guaibasaurus

Fixed now (apt install python3-dev), but I had initially added
PYTHON=>python3 to the .conf, unsuccessfully because I failed to install
the dev pkg.  After the first success I removed that line.  It should
still work if we do test python3 first, but if it does fail, then I'll
put that line back.

Thanks

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"The problem with the future is that it keeps turning into the present"
(Hobbes)

  58. Re: Time to drop plpython2?

    Joe Conway <mail@joeconway.com> — 2022-02-18T21:08:33Z

    On 2/18/22 15:53, Andres Freund wrote:
> the next run succeeded, with 'PYTHON' => 'python3' in build env. But
> presumably this just was because you installed the python3-devel package?


Ok, I guess I got confused when it failed due to the missing devel 
package, because I removed the PYTHON => 'python3' from the build env 
and it is still getting successfully past the configure stage. Sorry for 
the noise.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

  59. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-18T22:41:04Z

    Hi,

Thanks to some more buildfarm animal updates things are looking better. I
think there's now only three owners that haven't updated their animals
successfully. One of which I hadn't yet pinged (chipmunk / Heikki), done now.

There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
menagerie. Mark said yesterday that he's working on updating.


There's one further failure, but the symptoms are quite different. I've also
pinged its owner. I think it's a problem on the system, rather than our side,
but less certain than with the other cases:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=haddock&dt=2022-02-17%2023%3A36%3A09

checking Python.h usability... no
checking Python.h presence... yes
configure: WARNING: Python.h: present but cannot be compiled
configure: WARNING: Python.h:     check for missing prerequisite headers?
configure: WARNING: Python.h: see the Autoconf documentation
configure: WARNING: Python.h:     section "Present But Cannot Be Compiled"
configure: WARNING: Python.h: proceeding with the compiler's result
configure: WARNING:     ## ---------------------------------------------- ##
configure: WARNING:     ## Report this to pgsql-bugs@lists.postgresql.org ##
configure: WARNING:     ## ---------------------------------------------- ##
checking for Python.h... no
configure: error: header file <Python.h> is required for Python

configure:19158: ccache gcc -c -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -m32 -I/usr/include/python3.9 -DWAIT_USE_POLL -D_POSIX_PTHREAD_SEMANTICS -I/usr/include/libxml2  conftest.c >&5
In file included from /usr/include/python3.9/Python.h:8,
                 from conftest.c:235:
/usr/include/python3.9/pyconfig.h:1443: warning: "SIZEOF_LONG" redefined
 1443 | #define SIZEOF_LONG 8
      |
conftest.c:183: note: this is the location of the previous definition
  183 | #define SIZEOF_LONG 4
      |
In file included from /usr/include/python3.9/Python.h:8,
                 from conftest.c:235:
/usr/include/python3.9/pyconfig.h:1467: warning: "SIZEOF_SIZE_T" redefined
 1467 | #define SIZEOF_SIZE_T 8
      |
conftest.c:182: note: this is the location of the previous definition
  182 | #define SIZEOF_SIZE_T 4
      |
In file included from /usr/include/python3.9/Python.h:8,
                 from conftest.c:235:
/usr/include/python3.9/pyconfig.h:1476: warning: "SIZEOF_VOID_P" redefined
 1476 | #define SIZEOF_VOID_P 8
      |
conftest.c:181: note: this is the location of the previous definition
  181 | #define SIZEOF_VOID_P 4
      |
In file included from /usr/include/python3.9/Python.h:63,
                 from conftest.c:235:
/usr/include/python3.9/pyport.h:736:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  736 | #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
      |  ^~~~~


This is a 64bit host, targetting 32bit "CFLAGS' => '-m32'. However it linked
successfully against python 2.

Greetings,

Andres Freund

  60. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-18T23:09:19Z

    Andres Freund <andres@anarazel.de> writes:
> There's one further failure, but the symptoms are quite different. I've also
> pinged its owner. I think it's a problem on the system, rather than our side,
> but less certain than with the other cases:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=haddock&dt=2022-02-17%2023%3A36%3A09

This one was discussed on the buildfarm-owners list last month.
There's no 32-bit python3 on that box, and apparently no plans
to install one --- it sounded like the box is due for retirement
anyway.

			regards, tom lane

  61. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-18T23:19:40Z

    Hi,

On 2022-02-18 18:09:19 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > There's one further failure, but the symptoms are quite different. I've also
> > pinged its owner. I think it's a problem on the system, rather than our side,
> > but less certain than with the other cases:
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=haddock&dt=2022-02-17%2023%3A36%3A09
> 
> This one was discussed on the buildfarm-owners list last month.
> There's no 32-bit python3 on that box, and apparently no plans
> to install one --- it sounded like the box is due for retirement
> anyway.

Any chance that was a private response? I just looked in the buildfarm-members
list (I assume you meant that?), and didn't see anything:

https://www.postgresql.org/message-id/flat/3162195.1642093011%40sss.pgh.pa.us

Greetings,

Andres Freund

  62. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-18T23:27:11Z

    Andres Freund <andres@anarazel.de> writes:
> On 2022-02-18 18:09:19 -0500, Tom Lane wrote:
>> This one was discussed on the buildfarm-owners list last month.
>> There's no 32-bit python3 on that box, and apparently no plans
>> to install one --- it sounded like the box is due for retirement
>> anyway.

> Any chance that was a private response? I just looked in the buildfarm-members
> list (I assume you meant that?), and didn't see anything:
> https://www.postgresql.org/message-id/flat/3162195.1642093011%40sss.pgh.pa.us

Oh ... hm, yeah, looking at my mail logs, that came in as private mail and
there's no corresponding buildfarm-members traffic.  I did not keep the
message unfortunately, but the owner indicated that he wasn't planning to
bother updating python.  Which is fine, but maybe we should press him to
remove --with-python instead of disabling the box altogether --- we don't
have a lot of Solaris clones in the farm.

			regards, tom lane

  63. Re: Time to drop plpython2?

    Mark Wong <markwkm@postgresql.org> — 2022-02-19T02:00:28Z

    Hi everyone,

On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> menagerie. Mark said yesterday that he's working on updating.

I've made one pass.  Hopefully I didn't make any mistakes. :)

Regards,
Mark

  64. Re: Time to drop plpython2?

    Michael Paquier <michael@paquier.xyz> — 2022-02-19T06:23:42Z

    On Thu, Feb 17, 2022 at 10:08:55AM -0800, Andres Freund wrote:
> On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
>> I've pinged the owners of the animals failing so far:
>> - myna, butterflyfish
> 
> Fixed, as noted by Micheal on this thread.

Fixed is an incorrect word here, "temporarily bypassed" fits better :)

Unfortunately, I had to remove --with-python on both animals for the
time being, as I was not able to figure out why Python.h could not be
found in those installations, and it was Friday afternoon.  I'll try
to investigate and re-enable that some time next week.
--
Michael

  65. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-19T16:22:29Z

    Hi,

On 2022-02-19 02:00:28 +0000, Mark Wong wrote:
> On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> > There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> > menagerie. Mark said yesterday that he's working on updating.
> 
> I've made one pass.  Hopefully I didn't make any mistakes. :)

Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
animals that ran since still seem to be failing in the same spot...

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gadwall&dt=2022-02-19%2011%3A22%3A48

checking Python.h usability... no
checking Python.h presence... no
checking for Python.h... no
configure: error: header file <Python.h> is required for Python


For that machine (and the other debian based ones) the relevant package likely
is python3-dev.

For the Red Hat and Suse ones, it's likely python3-devel.


I've wondered before if it's worth maintaining a list of packages for
dependencies for at least the more popular distros. It's annoying to have to
figure it out everytime one needs to test something.


FWIW, here's the recipe I just used to verify the packages necessary for
Python.h to be found:

$ podman run --rm -it opensuse/leap
# zypper install -y python3
# ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
<file not found>
# zypper install -y python3-devel
# ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
-rw-r--r-- 1 root root 3221 Jan  4 14:04 /usr/include/python3.6m/Python.h

(Wow, zypper repos are expensive to refresh. And I thought dnf was slow doing
so, compared to apt.)

Greetings,

Andres Freund

  66. Re: Time to drop plpython2?

    Mark Wong <markwkm@postgresql.org> — 2022-02-21T17:49:32Z

    On Sat, Feb 19, 2022 at 08:22:29AM -0800, Andres Freund wrote:
> Hi,
> 
> On 2022-02-19 02:00:28 +0000, Mark Wong wrote:
> > On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> > > There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> > > menagerie. Mark said yesterday that he's working on updating.
> > 
> > I've made one pass.  Hopefully I didn't make any mistakes. :)
> 
> Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
> animals that ran since still seem to be failing in the same spot...
> 
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gadwall&dt=2022-02-19%2011%3A22%3A48
> 
> checking Python.h usability... no
> checking Python.h presence... no
> checking for Python.h... no
> configure: error: header file <Python.h> is required for Python
> 
> 
> For that machine (and the other debian based ones) the relevant package likely
> is python3-dev.
> 
> For the Red Hat and Suse ones, it's likely python3-devel.
> 
> 
> I've wondered before if it's worth maintaining a list of packages for
> dependencies for at least the more popular distros. It's annoying to have to
> figure it out everytime one needs to test something.
> 
> 
> FWIW, here's the recipe I just used to verify the packages necessary for
> Python.h to be found:
> 
> $ podman run --rm -it opensuse/leap
> # zypper install -y python3
> # ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
> <file not found>
> # zypper install -y python3-devel
> # ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
> -rw-r--r-- 1 root root 3221 Jan  4 14:04 /usr/include/python3.6m/Python.h
> 
> (Wow, zypper repos are expensive to refresh. And I thought dnf was slow doing
> so, compared to apt.)

Oops, made another pass for python3 dev libraries.

I can't seem to find archived ppc repos OpenSUSE Leap 43.2.  I'm
debating whether to disable python or upgrade/rebrand that animal for a
newer SUSE release.  I've stopped my cron jobs on this animal for the
time being.

Regards,
Mark

  67. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-02-21T18:05:37Z

    Hi,

On 2022-02-21 09:49:32 -0800, Mark Wong wrote:
> Oops, made another pass for python3 dev libraries.

Thanks!


> I can't seem to find archived ppc repos OpenSUSE Leap 43.2.  I'm
> debating whether to disable python or upgrade/rebrand that animal for a
> newer SUSE release.  I've stopped my cron jobs on this animal for the
> time being.

I assume you mean leap 42.3, that seemed to be the newest 4x.x? If so that's
been archived 2019-07-01 [1].

Leap's versioning is, uh, confusing. 13 -> 42 -> 15. Yea.

I don't think it's really useful to run out-of-support distribution
versions. Leaving security etc aside, not being able to install packages seems
sufficient reason to upgrade.

Greetings,

Andres Freund

[1] https://lists.opensuse.org/archives/list/security-announce@lists.opensuse.org/message/6CIQPV3H6J4AIIXUKUGI4IESMTHIFFFB/

  68. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-21T20:28:35Z

    Mark Wong <markwkm@postgresql.org> writes:
> On Sat, Feb 19, 2022 at 08:22:29AM -0800, Andres Freund wrote:
>> Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
>> animals that ran since still seem to be failing in the same spot...

> Oops, made another pass for python3 dev libraries.

You might need to do one more thing, which is manually blow away the cache
files under $BUILDFARM/accache.  For example, on demoiselle everything
looks fine in HEAD, but the back branches are failing like this:

checking for python... (cached) /usr/bin/python
./configure: line 10334: /usr/bin/python: No such file or directory
configure: using python 
./configure: line 10342: test: : integer expression expected
checking for Python sysconfig module... no
configure: error: sysconfig module not found

Very recent versions of the buildfarm script will discard accache
automatically after a configure or make failure, but I think the
REL_11 you're running here doesn't have that defense.  It'll only
flush accache after a change in the configure script in git.

			regards, tom lane

  69. Re: Time to drop plpython2?

    Mark Wong <markwkm@postgresql.org> — 2022-02-22T21:12:02Z

    On Mon, Feb 21, 2022 at 03:28:35PM -0500, Tom Lane wrote:
> Mark Wong <markwkm@postgresql.org> writes:
> > On Sat, Feb 19, 2022 at 08:22:29AM -0800, Andres Freund wrote:
> >> Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
> >> animals that ran since still seem to be failing in the same spot...
> 
> > Oops, made another pass for python3 dev libraries.
> 
> You might need to do one more thing, which is manually blow away the cache
> files under $BUILDFARM/accache.  For example, on demoiselle everything
> looks fine in HEAD, but the back branches are failing like this:
> 
> checking for python... (cached) /usr/bin/python
> ./configure: line 10334: /usr/bin/python: No such file or directory
> configure: using python 
> ./configure: line 10342: test: : integer expression expected
> checking for Python sysconfig module... no
> configure: error: sysconfig module not found
> 
> Very recent versions of the buildfarm script will discard accache
> automatically after a configure or make failure, but I think the
> REL_11 you're running here doesn't have that defense.  It'll only
> flush accache after a change in the configure script in git.

Take 3. :)

I've upgraded everyone to the v14 buildfarm scripts and made sure the
--test passed on HEAD on each one.  So I hopefully didn't miss any
(other than the one EOL OpenSUSE version that I will plan on upgrading.)

Regards,
Mark

  70. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-23T01:50:07Z

    Mark Wong <markwkm@postgresql.org> writes:
> Take 3. :)

> I've upgraded everyone to the v14 buildfarm scripts and made sure the
> --test passed on HEAD on each one.  So I hopefully didn't miss any
> (other than the one EOL OpenSUSE version that I will plan on upgrading.)

Thanks!

However ... it seems like most of your animals haven't run at all
in the last couple of days.  Did you maybe forget to re-enable
their cron jobs after messing with them, or something like that?

			regards, tom lane

  71. Re: Time to drop plpython2?

    Mark Wong <markwkm@postgresql.org> — 2022-02-23T15:58:53Z

    On Tue, Feb 22, 2022 at 08:50:07PM -0500, Tom Lane wrote:
> Mark Wong <markwkm@postgresql.org> writes:
> > Take 3. :)
> 
> > I've upgraded everyone to the v14 buildfarm scripts and made sure the
> > --test passed on HEAD on each one.  So I hopefully didn't miss any
> > (other than the one EOL OpenSUSE version that I will plan on upgrading.)
> 
> Thanks!
> 
> However ... it seems like most of your animals haven't run at all
> in the last couple of days.  Did you maybe forget to re-enable
> their cron jobs after messing with them, or something like that?

Uh oh, more user error.  I tested run_build but run_branches wasn't
happy.  I'll start working through that...

Regards,
Mark

  72. Re: Time to drop plpython2?

    Mark Wong <markwkm@postgresql.org> — 2022-02-24T16:37:00Z

    On Wed, Feb 23, 2022 at 07:59:01AM -0800, Mark Wong wrote:
> On Tue, Feb 22, 2022 at 08:50:07PM -0500, Tom Lane wrote:
> > Mark Wong <markwkm@postgresql.org> writes:
> > > Take 3. :)
> > 
> > > I've upgraded everyone to the v14 buildfarm scripts and made sure the
> > > --test passed on HEAD on each one.  So I hopefully didn't miss any
> > > (other than the one EOL OpenSUSE version that I will plan on upgrading.)
> > 
> > Thanks!
> > 
> > However ... it seems like most of your animals haven't run at all
> > in the last couple of days.  Did you maybe forget to re-enable
> > their cron jobs after messing with them, or something like that?
> 
> Uh oh, more user error.  I tested run_build but run_branches wasn't
> happy.  I'll start working through that...

I think I have most of them operational again.  I see some animals are
still failing on some branches, but still better overall?

I discovered that clang for gadwall and pintail got purged when I
removed python2, so I disabled those two animals.

Regards,
Mark

  73. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-07T01:30:15Z

    Hi,

On 2022-02-16 22:52:08 -0800, Andres Freund wrote:
> On 2022-02-16 11:58:50 -0800, Andres Freund wrote:
> > Cool, will apply 1) later today.
>
> Done. Curious how red the BF will turn out to be. Let's hope it's not
> too bad.

Now that the BF has stabilized, I've rebased and cleaned up the patches I'd
posted earlier. Attached for the first time is my attempt at cleaning up the
docs.

0003, the removal of code level support for Python 2, is now a good bit bigger
bigger, due to the removal of the last remnants of the Py2/3 porting layer.

I did so far leave in the "major version conflict" detection stuff in
plpy_main.c - that could again be useful? I'm leaning towards removing it, I'd
hope that there's not again such a painful transition, and we have the git
history if needed.

Regards,

Andres

  74. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-07T01:42:03Z

    Hi,

On 2022-03-06 17:30:15 -0800, Andres Freund wrote:
> 0003, the removal of code level support for Python 2, is now a good bit bigger
> bigger, due to the removal of the last remnants of the Py2/3 porting layer.

Oh, huh. Something here seems to be broken, causing a crash on windows, but
not elsewhere.

https://cirrus-ci.com/task/5288088032247808
stack trace and regression.diffs:
https://api.cirrus-ci.com/v1/artifact/task/5288088032247808/crashlog/crashlog-postgres.exe_1768_2022-03-07_01-07-48-535.txt
https://api.cirrus-ci.com/v1/artifact/task/5288088032247808/log/src/pl/plpython/regression.diffs

The previous version ran successfully on windows too. So I probably screwed
something up when removing the remaining Py2/3 porting layer bits... The fact
that it fails on windows but not elsewhere perhaps suggests it's something
around the width of long?

Greetings,

Andres Freund

  75. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-03-07T20:05:03Z

    Andres Freund <andres@anarazel.de> writes:
> Now that the BF has stabilized, I've rebased and cleaned up the patches I'd
> posted earlier. Attached for the first time is my attempt at cleaning up the
> docs.

I looked through this quickly, and have a couple of nitpicks.  The
PGFILEDESC value for jsonb_plpython is worded randomly differently
from those for hstore_plpython and ltree_plpython.  I think we should
make it match those.  I also noted a couple of typos in the docs patch.
See attached delta patch (I named it .txt in hopes of not confusing
the cfbot).

I kind of wonder if we still need "46.1. Python 2 vs. Python 3" at
all.  It certainly doesn't seem like it still deserves its position
of honor as the first subsection.  Perhaps move it down to be the
last subsection?

Also, grepping reveals that vcregress.pl still has two stray references to
"plpythonu".  I did not touch that here, but maybe that has something
to do with the ci failure?

> I did so far leave in the "major version conflict" detection stuff in
> plpy_main.c - that could again be useful? I'm leaning towards removing it, I'd
> hope that there's not again such a painful transition, and we have the git
> history if needed.

I think we should leave it in.  I foresee that somebody will want to build
plpython2u as an out-of-core extension, at least for a few years yet.
If they do, and the core language does not have its half of that guard,
it'd be bad.

			regards, tom lane

  76. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-08T01:00:07Z

    Hi,

On 2022-03-07 15:05:03 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Now that the BF has stabilized, I've rebased and cleaned up the patches I'd
> > posted earlier. Attached for the first time is my attempt at cleaning up the
> > docs.
> 
> I looked through this quickly, and have a couple of nitpicks.  The
> PGFILEDESC value for jsonb_plpython is worded randomly differently
> from those for hstore_plpython and ltree_plpython.  I think we should
> make it match those.

Makes sense.


> I also noted a couple of typos in the docs patch.
> See attached delta patch

Thanks, incorporated.


> I kind of wonder if we still need "46.1. Python 2 vs. Python 3" at
> all.  It certainly doesn't seem like it still deserves its position
> of honor as the first subsection.  Perhaps move it down to be the
> last subsection?

Agreed. I think it's useful to have it, just so we have a place mentioning
plpython[2]u. I put it as the second last subsection for now, somehow seemed
to belong before the env variable list? But...


> Also, grepping reveals that vcregress.pl still has two stray references to
> "plpythonu".

I left that in somewhat intentionally, it seemed the cleanest way to remove
plpython2 from vcregress.pl. It's purely cosmetic afaics.

A related question is whether we want to remove $(python_majorversion)
references in the makefiles?


> I did not touch that here, but maybe that has something to do with the ci
> failure?

The CI failure was caused by me screwing up search and replace
:(. Accidentally replaced PyString_AsString with PyUnicode_AsString instead of
P*L*yUnicode_AsString.

Somewhat surprised at that causing problems only on windows (even valgrind
didn't, although the required suppressions might have been the cause of
that). But well, better there than nowhere...


> > I did so far leave in the "major version conflict" detection stuff in
> > plpy_main.c - that could again be useful? I'm leaning towards removing it, I'd
> > hope that there's not again such a painful transition, and we have the git
> > history if needed.
> 
> I think we should leave it in.  I foresee that somebody will want to build
> plpython2u as an out-of-core extension, at least for a few years yet.
> If they do, and the core language does not have its half of that guard,
> it'd be bad.

Makes sense. Added a comment to that effect.


I noticed a few references to PLyString in function names. Probably should
replace them with PLyUnicode. Didn't do that yet in the attached...


Greetings,

Andres Freund

  77. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-03-08T01:11:52Z

    Andres Freund <andres@anarazel.de> writes:
> A related question is whether we want to remove $(python_majorversion)
> references in the makefiles?

I wouldn't.  I'm doubtful of your theory that there will never be
a Python 4.

This version seems ready-to-go to me, or at least ready to see
what the buildfarm makes of it.

			regards, tom lane

  78. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-08T02:37:49Z

    On 2022-03-07 20:11:52 -0500, Tom Lane wrote:
> This version seems ready-to-go to me, or at least ready to see
> what the buildfarm makes of it.

Pushed. Let's see...

  79. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-03-08T04:39:39Z

    Andres Freund <andres@anarazel.de> writes:
> On 2022-03-07 20:11:52 -0500, Tom Lane wrote:
>> This version seems ready-to-go to me, or at least ready to see
>> what the buildfarm makes of it.

> Pushed. Let's see...

wrasse says you were too quick to drop plpython_error_5.out.
Otherwise looks pretty good so far.

			regards, tom lane

  80. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-08T04:59:16Z

    Hi,

On 2022-03-07 23:39:39 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2022-03-07 20:11:52 -0500, Tom Lane wrote:
> >> This version seems ready-to-go to me, or at least ready to see
> >> what the buildfarm makes of it.
> 
> > Pushed. Let's see...
> 
> wrasse says you were too quick to drop plpython_error_5.out.

Does look like it. I'll try to find a distribution with an old python...


> Otherwise looks pretty good so far.

crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
pre-existing issue that just didn't happen to cause problems...

Greetings,

Andres Freund

  81. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-08T18:42:31Z

    Hi,

On 2022-03-07 20:59:16 -0800, Andres Freund wrote:
> On 2022-03-07 23:39:39 -0500, Tom Lane wrote:
> > Andres Freund <andres@anarazel.de> writes:
> > wrasse says you were too quick to drop plpython_error_5.out.
> 
> Does look like it. I'll try to find a distribution with an old python...

debian 8 did the trick.

The output from older python was in plpython_error.out the output from newer
python in plpython_error_5.out. Looked like python2 specific output.

Pushed.

I guess I should have grokked the _5 suffix - but the _3 suffix was used for
python3 specific content, so it wasn't obvious. Should we rename the file to
plpython_error_35.out?

A bit depressing to have a 500 line alternative output file for a one line
diff :(.


> > Otherwise looks pretty good so far.
> 
> crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
> pre-existing issue that just didn't happen to cause problems...

Fixed that.

Greetings,

Andres Freund

  82. Re: Time to drop plpython2?

    Tom Lane <tgl@sss.pgh.pa.us> — 2022-03-08T18:49:15Z

    Andres Freund <andres@anarazel.de> writes:
> A bit depressing to have a 500 line alternative output file for a one line
> diff :(.

Yeah.  How badly do we need that particular test case?

			regards, tom lane

  83. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-08T19:03:18Z

    Hi,

On 2022-03-08 13:49:15 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > A bit depressing to have a 500 line alternative output file for a one line
> > diff :(.
>
> Yeah.  How badly do we need that particular test case?

A bit hard to tell. The test was introduced in

commit 2bd78eb8d51cc9ee03ba0287b23ff4c266dcd9b9
Author: Peter Eisentraut <peter_e@gmx.net>
Date:   2011-04-06 22:36:06 +0300

    Add traceback information to PL/Python errors

    This mimics the traceback information the Python interpreter prints
    with exceptions.

    Jan Urbański

and contains this ominous line:

+/* AttributeError at toplevel used to give segfaults with the traceback
+*/

Greetings,

Andres Freund

  84. Re: Time to drop plpython2?

    Andres Freund <andres@anarazel.de> — 2022-03-08T20:02:00Z

    Hi,

On 2022-03-08 10:42:31 -0800, Andres Freund wrote:
> > crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
> > pre-existing issue that just didn't happen to cause problems...
> 
> Fixed that.

Hm. Now crake failed in XversionUpgrade-REL9_2_STABLE-HEAD:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-03-08%2018%3A47%3A22

except that the log doesn't actually indicate any problem? Andrew, any hint?

Greetings,

Andres Freund

  85. Re: Time to drop plpython2?

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2022-03-09T09:29:17Z

    On 08.03.22 20:03, Andres Freund wrote:
> Hi,
> 
> On 2022-03-08 13:49:15 -0500, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> A bit depressing to have a 500 line alternative output file for a one line
>>> diff :(.
>>
>> Yeah.  How badly do we need that particular test case?
> 
> A bit hard to tell. The test was introduced in
> 
> commit 2bd78eb8d51cc9ee03ba0287b23ff4c266dcd9b9
> Author: Peter Eisentraut <peter_e@gmx.net>
> Date:   2011-04-06 22:36:06 +0300
> 
>      Add traceback information to PL/Python errors

We would probably try to write this test differently today, but at this 
point I wouldn't bother and just wait for Python 3.5 to fall off the end 
of the conveyor belt.

  86. Re: Time to drop plpython2?

    Andrew Dunstan <andrew@dunslane.net> — 2022-03-09T12:53:54Z

    On 3/8/22 15:02, Andres Freund wrote:
> Hi,
>
> On 2022-03-08 10:42:31 -0800, Andres Freund wrote:
>>> crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
>>> pre-existing issue that just didn't happen to cause problems...
>> Fixed that.
> Hm. Now crake failed in XversionUpgrade-REL9_2_STABLE-HEAD:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-03-08%2018%3A47%3A22
>
> except that the log doesn't actually indicate any problem? Andrew, any hint?
>


That was a snafu on my part, as I was adding extension upgrade / amcheck
testing. It's fixed now, so please disregard this one.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com