Horton

Thread

Commits

  1. Adjust hints and docs to suggest CREATE EXTENSION not CREATE LANGUAGE.

  1. obsoleting plpython2u and defaulting plpythonu to plpython3u

    Pavel Raiskup <praiskup@redhat.com> — 2018-04-24T11:13:17Z

    Forwarding to hackers.  Hope that's OK after few days.

Pavel

----------  Forwarded Message  ----------

Subject: obsoleting plpython2u and defaulting plpythonu to plpython3u
Date: Thursday, April 19, 2018, 11:56:40 AM CEST
From: Pavel Raiskup <praiskup@redhat.com>
To: pgsql-general <pgsql-general@postgresql.org>

Per current plpython docs:

    The language named plpythonu implements PL/Python based on the default
    Python language variant, which is currently Python 2. (This default is
    independent of what any local Python installations might consider to
    be their “default”, for example, what /usr/bin/python might be.) The
    default will probably be changed to Python 3 in a distant future
    release of PostgreSQL, depending on the progress of the migration to
    Python 3 in the Python community.

.. the status quo seems to be bit optimistic with the "distant future",
and we should start thinking about dropping plpython2 support, same as
upstream (a bit optimistically too, IMO) does [1].

The docs don't suggest the explicit use of plpython2, but still the docs
are not discouraging from it -- so it is likely some clusters run against
that.

What's the expected future migration path from plpython2 to plpython3 in
such cases?  I'm thinking about rewrite of the docs and creating some
scripting which could simplify the migration steps.  Would such patches be
welcome at this point?

[1] https://pythonclock.org/

Pavel
-----------------------------------------

  2. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2018-04-24T22:17:05Z

    On 4/24/18 07:13, Pavel Raiskup wrote:
> .. the status quo seems to be bit optimistic with the "distant future",
> and we should start thinking about dropping plpython2 support, same as
> upstream (a bit optimistically too, IMO) does [1].

I don't think we are going to drop Python 2 support anytime soon.

> What's the expected future migration path from plpython2 to plpython3 in
> such cases?  I'm thinking about rewrite of the docs and creating some
> scripting which could simplify the migration steps.  Would such patches be
> welcome at this point?

I'm not sure what you have in mind.  In many cases, you can just change
the LANGUAGE clause.  I suppose you could run 2to3 over the function
body?  Write a PL/Python function to run 2to3?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  3. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Pavel Raiskup <praiskup@redhat.com> — 2018-04-25T08:50:17Z

    Thanks Peter,

On Wednesday, April 25, 2018 12:17:05 AM CEST Peter Eisentraut wrote:
> On 4/24/18 07:13, Pavel Raiskup wrote:
> > .. the status quo seems to be bit optimistic with the "distant future",
> > and we should start thinking about dropping plpython2 support, same as
> > upstream (a bit optimistically too, IMO) does [1].
> 
> I don't think we are going to drop Python 2 support anytime soon.

Well, no hurry, just saying that we should start thinking about it.
Mostly to have proper plan in advance because production support of
plpython2 will be at least risky once Python 2 is EOL...

Linux distributions are spending a lot of efforts to make Python 3 the
default; and the plan is to drop Python 2 support entirely if possible.

So full drop is not what ask for now (it will be needed for lts/enterprise
distros for quite some time anyway), I just think that plpython3 should be
the default (by build system, by 'plpythonu'=>'plpython3u', etc.).  And
maybe we could plan "deprecating" the plpython2 first in future major
releases (not really 11 or 12, I have nothing concrete in mind now).

> > What's the expected future migration path from plpython2 to plpython3 in
> > such cases?  I'm thinking about rewrite of the docs and creating some
> > scripting which could simplify the migration steps.  Would such patches be
> > welcome at this point?
> 
> I'm not sure what you have in mind.  In many cases, you can just change
> the LANGUAGE clause.  I suppose you could run 2to3 over the function
> body?  Write a PL/Python function to run 2to3?

Something along those lines, yes.  Well, I don't have a real plan.  I'd
like to discuss what's the expected way out, how to give a chance to users
to migrate from plpython2 to plpython3 _now_ (since both are yet
available).

I _guess_ the users don't care much about 2 vs. 3 in plpython - so they
usually simply live with implicit plpython version - which would OTOH mean
that the majority will have to face the migration sooner or later.

Pavel

  4. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2018-04-26T20:12:16Z

    On 4/25/18 04:50, Pavel Raiskup wrote:
> So full drop is not what ask for now (it will be needed for lts/enterprise
> distros for quite some time anyway), I just think that plpython3 should be
> the default (by build system, by 'plpythonu'=>'plpython3u', etc.).

I don't think we should do that unless there is an update to PEP 394.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  5. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-26T20:39:44Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 4/25/18 04:50, Pavel Raiskup wrote:
>> So full drop is not what ask for now (it will be needed for lts/enterprise
>> distros for quite some time anyway), I just think that plpython3 should be
>> the default (by build system, by 'plpythonu'=>'plpython3u', etc.).

> I don't think we should do that unless there is an update to PEP 394.

PEP 394 points out that some distros (at least Arch) have already switched
"python" to mean "python3", and I gather from Pavel's inquiry that the day
is in sight when Fedora will do that.  Perhaps we should think about
providing a configure switch to control whether "plpythonu" means
"plpython2u" or "plpython3u", so that distros can reasonably easily sync
PG's behavior with whatever they've chosen to do with the python shell
command.  I agree that it's not PG's business to be out in front of this
change, but maybe we shouldn't be holding it back, either.

I'm not very sure how many moving parts would be involved in making that
happen.  One thing we should probably do first is retire the pg_pltemplate
system catalog in favor of making PL extension scripts self-contained,
as you'd mentioned recently would be a good project to finish off.

			regards, tom lane

  6. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2018-04-26T22:29:59Z

    
On 04/26/2018 04:39 PM, Tom Lane wrote:
> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>> On 4/25/18 04:50, Pavel Raiskup wrote:
>>> So full drop is not what ask for now (it will be needed for lts/enterprise
>>> distros for quite some time anyway), I just think that plpython3 should be
>>> the default (by build system, by 'plpythonu'=>'plpython3u', etc.).
>> I don't think we should do that unless there is an update to PEP 394.
> PEP 394 points out that some distros (at least Arch) have already switched
> "python" to mean "python3", and I gather from Pavel's inquiry that the day
> is in sight when Fedora will do that.  Perhaps we should think about
> providing a configure switch to control whether "plpythonu" means
> "plpython2u" or "plpython3u", so that distros can reasonably easily sync
> PG's behavior with whatever they've chosen to do with the python shell
> command.  I agree that it's not PG's business to be out in front of this
> change, but maybe we shouldn't be holding it back, either.
>
> I'm not very sure how many moving parts would be involved in making that
> happen.  One thing we should probably do first is retire the pg_pltemplate
> system catalog in favor of making PL extension scripts self-contained,
> as you'd mentioned recently would be a good project to finish off.
>
> 			


After installing python3, I switched buildfarm animal prion to build and
test with python3 instead of python2 a day or two ago. All this required
was setting PYTHON=/usr/bin/python3 in the environment. Everything else
Just Worked.

cheers

andrew

-- 

Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  7. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-26T22:41:50Z

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> After installing python3, I switched buildfarm animal prion to build and
> test with python3 instead of python2 a day or two ago. All this required
> was setting PYTHON=/usr/bin/python3 in the environment. Everything else
> Just Worked.

That just means we've hacked the regression tests to the point where
they work in that environment ;-).  An installation from that code
would still think plpythonu means plpython2u.

			regards, tom lane

  8. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Robert Haas <robertmhaas@gmail.com> — 2018-04-27T14:50:35Z

    On Thu, Apr 26, 2018 at 4:39 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm not very sure how many moving parts would be involved in making that
> happen.  One thing we should probably do first is retire the pg_pltemplate
> system catalog in favor of making PL extension scripts self-contained,
> as you'd mentioned recently would be a good project to finish off.

Incidentally, it recently came to my notice that we're still issuing
hints that say:

Use CREATE LANGUAGE to load the language into the database.

That is now, and has been for some time, the wrong advice.  We should
really be recommending CREATE EXTENSION.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2018-04-27T15:10:31Z

    On 4/26/18 16:39, Tom Lane wrote:
>> I don't think we should do that unless there is an update to PEP 394.
> 
> PEP 394 points out that some distros (at least Arch) have already switched
> "python" to mean "python3",

My recollection is that PEP 394 was written because Arch jumped the gun
and the PEP is saying "do this instead".

> and I gather from Pavel's inquiry that the day
> is in sight when Fedora will do that.

According to the plan I found at
<https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3>, they
will also wait for an update to PEP 394 ("and we will drive that update
if needed").

So that's all great.  There are some opportunities for prep work in the
meantime that have been mentioned.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  10. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-27T15:44:52Z

    Robert Haas <robertmhaas@gmail.com> writes:
> Incidentally, it recently came to my notice that we're still issuing
> hints that say:
> Use CREATE LANGUAGE to load the language into the database.
> That is now, and has been for some time, the wrong advice.  We should
> really be recommending CREATE EXTENSION.

Right.  It's too late in the v11 cycle to consider any of this other work
right now, but I think it'd be reasonable to adjust those message(s) for
v11 ... any objections?  I see two places in functioncmds.c that ought
to change, as well as a few examples in the SGML docs.

			regards, tom lane

  11. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-27T15:51:57Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 4/26/18 16:39, Tom Lane wrote:
>> and I gather from Pavel's inquiry that the day
>> is in sight when Fedora will do that.

> According to the plan I found at
> <https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3>, they
> will also wait for an update to PEP 394 ("and we will drive that update
> if needed").

Note that there's a specific timeline there: they expect PEP 394 to get
changed not more than 2 years from now.  I am not sure if Red Hat has
enough pull in the Python community to guarantee that will happen, but
do you want to bet against it?

> So that's all great.  There are some opportunities for prep work in the
> meantime that have been mentioned.

We should evidently plan on making this stuff happen in v12, or we're
going to be behind the curve.

			regards, tom lane

  12. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Pavel Raiskup <praiskup@redhat.com> — 2018-04-27T16:38:05Z

    On Friday, April 27, 2018 5:51:57 PM CEST Tom Lane wrote:
> Note that there's a specific timeline there: they expect PEP 394 to get
> changed not more than 2 years from now.  I am not sure if Red Hat has
> enough pull in the Python community to guarantee that will happen, but
> do you want to bet against it?

Well, also it depends what's meant by "get PEP 394 changed".  My guess is
that distros might well avoid providing /bin/python if that PEP is
a concern..  that's actually typical situation in Fedora even now; python2
package isn't installed by default.  And it can disappear entirely if
that's too expensive for maintenance.

To me, PEP 394 is mostly about python distributors - not about python
projects;  so IMO for PostgreSQL that PEP is a bit orthogonal thing.
For quite some time already, doing 'CREATE EXTENSION plpython' implies at
least some (maybe serious) migration headaches in not so distant future.

> > So that's all great.  There are some opportunities for prep work in the
> > meantime that have been mentioned.
> 
> We should evidently plan on making this stuff happen in v12, or we're
> going to be behind the curve.

Agreed, but I'm still not sure what would be the expected 2->3 migration
path.  If that change happens in PG v12, people will migrate to that from
v11 and there's the trouble.  Would it technically possible to upgrade
such clusters with pg_upgrade when 'plpythonu' silently goes from
'plpython2.so' to 'plpython3.so' behind the scenes?

Or is it expected that people will migrate semi-automatically/manually
from plpython2 to plpython3 on v11 - before hitting pg_upgrade?

Pavel

  13. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-27T17:19:43Z

    Pavel Raiskup <praiskup@redhat.com> writes:
> On Friday, April 27, 2018 5:51:57 PM CEST Tom Lane wrote:
>> We should evidently plan on making this stuff happen in v12, or we're
>> going to be behind the curve.

> Agreed, but I'm still not sure what would be the expected 2->3 migration
> path.  If that change happens in PG v12, people will migrate to that from
> v11 and there's the trouble.  Would it technically possible to upgrade
> such clusters with pg_upgrade when 'plpythonu' silently goes from
> 'plpython2.so' to 'plpython3.so' behind the scenes?
> Or is it expected that people will migrate semi-automatically/manually
> from plpython2 to plpython3 on v11 - before hitting pg_upgrade?

Good questions.  I think that the most useful thing we could do in the
near term is to cause functions declared with LANGUAGE 'plpythonu'
to be declared with LANGUAGE 'plpython2u' as-stored.  It'd then be on
the user's head to migrate at some point, changing that to LANGUAGE
'plpython3u' along with any required changes to the function body.
(This parallels the remarks in the Fedora migration plan about
discouraging packages from referencing "python" with no version number.)
The time frame for doing that would depend on the user's tolerance for
running out-of-support language code and/or their platform's tolerance for
shipping same.  I'd imagine it would be quite a long time before PG proper
would say we no longer support python2, but individual distros would stop
shipping those packages whenever their policy dictates.

If we go down that path, then 'plpythonu' isn't really a language anymore,
nor an extension --- it's just an alias that CREATE FUNCTION recognizes
and replaces.  In the past we've had needs of that sort and generally just
hacked it with some hard-wired code (see, eg, the overdue-for-retirement
code in ResolveOpClass()).  I'm tempted to propose that we handle this
case with something equally grotty, that is basically do this in
CreateFunction:

	if (strcmp(language, "plpythonu") == 0)
	    language = DEFAULT_PYTHON_LANGUAGE;

where DEFAULT_PYTHON_LANGUAGE is defined as either "plpython2u" or
"plpython3u" according to a configure switch.  You could imagine
building some significantly more complicated infrastructure, but
I doubt it would be worth the effort; how many other use-cases are
we likely to have?

If we agree that something like this is the way to go, I wonder if we
shouldn't squeeze it into v11.  If we wait till v12, that's not going
to ship till late 2019, which is going to be getting uncomfortably
close to when PEP 394 might change if we believe Fedora's schedule.

			regards, tom lane

  14. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Andres Freund <andres@anarazel.de> — 2018-04-27T17:28:20Z

    On 2018-04-27 13:19:43 -0400, Tom Lane wrote:
> If we agree that something like this is the way to go, I wonder if we
> shouldn't squeeze it into v11.  If we wait till v12, that's not going
> to ship till late 2019, which is going to be getting uncomfortably
> close to when PEP 394 might change if we believe Fedora's schedule.

I think that'd be reasonable.

Another alternative would be to have a 'plpython' extension that depends
on plpython2. That'd require users to specify CASCADE when creating it,
but that actually seems like it could be a useful hint...  I think it's
probably not worth going that route though, because reassigning objects
from one extension to another is more work than reasonable...

Greetings,

Andres Freund

  15. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-04-27T17:34:58Z

    Andres Freund <andres@anarazel.de> writes:
> Another alternative would be to have a 'plpython' extension that depends
> on plpython2. That'd require users to specify CASCADE when creating it,
> but that actually seems like it could be a useful hint...  I think it's
> probably not worth going that route though, because reassigning objects
> from one extension to another is more work than reasonable...

Yeah, there's a separate set of questions about what happens during
pg_upgrade of a database containing the existing plpythonu extension.

You could imagine hacking the dump/reload case by defining plpythonu
as an empty extension with a dependency on plpython2u (or, maybe
someday in future, a dependency on plpython3u).  But that won't do
the trick for binary upgrades; we might need special-case code in
pg_dump/pg_upgrade to fix that.

			regards, tom lane

  16. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Andres Freund <andres@anarazel.de> — 2018-04-27T17:37:32Z

    On 2018-04-27 13:34:58 -0400, Tom Lane wrote:
> Yeah, there's a separate set of questions about what happens during
> pg_upgrade of a database containing the existing plpythonu extension.
> 
> You could imagine hacking the dump/reload case by defining plpythonu
> as an empty extension with a dependency on plpython2u (or, maybe
> someday in future, a dependency on plpython3u).  But that won't do
> the trick for binary upgrades; we might need special-case code in
> pg_dump/pg_upgrade to fix that.

One way to deal with that is have an upgrade script that does the
reassignment. Seems a mite cleaner than doing it in pg_dump. We could
just have a query updating pg_depend to reference the new extension
rather than the old.  That'd mean we'd be in the "old" situation before
somebody an extension update ran, but that seems hard to avoid?

Greetings,

Andres Freund

  17. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2018-04-30T19:12:30Z

    On 4/27/18 12:38, Pavel Raiskup wrote:
> Well, also it depends what's meant by "get PEP 394 changed".  My guess is
> that distros might well avoid providing /bin/python if that PEP is
> a concern..  that's actually typical situation in Fedora even now; python2
> package isn't installed by default.  And it can disappear entirely if
> that's too expensive for maintenance.

I'm wondering who this situation is going to be resolved.  Is anyone
going to switch /usr/bin/python to Python 3 in the foreseeable future?
A few have tried and regretted it.

Or is /usr/bin/python just going to go away when Python 2 goes away?

What we do in PostgreSQL would IMO depend on which of these choices is
going to be the dominant one.

If the former, then we can just switch plpythonu around to be Python 3
(perhaps a configure option).

If the latter, then we don't need to do anything, except perhaps warn
people to upgrade.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  18. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Yuriy Zhuravlev <stalkerg@gmail.com> — 2018-04-30T22:33:47Z

    And Gentoo too.

> PEP 394 points out that some distros (at least Arch) have already switched
>

  19. Re: obsoleting plpython2u and defaulting plpythonu to plpython3u

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-08-23T18:27:47Z

    Pavel Raiskup <praiskup@redhat.com> writes:
> On Wednesday, April 25, 2018 12:17:05 AM CEST Peter Eisentraut wrote:
>> On 4/24/18 07:13, Pavel Raiskup wrote:
>>> What's the expected future migration path from plpython2 to plpython3 in
>>> such cases?  I'm thinking about rewrite of the docs and creating some
>>> scripting which could simplify the migration steps.  Would such patches be
>>> welcome at this point?

>> I'm not sure what you have in mind.  In many cases, you can just change
>> the LANGUAGE clause.  I suppose you could run 2to3 over the function
>> body?  Write a PL/Python function to run 2to3?

> Something along those lines, yes.  Well, I don't have a real plan.  I'd
> like to discuss what's the expected way out, how to give a chance to users
> to migrate from plpython2 to plpython3 _now_ (since both are yet
> available).
> I _guess_ the users don't care much about 2 vs. 3 in plpython - so they
> usually simply live with implicit plpython version - which would OTOH mean
> that the majority will have to face the migration sooner or later.

It's still not very clear to me whether we can or should change anything
about what "plpythonu" means.  However, it seems like we definitely ought
to be encouraging PL/Python users to switch their code to Python 3.

As an exercise, I wrote a little script that runs through all the Python 2
functions in a database and converts them to Python 3, insofar as "2to3"
can do that.  This is by a noticeable margin the largest Python script
I've ever written, so I'm sure it sucks stylistically and perhaps
functionally too, but it will serve to get some discussion started.
Should we provide something like this to users, and if so how exactly
(installed file and/or documentation)?  Would it be sane to squeeze
into v12 at this late date?  I suspect that Fedora might want to have it
in their v12 distro even if we only put it into HEAD.

			regards, tom lane