Thread

1. Python 3.1 support

   Peter Eisentraut <peter_e@gmx.net> — 2009-11-12T19:54:01Z

   Here's the patch to support Python >=3.1 with PL/Python.  The
   compatibility code is mostly in line with the usual 2->3 C porting
   practice and is documented inline.
   
   I needed to create an arguably weird hack to manage the regression
   tests.  Instead of creating a new expected file for pretty much every
   test file and also for some input files (where Python syntax had
   changed), a sed script creates a complete Python 3 compatible set of
   input and output files.  Doesn't look pretty but works quite well.  If
   anyone has a better idea, please let me know.
   

2. Re: Python 3.1 support

   Tom Lane <tgl@sss.pgh.pa.us> — 2009-11-12T21:06:19Z

   Peter Eisentraut <peter_e@gmx.net> writes:
   > Here's the patch to support Python >=3.1 with PL/Python.  The
   > compatibility code is mostly in line with the usual 2->3 C porting
   > practice and is documented inline.
   
   There was considerable debate earlier about whether we wanted to treat
   Python 3 as a separate PL so it could be available in parallel with
   plpython 2, because of the user-level coding incompatibilities.  It
   looks like this patch simply ignores that problem.  What is going to
   happen to plpython functions that depend on 2.x behavior?
   
   			regards, tom lane
   
   

3. Re: Python 3.1 support

   James William Pye <lists@jwp.name> — 2009-11-13T01:42:29Z

   On Nov 12, 2009, at 12:54 PM, Peter Eisentraut wrote:
   
   > Here's the patch to support Python >=3.1 with PL/Python.
   
   :\
   
   I was hoping to be able to use Python 3 to draw a clear distinction between plpython and the would-be "plpython3" that I've been working on. I understand that you're not in favor of a brand new implementation for Python 3. Despite my dislike for that position(well, it would seem to be in opposition to my initiative, so naturally =), I don't entirely disagree with your rationale[wrt doing things more incrementally]. For me, plpython has never been what I would call a pleasure to use, and many of the gripes that I have with it are, IMO, entrenched far enough into the implementation that any efforts to change it would(should? =) cause unacceptable breakage in user applications(?). Well, as far as additional Python interfaces are concerned, a lot of redundant functionality, but that's not even the half of it.
   
   
   [I was hoping to get to a status message this weekend,
    but it seems like I should follow-up here. =]
   
   
   So here's where I'm at:
   --
   Mostly documentation improvements since I last pinged -hackers.
   Still, *sigh*, filling in documentation and fighting bugs as I go.
   Currently resolving a bug instantiating MD arrays from nested lists.
   Once I'm "finished" with the docs, I'm going to start looking for refcount leaks.
   No major additions or changes are planned, but I have been making some minor additions as I write more docs.
   
   Overview/Features:
     http://wiki.postgresql.org/wiki/WIP:plpython3
   Documentation:
     http://python.projects.postgresql.org/pldocs/plpython3.html
   git repo[see the plpython3 branch]:
     http://git.postgresql.org/gitweb?p=plpython3.git;a=summary
   
   Most of the documented interfaces have tests. I only have two platforms at my disposal, so I do fear that this will not "just work" on all of PG's supported platforms. Specifically, I've ran the tests on freebsd/amd64 and Mac10.6/intel(of course 10.5 as well for some earlier revisions). [err, actually, it's been a while since I ran the tests on freebsd.]
   --
   
   
   plpython3 is turning out to be kinda beefy(~974K diff[eh, there is some fluff in there]), and I can't say that I've seen much interest in it, so I can't really blame anyone if -hackers ends up taking a pass on it. (python3 is too far away for most folk to care? folk are content with plpython?)
   
   
   eh, cheers, either way. =)
   

4. Re: Python 3.1 support

   Peter Eisentraut <peter_e@gmx.net> — 2009-11-13T08:51:35Z

   On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
   > Peter Eisentraut <peter_e@gmx.net> writes:
   > > Here's the patch to support Python >=3.1 with PL/Python.  The
   > > compatibility code is mostly in line with the usual 2->3 C porting
   > > practice and is documented inline.
   > 
   > There was considerable debate earlier about whether we wanted to treat
   > Python 3 as a separate PL so it could be available in parallel with
   > plpython 2, because of the user-level coding incompatibilities.  It
   > looks like this patch simply ignores that problem.
   
   Exactly how to package that is something to be determined by the
   packagers, and we can give them the support they need.  But first you
   need code that works with Python 3, which is what this patch does.
   
   > What is going to happen to plpython functions that depend on 2.x behavior?
   
   The porting path from 2 to 3 is pretty well established.  You first port
   to 2.6, then remove all the old features, then move to 3.x.  This is not
   something we have to reinvent.  The only significant differences that
   you can't use in 2.6 without future imports are unicode literals and the
   print function, both of which are not in common use in PL/Python.
   
   
   
   

5. Re: Python 3.1 support

   Peter Eisentraut <peter_e@gmx.net> — 2009-11-13T11:47:13Z

   On tor, 2009-11-12 at 18:42 -0700, James Pye wrote:
   > For me, plpython has never been what I would call a pleasure to use,
   > and many of the gripes that I have with it are, IMO, entrenched far
   > enough into the implementation that any efforts to change it
   > would(should? =) cause unacceptable breakage in user applications(?).
   
   Has this list of gripes ever been brought up and discussed in this
   forum?
   
   
   

6. Re: Python 3.1 support

   James William Pye <lists@jwp.name> — 2009-11-13T18:27:06Z

   On Nov 13, 2009, at 4:47 AM, Peter Eisentraut wrote:
   > Has this list of gripes ever been brought up and discussed in this
   > forum?
   
   
   Some are TODOs, so in part by other people. Some were briefly touched on in the recent past discussions(around the time that I announced the WIP). Native typing vs conversion, function fragments vs function modules.
   I don't think native typing has seen any actual discussion, but I do recall mentioning it as something that I wanted to do(implicitly griped?).
   
   ...
   
   There is a difference in the situation from the discussion before. Prior, it was, "I would like to implement a new PL for Python 3 with these features", and now, it is, "I have implemented a new PL for Python 3 with these features".
   Simply, -hackers can choose among moving forward with Python 3 support in plpython or going with "plpython3" or even both, I suppose(?). Naturally, I'm biased toward something that involves plpython3, so I don't think I can(should?) be of much help to -hackers as a Python & PG user in any such decision. Of course, excepting the provision of justifications for my implementation/design choices...
   
   
   I would really love to see some input from Python users.
   
   
   I certainly don't want to waste time trying to get something into pgsql that Python users don't want.
   
   
   [here's a gripe that I haven't brought up as I think it is a matter of taste]
   
   I find (plpython) trigger functions to be a bit odd. I think it's the way in which manipulation/suppression decisions are made in BEFORE ROW triggers(return "OK", "SKIP", etc).. [label this as opinion at this point as I have yet to be able to nail down what, specifically, is "wrong" or un-pythonic about them.]
   
   Also, having distinct entry points to handle trigger events helps reduce potential errors by forcing the user to explicitly state the events that the trigger function can handle. Currently, in plpython, users *should* do sanity checks.
   
   Function modules opened the door for implementing this in a natural way, multiple functions(entry points) in the function module.
   
   http://python.projects.postgresql.org/pldocs/plpython3-programming.html#PLPYTHON3-FUNCTIONS-TRIGGERS
   

7. Re: Python 3.1 support

   Peter Eisentraut <peter_e@gmx.net> — 2009-11-15T13:37:17Z

   On fre, 2009-11-13 at 11:27 -0700, James Pye wrote:
   > Some are TODOs, so in part by other people. Some were briefly touched
   > on in the recent past discussions(around the time that I announced the
   > WIP). Native typing vs conversion, function fragments vs function
   > modules.
   
   I'm of course only one user, but these two features don't excite me at
   all, and certainly not enough to go through the pain of dealing with a
   second implementation.
   
   
   

8. Re: Python 3.1 support

   James William Pye <lists@jwp.name> — 2009-11-16T01:39:33Z

   On Nov 15, 2009, at 6:37 AM, Peter Eisentraut wrote:
   
   > but these two features don't excite me at all, 
   
   hrm.. at all?
   
   I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around...
   
   AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object.
   
   OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.)
   
   [looking at the PL/Python todo list..]
   
   Excepting DB-API and trusted, I believe all the current PL/Python TODOs are fulfilled or N/A in plpython3... ugh, the docs are not yet complete, but I like to think of them as "better" anyways. :P
   
   
   > the pain of dealing with a second implementation.
   
   What pain are you anticipating? Maintenance?
   

9. Re: Python 3.1 support

   Peter Eisentraut <peter_e@gmx.net> — 2009-11-18T15:37:29Z

   On sön, 2009-11-15 at 18:39 -0700, James Pye wrote:
   > I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around...
   
   The question is whether it helps the user, not the implementer.  As far
   as I can tell, it just creates more typing for no benefit whatsoever.
   Also, it's inconsistent with normal Python script files and with other
   PLs.
   
   > AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object.
   > 
   > OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.)
   
   Right, if I use PL/Python, I do it because I want to use Python.  I
   don't need another PostgreSQL implementation on top of Python.  The
   maintenance effort required to keep those two consistent aside.
   
   Again, I'm only one user.  But so far I haven't seen anyone else speak
   up here, and clearly accepting this for inclusion will need nontrivial
   convincing.
   
   > > the pain of dealing with a second implementation.
   > 
   > What pain are you anticipating? Maintenance?
   
   Right.
   
   
   

10. Re: Python 3.1 support

    Nathan Boley <npboley@gmail.com> — 2009-11-18T16:43:14Z

    > Again, I'm only one user.  But so far I haven't seen anyone else speak
    > up here, and clearly accepting this for inclusion will need nontrivial
    > convincing.
    
    Well, FWIW, I am excited about better type integration.
    
    Also, I am a little skeptical about this patch. I am sorry if this has
    already been discussed, but would this mean that I need to choose
    whether pl/python is built against Python 2.* or Python 3.*?
    
    -Nathan
    
    

11. Re: Python 3.1 support

    Tom Lane <tgl@sss.pgh.pa.us> — 2009-11-18T17:06:58Z

    Nathan Boley <npboley@gmail.com> writes:
    > Also, I am a little skeptical about this patch. I am sorry if this has
    > already been discussed, but would this mean that I need to choose
    > whether pl/python is built against Python 2.* or Python 3.*?
    
    Yes.  That's exactly what I was complaining about upthread.  I'm not
    a Python user, but from what I can gather of the 2-to-3 changes,
    having to choose one at package build time is going to be a disaster.
    
    			regards, tom lane
    
    

12. Re: Python 3.1 support

    Joshua D. Drake <jd@commandprompt.com> — 2009-11-18T17:21:23Z

    On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
    > Nathan Boley <npboley@gmail.com> writes:
    > > Also, I am a little skeptical about this patch. I am sorry if this has
    > > already been discussed, but would this mean that I need to choose
    > > whether pl/python is built against Python 2.* or Python 3.*?
    > 
    > Yes.  That's exactly what I was complaining about upthread.  I'm not
    > a Python user, but from what I can gather of the 2-to-3 changes,
    > having to choose one at package build time is going to be a disaster.
    > 
    
    Agreed. We really need to have a plpython and plpython3. Heck this would
    play nicely too because we support backward compatibility but also
    upward version differences.
    
    Joshua D. Drake
    
    
    > 			regards, tom lane
    > 
    
    
    -- 
    PostgreSQL.org Major Contributor
    Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
    Consulting, Training, Support, Custom Development, Engineering
    If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
    
    
    

13. Re: Python 3.1 support

    Tom Lane <tgl@sss.pgh.pa.us> — 2009-11-18T17:28:55Z

    "Joshua D. Drake" <jd@commandprompt.com> writes:
    > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
    >> Yes.  That's exactly what I was complaining about upthread.  I'm not
    >> a Python user, but from what I can gather of the 2-to-3 changes,
    >> having to choose one at package build time is going to be a disaster.
    
    > Agreed. We really need to have a plpython and plpython3.
    
    Peter was concerned about duplicative maintenance effort, but what I
    think this patch shows is that (at least for the near future) both
    could be built from a single source file.  What we need is configure
    and makefile support to do that.
    
    			regards, tom lane
    
    

14. Re: Python 3.1 support

    Joshua D. Drake <jd@commandprompt.com> — 2009-11-18T17:48:31Z

    On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote:
    > "Joshua D. Drake" <jd@commandprompt.com> writes:
    > > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
    > >> Yes.  That's exactly what I was complaining about upthread.  I'm not
    > >> a Python user, but from what I can gather of the 2-to-3 changes,
    > >> having to choose one at package build time is going to be a disaster.
    > 
    > > Agreed. We really need to have a plpython and plpython3.
    > 
    > Peter was concerned about duplicative maintenance effort, but what I
    > think this patch shows is that (at least for the near future) both
    > could be built from a single source file.  What we need is configure
    > and makefile support to do that.
    
    Ahh, so we would have:
    
    --enable-plpython2=/usr/bin/python2
    --enable-plpython3=/usr/bin/python3
    
    ?
    
    That seems reasonable if we can run both. Although I wonder if longer
    term (2.x is going to be support a long time) we will end up with
    frustration within the single source file trying to keep things
    straight.
    
    Joshua D. Drake
    
    
    > 
    > 			regards, tom lane
    > 
    
    
    -- 
    PostgreSQL.org Major Contributor
    Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
    Consulting, Training, Support, Custom Development, Engineering
    If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
    
    
    

15. Re: Python 3.1 support

    Tom Lane <tgl@sss.pgh.pa.us> — 2009-11-18T17:50:33Z

    "Joshua D. Drake" <jd@commandprompt.com> writes:
    > On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote:
    >> Peter was concerned about duplicative maintenance effort, but what I
    >> think this patch shows is that (at least for the near future) both
    >> could be built from a single source file.
    
    > That seems reasonable if we can run both. Although I wonder if longer
    > term (2.x is going to be support a long time) we will end up with
    > frustration within the single source file trying to keep things
    > straight.
    
    Once it gets to the point where it's more trouble to keep them together
    than not, we can split the source.  But judging from this patch, a
    single source file is the ticket for the moment.
    
    			regards, tom lane
    
    

16. Re: Python 3.1 support

    Nathan Boley <npboley@gmail.com> — 2009-11-18T19:32:44Z

    > Here's the patch to support Python >=3.1 with PL/Python.  The
    > compatibility code is mostly in line with the usual 2->3 C porting
    > practice and is documented inline.
    
    
    I took a cursory look at this patch and, while the logic seems sound
    and roughly in line with the suggested python porting procedure, I'm
    not quite certain what this implies for potential future patches.
    
    For instance, if I wanted to write a type converter for bytea -> the
    python 3 byte type would the expectation be that I ensure that it
    works in Python 2? Or is an ifdef that ignores it in the case of
    Python 2 OK, and we can just put a note in the docs.
    
    Also, how far back do we want to maintain 2.x compatibility? 2.0? If I
    wanted to submit a patch that makes use of the list sort method, do I
    need to ensure that it can either use the cmp arguments or a key
    argument?
    
    What if I wanted to implement a set returning function that made use
    of an iterators next() method. Would I just put ifdefs around the code
    or a preprocessor definition that defines NEXT as next() for Python
    2.x and __next__() for 3.x?
    
    I guess that my first impression is that Python broke compatibility
    for a reason, and that either plpython can't evolve, or it will
    quickly become impossible to maintain. That being said, I mostly buy
    the maintenance arguments from the previous discussion, but if we want
    to have plpython and plpython3, a bunch of defines and ifdefs does not
    seem like the best way to do this.
    
    Would a better approach be to maintain compatibility layer? ie
    plython_compat.h/c
    plython2.c
    plython3.c
    
    Then patches that apply to a python3 can be applied to plython3.c and
    any changed function can be ripped out of plython_compat and moved
    into plpython2.
    
    I'm sorry to snipe from the sidelines like this. If we didn't expect
    plpython to evolve then this patch seems like the correct approach,
    but there is clearly some desire to expand plpython and following this
    path seems like it will end in a much more painful split in the future
    or a necessary rewrite.
    
    If there is some consensus that this is the best approach, then I will
    do a more comprehensive review.
    
    -Nathan
    
    

17. Re: Python 3.1 support

    James William Pye <lists@jwp.name> — 2009-11-18T20:36:06Z

    On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote:
    > The question is whether it helps the user, not the implementer.
    
    Sure, but do you have a patch waiting to implement tracebacks?
    
    I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without some significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack.
    
    What helped the implementer here would help the user.
    
    >  As far
    > as I can tell, it just creates more typing for no benefit whatsoever.
    
    "def main(*args):" is annoying, but not entirely lamentable...
    It's explicit, as well(no need to document munging that occurs behind the scenes).
    
    Also, compare the cases where you need to cache some initialized data:
    
    if 'key' not in SD:
     ...
     SD['key'] = my_newly_initialized_data
    ...
    
    
    With function modules, SD is not needed as you have your module globals to keep your locally cached data in:
    
    ...
    data = my_newly_initialized_data
    
    def main(*args):
     ...
    
    
    > Also, it's inconsistent with normal Python script files
    
    Hinges on whether "normal" is actually normal.
    I often use the __name__ convention in script files myself:
    
    if __name__ == '__main__':
     main(...)
    
    That is, using that convention, the script can be import'd and used without executing the "script functionality". (It has proven to be very handy a few times now)
    
    >  and with other PLs.
    
    I don't understand why that's a significant enough interest to note.
    
    > I don't need another PostgreSQL implementation on top of Python.
    
    Indeed, and I do understand that. That is, I have removed some features with that very thought in mind. (OTOH, I consider the date_part properties on datetime types to be special: too likely useful.)
    
    [tho, "PostgreSQL implementation"? I think I understand what you were getting at, but..]
    
    > The maintenance effort required to keep those two consistent aside.
    
    I don't think there are many consistency issues here.
    What did you have in mind?
    
    > Again, I'm only one user.  But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing.
    
    Agreed. It would seem quite doomed.
    
    At this point, I'm not going to try getting it into PG. (apparent futility and such)
    

18. Re: Python 3.1 support

    James William Pye <lists@jwp.name> — 2009-11-19T01:20:54Z

    On Nov 18, 2009, at 1:36 PM, James Pye wrote:
    > At this point, I'm not going to try getting it into PG. (apparent futility and such)
    
    ugh, on second thought, I think I've written a bit too much code to stop now. I'm going to get plpython3 as far as I can and submit it to the next commitfest.
    

19. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-19T10:05:42Z

    On ons, 2009-11-18 at 09:48 -0800, Joshua D. Drake wrote:
    > Although I wonder if longer
    > term (2.x is going to be support a long time) we will end up with
    > frustration within the single source file trying to keep things
    > straight.
    
    There are five million Python modules with C code out there with the
    same problem.  Considerable effort has been put in by Python upstream to
    make the effort manageable.  No one in their right mind is going to
    create two separate source files just because in the future the mythical
    differences will be too big, when clearly the effort is going into a
    direction to reduce the differences.
    
    If you look into the source file, there is already special code for
    Python 2.2, 2.3, 2.4, 2.5, 2.6, and now 3.1.  The chunk for 3.1 is a bit
    bigger, but only a bit, and well, that's why it's 3.x and not 2.x.  No
    one has ever suggested, we might need to support Python 2.2 for a long
    time, let's create a separate source file.
    
    I agree, there will probably need to be some configuration/build support
    on top of this, but that's something we should work out independently of
    how to manage the source file.
    
    
    

20. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-19T10:12:28Z

    On ons, 2009-11-18 at 08:43 -0800, Nathan Boley wrote:
    > > Again, I'm only one user.  But so far I haven't seen anyone else speak
    > > up here, and clearly accepting this for inclusion will need nontrivial
    > > convincing.
    > 
    > Well, FWIW, I am excited about better type integration.
    
    Let's clarify, as there are two different models being proposed here.
    The first approach, which is currently implemented (and some patches
    pending), is to convert a PostgreSQL type to the "nearest" Python type.
    For example, text to string, int to int, array to list, timestamp to
    datetime.datetime, etc.  The other approach, which is what James Pye's
    new implementation proposes (as I understand it), is to convert
    PostgreSQL types into specially made Python objects, such as
    Postgres.types.record or Postgres.types.timestamp.
    
    > Also, I am a little skeptical about this patch. I am sorry if this has
    > already been discussed, but would this mean that I need to choose
    > whether pl/python is built against Python 2.* or Python 3.*?
    
    Yeah, see later discussion about how to resolve this.  But I think in
    practice, unless you use lots of print statements in your stored
    procedures (?!?), this problem is exaggerated.
    
    
    

21. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-19T12:00:51Z

    On ons, 2009-11-18 at 11:32 -0800, Nathan Boley wrote:
    > I took a cursory look at this patch and, while the logic seems sound
    > and roughly in line with the suggested python porting procedure, I'm
    > not quite certain what this implies for potential future patches.
    > 
    > For instance, if I wanted to write a type converter for bytea -> the
    > python 3 byte type would the expectation be that I ensure that it
    > works in Python 2? Or is an ifdef that ignores it in the case of
    > Python 2 OK, and we can just put a note in the docs.
    
    Note that this is already implemented.  The main point of the patch is
    to provide a small compatibility layer so that these kinds of issues are
    practically nonexistent.  The fact that you didn't notice might prove
    that the patch does its job. ;-)
    
    > Also, how far back do we want to maintain 2.x compatibility? 2.0?
    
    We handle this on an ad hoc basis.  We currently support Python 2.2 and
    later, and this cutoff exists -- this is my interpretation of history --
    because 2.2 introduced iterators and no one bothered(?) to put ifdefs
    around the code in PL/Python that provides iterator support.  Over the
    years, we will probably drop support for other older Python versions,
    but there is no process or plan for that.  Right now, the support for
    Python 2.2 is about three lines, so it's not a bother, but when someone
    comes and implements a major feature that, say, requires Python 2.3, we
    can probably drop 2.2.  But when the major feature requires 2.6, we
    probably don't want to drop 2.5 quite yet at this time.  It's a judgment
    call.
    
    > If I
    > wanted to submit a patch that makes use of the list sort method, do I
    > need to ensure that it can either use the cmp arguments or a key
    > argument?
    
    Any patch one is likely to submit will be a C patch, not a Python patch.
    But anyway, the "key" argument was introduced in Python 2.4, and so we'd
    have to come to a decision in the community about whether Python 2.3
    support is worth keeping versus the value of that new feature.  See above.
    
    But anyway, this problem has nothing to do with my patch; it has already
    existed in the same form forever.
    
    > What if I wanted to implement a set returning function that made use
    > of an iterators next() method. Would I just put ifdefs around the code
    > or a preprocessor definition that defines NEXT as next() for Python
    > 2.x and __next__() for 3.x?
    
    Again, you would likely submit a C patch, and the iterator API is the
    same between 2.x and 3.x.
    
    > I guess that my first impression is that Python broke compatibility
    > for a reason, and that either plpython can't evolve, or it will
    > quickly become impossible to maintain.
    
    I think this is an exaggeration of reality.  Python 3 removed deprecated
    features.  There is a perfectly good migration path that covers most
    code: Switch to Python 2.6, switch to the new features, remove the old
    features, switch to Python 3.x.  This applies both on the Python and the
    C level.  They did not break compatibility with the intention of making
    every module author out there reimplement their thing from scratch.
    Otherwise Python 2.6 would make very little sense at all.
    
    Take a look at an example closer to home: PostgreSQL breaks C API
    compatibility in almost every major release.  We do this to remove cruft
    and support new features.  The intent is not to make Slony and PostGIS
    and all the other modules reimplement their product from scratch every
    time.  They put in a few ifdefs, sometimes they complain about it ;-),
    and then the problem is solved.
    
    > That being said, I mostly buy
    > the maintenance arguments from the previous discussion, but if we want
    > to have plpython and plpython3, a bunch of defines and ifdefs does not
    > seem like the best way to do this.
    
    These ifdefs were not my idea.  They are in some cases directly and in
    some cases in spirit from the Python 2.6 header files, so they are the
    official way to do this.
    
    > Would a better approach be to maintain compatibility layer? ie
    > plython_compat.h/c
    > plython2.c
    > plython3.c
    > 
    > Then patches that apply to a python3 can be applied to plython3.c and
    > any changed function can be ripped out of plython_compat and moved
    > into plpython2.
    
    As I tried to explain above, we have always had a rolling feature model
    of sorts, even across various Python 2.x versions.  If you want to try
    it out, you could take the current source and split it up into
    plpython22.c, plpython23.c, etc. and see if that becomes useful.
    
    
    
    

22. Re: Python 3.1 support

    James William Pye <lists@jwp.name> — 2009-11-19T17:01:48Z

    On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote:
    > The other approach, which is what James Pye's
    > new implementation proposes (as I understand it), is to convert
    > PostgreSQL types into specially made Python objects, such as
    > Postgres.types.record or Postgres.types.timestamp.
    
    Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum). So more like "copied into Python memory", and associated with its respective type. Wrapped in a Python object?
    
    One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's no type I/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it could be quite beneficial.
    

23. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-19T18:21:01Z

    On ons, 2009-11-18 at 12:28 -0500, Tom Lane wrote:
    > "Joshua D. Drake" <jd@commandprompt.com> writes:
    > > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
    > >> Yes.  That's exactly what I was complaining about upthread.  I'm not
    > >> a Python user, but from what I can gather of the 2-to-3 changes,
    > >> having to choose one at package build time is going to be a disaster.
    > 
    > > Agreed. We really need to have a plpython and plpython3.
    > 
    > Peter was concerned about duplicative maintenance effort, but what I
    > think this patch shows is that (at least for the near future) both
    > could be built from a single source file.  What we need is configure
    > and makefile support to do that.
    
    By the way, it occurred to me that having two different versions of
    libpython loaded into the same process is probably not going to work
    sanely.  So whatever solution we come up with for the Python 3
    transition, the possibilities for a jolly back-and-forth are probably
    going to be quite limited.
    
    
    
    

24. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-19T18:32:58Z

    On ons, 2009-11-18 at 13:36 -0700, James Pye wrote:
    > On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote:
    > > The question is whether it helps the user, not the implementer.
    > 
    > Sure, but do you have a patch waiting to implement tracebacks?
    > 
    > I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without some significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack.
    > 
    > What helped the implementer here would help the user.
    
    But you wouldn't, for example, get away with breaking SQL (or even
    improving it incompatibly) to facilitate a better elog.
    
    > >  As far
    > > as I can tell, it just creates more typing for no benefit whatsoever.
    > 
    > "def main(*args):" is annoying, but not entirely lamentable...
    > It's explicit, as well(no need to document munging that occurs behind the scenes).
    > 
    > Also, compare the cases where you need to cache some initialized data:
    > 
    > if 'key' not in SD:
    >  ...
    >  SD['key'] = my_newly_initialized_data
    > ...
    > 
    > 
    > With function modules, SD is not needed as you have your module globals to keep your locally cached data in:
    > 
    > ...
    > data = my_newly_initialized_data
    > 
    > def main(*args):
    >  ...
    
    I can see that this creates other options for structuring code, but it
    doesn't actually match my way of thinking.  (Obviously, I'm biased, but
    anyway.)  I think of a PL/Python function as a Python script file stored
    in the database.  When you call it, arguments are passed just like a
    Python script receives arguments from the shell.  When Python scripts
    want to share data, they might use a file (or perhaps a database server
    in advanced cases) and do
    
    if not file exists:
        create the file
        fill it with data
    
    This is in my mind quite analogous to how the SD business works.
    
    The analogy to your approach, as I understand it, would be that multiple
    instances of the same script file will automatically share their global
    variables.  That could be quite interesting, actually, but it's not how
    it works, and in most cases it's better that way.
    
    
    
    

25. Re: Python 3.1 support

    Tom Lane <tgl@sss.pgh.pa.us> — 2009-11-19T18:43:15Z

    Peter Eisentraut <peter_e@gmx.net> writes:
    > By the way, it occurred to me that having two different versions of
    > libpython loaded into the same process is probably not going to work
    > sanely.
    
    Why not?  There's no way they'd even know about each other.  We tell
    the loader not to make the symbols globally visible.
    
    But in any case, my main concern here is that I don't want to have
    to predetermine which python version a user of Red Hat/Fedora will
    have to use.  If they can only use one at a time, that's still a
    good bit better than not having a choice at all.
    
    			regards, tom lane
    
    

26. Re: Python 3.1 support

    James William Pye <lists@jwp.name> — 2009-11-19T20:12:34Z

    On Nov 19, 2009, at 11:32 AM, Peter Eisentraut wrote:
    > But you wouldn't, for example, get away with breaking SQL (or even
    > improving it incompatibly) to facilitate a better elog.
    
    This doesn't fit the situation.
    
    I'm not breaking PL/Python. I'm trying to add PL/Python3. =)
    
    > I think of a PL/Python function as a Python script file stored
    > in the database.
    
    For Python, I think that's a mistake. Python scripts are independent applications.
    
    [tho, I think this does illuminate our perspectives...]
    

27. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-20T06:04:43Z

    On tor, 2009-11-19 at 13:43 -0500, Tom Lane wrote:
    > But in any case, my main concern here is that I don't want to have
    > to predetermine which python version a user of Red Hat/Fedora will
    > have to use.  If they can only use one at a time, that's still a
    > good bit better than not having a choice at all.
    
    By the way, mod_wsgi supports Python 3 already (same patch as here, in
    principle).  From the Fedora wiki page, I gather that no one has really
    looked into packaging that yet for Python 3, but if someone does, maybe
    we can cross-inspire ourselves.
    
    
    

28. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-20T07:02:43Z

    On tor, 2009-11-19 at 13:12 -0700, James Pye wrote:
    > > I think of a PL/Python function as a Python script file stored
    > > in the database.
    > 
    > For Python, I think that's a mistake. Python scripts are independent applications.
    
    Is there any precedent for the sort of behavior that you are
    implementing, that is, automatic sharing of variables between
    independent executions of the same source container?
    
    
    
    

29. Re: Python 3.1 support

    James William Pye <lists@jwp.name> — 2009-11-20T08:20:04Z

    On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote:
    > Is there any precedent for the sort of behavior that you are
    > implementing, that is, automatic sharing of variables between
    > independent executions of the same source container?
    
    import foo
    
    # bar is a regular, def'd function.
    foo.bar()
    
    ...
    
    # even in another thread, doesn't matter..
    foo.bar()
    
    
    In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary).
    
    A plpython3 function *is* a Python module.
    

30. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-11-20T08:26:45Z

    On fre, 2009-11-20 at 01:20 -0700, James Pye wrote:
    > On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote:
    > > Is there any precedent for the sort of behavior that you are
    > > implementing, that is, automatic sharing of variables between
    > > independent executions of the same source container?
    > 
    > import foo
    > 
    > # bar is a regular, def'd function.
    > foo.bar()
    > 
    > ...
    > 
    > # even in another thread, doesn't matter..
    > foo.bar()
    > 
    > 
    > In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary).
    
    That's not what I meant, because this is the same execution of the same
    source container, with threads explicitly started somewhere.  You could
    do the same in a plpython function (in theory, at least).
    
    What I mean is more like, you execute the same source file twice in a
    row, and the global variables are saved for the second run.
    
    
    

31. Re: Python 3.1 support

    James William Pye <lists@jwp.name> — 2009-11-20T08:59:56Z

    On Nov 20, 2009, at 1:26 AM, Peter Eisentraut wrote:
    > because this is the same execution
    
    Hrm, not necessarily. foo could be imported by another, completely independent part of the program. foo is cached in sys.modules. bar() is executed and it's still the same globals(). shared.
    

32. Re: Python 3.1 support

    Tino Wildenhain <tino@wildenhain.de> — 2009-11-20T09:30:05Z

    Am 19.11.2009 18:01, schrieb James Pye:
    > On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote:
    >> The other approach, which is what James Pye's
    >> new implementation proposes (as I understand it), is to convert
    >> PostgreSQL types into specially made Python objects, such as
    >> Postgres.types.record or Postgres.types.timestamp.
    >
    > Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum). So more like "copied into Python memory", and associated with its respective type. Wrapped in a Python object?
    
    Yes "wrapped" is the term commonly used for that. And I must say I like 
    it and I used plpy where I could.
    
    > One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's no type I/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it could be quite beneficial.
    
    Exactly and you have all the meta information about the original 
    postgres type. IIRC there were some thoughts of having something like 
    that in a DBAPI interface as well (similar for example to cx_Oracle).
    
    Tino
    
    

33. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-12-10T23:19:21Z

    On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
    > There was considerable debate earlier about whether we wanted to treat
    > Python 3 as a separate PL so it could be available in parallel with
    > plpython 2, because of the user-level coding incompatibilities.  It
    > looks like this patch simply ignores that problem.  What is going to
    > happen to plpython functions that depend on 2.x behavior?
    
    I have a proposal for how to handle this, and a prototype patch
    attached.  This follows essentially what the CPython distribution itself
    does, which will make this tolerably easy to follow for users.
    
    We install plpython as plpython2.so or plpython3.so, depending on the
    version used to build it.  Then, plpython.so is a symlink to
    plpython2.so.
    
    We then create three language definition templates:
    
    plpythonu -> plpython.so
    plpython2u -> plpython2.so
    plpython3u -> plpython3.so
    
    In the far future we flip the default symlink to plpython3.so, maybe in
    about 5 years when Python 2.x expires.
    
    This gives the users the following options and scenarios:
    
    - Existing users don't have to do anything, until maybe in five years
    they will notice that their OS doesn't ship Python 2 anymore and they
    will have to act anyway.  In practice, by then they might have adjusted
    their coding style to Python 2.6/2.7 norms and their functions will
    migrate to 3.x without change.
    
    - Users who know that they have heavily Python 2.x dependent code and
    don't want to ever change it can make use of the plpython2u language
    name, just like they should probably change their scripts to use
    something like #!/usr/bin/python2.
    
    - Users who want to dive into Python 3 can use the plpython3u language
    name, which will basically keep working forever by today's standards.
    Those users would probably also use #!/usr/bin/python3 or the like in
    their scripts.  In the far future they might like to remove the "3".
    
    - Daredevils can change symlink manually and make plpython3.so the
    default plpythonu implementation.  Those people would probably also
    make /usr/bin/python be version 3.
    
    Comments?
    

34. Re: Python 3.1 support

    Joshua D. Drake <jd@commandprompt.com> — 2009-12-11T17:59:27Z

    On Fri, 2009-12-11 at 01:19 +0200, Peter Eisentraut wrote:
    > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
    > > There was considerable debate earlier about whether we wanted to treat
    > > Python 3 as a separate PL so it could be available in parallel with
    > > plpython 2, because of the user-level coding incompatibilities.  It
    > > looks like this patch simply ignores that problem.  What is going to
    > > happen to plpython functions that depend on 2.x behavior?
    > 
    > I have a proposal for how to handle this, and a prototype patch
    > attached.  This follows essentially what the CPython distribution itself
    > does, which will make this tolerably easy to follow for users.
    > 
    > We install plpython as plpython2.so or plpython3.so, depending on the
    > version used to build it.  Then, plpython.so is a symlink to
    > plpython2.so.
    > 
    > We then create three language definition templates:
    > 
    > plpythonu -> plpython.so
    > plpython2u -> plpython2.so
    > plpython3u -> plpython3.so
    > 
    
    > 
    > Comments?
    
    Well as a Python guy... makes sense to me :)
    
    Joshua D. Drake
    
    
    -- 
    PostgreSQL.org Major Contributor
    Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
    Consulting, Training, Support, Custom Development, Engineering
    If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
    
    
    

35. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-12-14T18:42:12Z

    I wrote:
    > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
    > > There was considerable debate earlier about whether we wanted to treat
    > > Python 3 as a separate PL so it could be available in parallel with
    > > plpython 2, because of the user-level coding incompatibilities.  It
    > > looks like this patch simply ignores that problem.  What is going to
    > > happen to plpython functions that depend on 2.x behavior?
    > 
    > I have a proposal for how to handle this, and a prototype patch
    > attached.  This follows essentially what the CPython distribution itself
    > does, which will make this tolerably easy to follow for users.
    > 
    > We install plpython as plpython2.so or plpython3.so, depending on the
    > version used to build it.  Then, plpython.so is a symlink to
    > plpython2.so.
    
    So here is the potentially final patch for this, including the original
    port of plpython.c itself, build system adjustments, and documentation.
    
    
    

36. Re: Python 3.1 support

    Robert Haas <robertmhaas@gmail.com> — 2009-12-14T18:47:34Z

    On Mon, Dec 14, 2009 at 1:42 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
    > I wrote:
    >> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
    >> > There was considerable debate earlier about whether we wanted to treat
    >> > Python 3 as a separate PL so it could be available in parallel with
    >> > plpython 2, because of the user-level coding incompatibilities.  It
    >> > looks like this patch simply ignores that problem.  What is going to
    >> > happen to plpython functions that depend on 2.x behavior?
    >>
    >> I have a proposal for how to handle this, and a prototype patch
    >> attached.  This follows essentially what the CPython distribution itself
    >> does, which will make this tolerably easy to follow for users.
    >>
    >> We install plpython as plpython2.so or plpython3.so, depending on the
    >> version used to build it.  Then, plpython.so is a symlink to
    >> plpython2.so.
    >
    > So here is the potentially final patch for this, including the original
    > port of plpython.c itself, build system adjustments, and documentation.
    
    I think you forgot to actually attach it...
    
    ...Robert
    
    

37. Re: Python 3.1 support

    Peter Eisentraut <peter_e@gmx.net> — 2009-12-14T18:50:06Z

    I wrote:
    > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
    > > There was considerable debate earlier about whether we wanted to treat
    > > Python 3 as a separate PL so it could be available in parallel with
    > > plpython 2, because of the user-level coding incompatibilities.  It
    > > looks like this patch simply ignores that problem.  What is going to
    > > happen to plpython functions that depend on 2.x behavior?
    > 
    > I have a proposal for how to handle this, and a prototype patch
    > attached.  This follows essentially what the CPython distribution itself
    > does, which will make this tolerably easy to follow for users.
    > 
    > We install plpython as plpython2.so or plpython3.so, depending on the
    > version used to build it.  Then, plpython.so is a symlink to
    > plpython2.so.
    
    So here is the potentially final patch for this, including the original
    port of plpython.c itself, build system adjustments, and documentation.
    
    Really attached this time.