Horton

Thread

Commits

  1. Enable Python Limited API for PL/Python on MSVC

  2. Activate Python "Limited API" in PL/Python

  3. Prepare for Python "Limited API" in PL/Python

  4. Remove obsolete Python version check

  1. Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2024-12-02T08:51:17Z

    This patch changes PL/Python to use the Python "limited API". This API 
has stronger ABI stability guarantees.[0] This means, you can build 
PL/Python against any Python 3.x version and use any other Python 3.x 
version at run time.

This is especially useful for binary packages where the operating system 
does not come with a fixed suitable version of Python. For example, 
Postgres.app (for macOS) would prefer to link against the Python version 
supplied by python.org (Python.app). But that has a 3.x version that 
changes over time. So instead they bundle a Python version inside 
Postgres.app. The Windows installer used to also bundle Python but as of 
PG17 you have to get it yourself, but you have to get a very specific 
version [1], which is unsatisfactory. This patch fixes that: You can use 
any Python version independent of what PL/Python was built against. 
(There is a mechanism to say "at least 3.N", but for this patch, we 
don't need that, we can stick with the current minimum of 3.2.)

(I have only tested the macOS side of this, not the Windows side. In 
fact, the patch currently doesn't build on Windows on CI. I haven't 
figured out why.)

For Linux-style packaging, I don't think this would have any benefit for 
users right now, since the OS comes with a Python installation and all 
the packages are built against that. But it could potentially be helpful 
for packagers. For example, on Debian, this could detach the postgresql 
packages from python version transitions. But AFAICT, the Python 
packaging layout is not prepared for that. (There are only 
libpython3.x.so libraries, no libpython3.so that one would have to link 
against.)

Finally, I think this patch is part of a path toward making PL/Python 
thread-safe. I don't think the patch by itself changes anything, but if 
you read through [2], using heap types is part of the things mentioned 
there.

[0]: https://docs.python.org/3/c-api/stable.html
[1]: 
https://github.com/EnterpriseDB/edb-installers/blob/REL-17/server/resources/installation-notes.html#L34-L36
[2]: https://docs.python.org/3/howto/isolating-extensions.html

  2. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2025-01-07T07:34:39Z

    On 02.12.24 09:51, Peter Eisentraut wrote:
> This patch changes PL/Python to use the Python "limited API". This API 
> has stronger ABI stability guarantees.[0] This means, you can build PL/ 
> Python against any Python 3.x version and use any other Python 3.x 
> version at run time.
> 
> This is especially useful for binary packages where the operating system 
> does not come with a fixed suitable version of Python. For example, 
> Postgres.app (for macOS) would prefer to link against the Python version 
> supplied by python.org (Python.app). But that has a 3.x version that 
> changes over time. So instead they bundle a Python version inside 
> Postgres.app. The Windows installer used to also bundle Python but as of 
> PG17 you have to get it yourself, but you have to get a very specific 
> version [1], which is unsatisfactory. This patch fixes that: You can use 
> any Python version independent of what PL/Python was built against. 
> (There is a mechanism to say "at least 3.N", but for this patch, we 
> don't need that, we can stick with the current minimum of 3.2.)
> 
> (I have only tested the macOS side of this, not the Windows side. In 
> fact, the patch currently doesn't build on Windows on CI. I haven't 
> figured out why.)

A bit more exploration of that Windows build failure:

This patch changes it so that, on Windows, plpython is linked against a 
library called "python3.lib" instead of previously "python311.lib" or 
similar.  The build failure is [0]

     LINK : fatal error LNK1104: cannot open file 'python3.lib'

[0]: https://cirrus-ci.com/task/5738515637469184

which suggests that the file isn't there.  But we have tested this 
internally and the build succeeded locally.  I also don't see anything 
in the Python installer code or documentation that suggests that there 
is an option not to install that file or something like that.  It should 
be installed right next to the python3XX.lib file.

How could one debug this?  Is there a way to log into the CI image or 
get a directory dump or something like that?

  3. Re: Use Python "Limited API" in PL/Python

    Jakob Egger <jakob@eggerapps.at> — 2025-01-14T15:51:05Z

    On 07.01.2025, at 08:34, Peter Eisentraut <peter@eisentraut.org> wrote:
> On 02.12.24 09:51, Peter Eisentraut wrote:
>> This patch changes PL/Python to use the Python "limited API". This API has stronger ABI stability guarantees.[0] This means, you can build PL/ Python against any Python 3.x version and use any other Python 3.x version at run time.
>> This is especially useful for binary packages where the operating system does not come with a fixed suitable version of Python. For example, Postgres.app (for macOS) would prefer to link against the Python version supplied by python.org (Python.app). But that has a 3.x version that changes over time. So instead they bundle a Python version inside Postgres.app. The Windows installer used to also bundle Python but as of PG17 you have to get it yourself, but you have to get a very specific version [1], which is unsatisfactory. This patch fixes that: You can use any Python version independent of what PL/Python was built against. (There is a mechanism to say "at least 3.N", but for this patch, we don't need that, we can stick with the current minimum of 3.2.)
>> (I have only tested the macOS side of this, not the Windows side. In fact, the patch currently doesn't build on Windows on CI. I haven't figured out why.)
> 
> A bit more exploration of that Windows build failure:
> 
> This patch changes it so that, on Windows, plpython is linked against a library called "python3.lib" instead of previously "python311.lib" or similar.  The build failure is [0]
> 
>    LINK : fatal error LNK1104: cannot open file 'python3.lib'
> 
> [0]: https://cirrus-ci.com/task/5738515637469184
> 
> which suggests that the file isn't there.  But we have tested this internally and the build succeeded locally.  I also don't see anything in the Python installer code or documentation that suggests that there is an option not to install that file or something like that.  It should be installed right next to the python3XX.lib file.


I think the problem is that meson doesn't know we want to use the limited API.
I found this related issue on Github: https://github.com/mesonbuild/meson/issues/13824

Someone suggests to change the Python dependency in the meson.build file like this:

py_dep = declare_dependency(
    dependencies: py_dep,
    compile_args: '-DPy_LIMITED_API='0x030100',
)

I've tried to create a patch with this change. I'm attaching it to this message so that cfbot picks it up. (I was unable to reproduce the issue locally)


Best regards,
Jakob



  4. Re: Use Python "Limited API" in PL/Python

    Jakob Egger <jakob@eggerapps.at> — 2025-01-15T11:28:22Z

    On 14.01.2025, at 16:51, Jakob Egger <jakob@eggerapps.at> wrote:
> I've tried to create a patch with this change. I'm attaching it to this message so that cfbot picks it up. (I was unable to reproduce the issue locally)

Apologies, please disregard my last patch. It does not work.

It looks like meson just can't build libraries that link with Python that use the limited API.

My understanding of Meson is limited, so I am unsure how to best work around this limitation.

I've spent two days trying to figure it out; I'm at the end of my wits.

Best regards,
Jakob

  5. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2025-01-15T22:20:58Z

    On 15.01.25 12:28, Jakob Egger wrote:
> On 14.01.2025, at 16:51, Jakob Egger <jakob@eggerapps.at> wrote:
>> I've tried to create a patch with this change. I'm attaching it to 
>> this message so that cfbot picks it up. (I was unable to reproduce the 
>> issue locally)
> 
> Apologies, please disregard my last patch. It does not work.
> 
> It looks like meson just can't build libraries that link with Python 
> that use the limited API.
> 
> My understanding of Meson is limited, so I am unsure how to best work 
> around this limitation.
> 
> I've spent two days trying to figure it out; I'm at the end of my wits.

Thanks for checking.  I've also been poking at this, without success.

What we could do is just not activate the limited API on Windows 
(actually MSVC, since it works on MinGW).  If someone later finds a way 
to make it work on Windows/MSVC, then it's easy to change.

So here is an updated patch set with that change.  It should pass cfbot 
like this.

  6. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2025-03-03T10:17:48Z

    On 15.01.25 23:20, Peter Eisentraut wrote:
> On 15.01.25 12:28, Jakob Egger wrote:
>> On 14.01.2025, at 16:51, Jakob Egger <jakob@eggerapps.at> wrote:
>>> I've tried to create a patch with this change. I'm attaching it to 
>>> this message so that cfbot picks it up. (I was unable to reproduce 
>>> the issue locally)
>>
>> Apologies, please disregard my last patch. It does not work.
>>
>> It looks like meson just can't build libraries that link with Python 
>> that use the limited API.
>>
>> My understanding of Meson is limited, so I am unsure how to best work 
>> around this limitation.
>>
>> I've spent two days trying to figure it out; I'm at the end of my wits.
> 
> Thanks for checking.  I've also been poking at this, without success.
> 
> What we could do is just not activate the limited API on Windows 
> (actually MSVC, since it works on MinGW).  If someone later finds a way 
> to make it work on Windows/MSVC, then it's easy to change.
> 
> So here is an updated patch set with that change.  It should pass cfbot 
> like this.

Update for the hackers list: This patch set was briefly committed but 
had to be reverted because it crashed on some older Python versions; see 
[0].

I have locally reproduced the problem with Python 3.6 and 3.7; Python 
3.8 is ok.  This matches the results from the buildfarm.

I have poked at this a bit more but haven't gotten a good idea where or 
how to fix it so far.  Help welcome.


[0]: 
https://www.postgresql.org/message-id/flat/E1tnJNu-000CZW-0X%40gemulon.postgresql.org

  7. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2025-03-05T16:40:22Z

    On 03.03.25 11:17, Peter Eisentraut wrote:
> Update for the hackers list: This patch set was briefly committed but 
> had to be reverted because it crashed on some older Python versions; see 
> [0].
> 
> I have locally reproduced the problem with Python 3.6 and 3.7; Python 
> 3.8 is ok.  This matches the results from the buildfarm.
> 
> I have poked at this a bit more but haven't gotten a good idea where or 
> how to fix it so far.  Help welcome.

I have figured this out.  There was a Python API change/bugfix between 
3.7 and 3.8 that directly affects this patch.  The relevant commit is 
<https://github.com/python/cpython/commit/364f0b0f19c>.  I have applied 
the workarounds described there to my patch set, and now it works for 
3.6 and 3.7 as well.

  8. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2025-03-14T08:41:29Z

    On 05.03.25 17:40, Peter Eisentraut wrote:
> On 03.03.25 11:17, Peter Eisentraut wrote:
>> Update for the hackers list: This patch set was briefly committed but 
>> had to be reverted because it crashed on some older Python versions; 
>> see [0].
>>
>> I have locally reproduced the problem with Python 3.6 and 3.7; Python 
>> 3.8 is ok.  This matches the results from the buildfarm.
>>
>> I have poked at this a bit more but haven't gotten a good idea where 
>> or how to fix it so far.  Help welcome.
> 
> I have figured this out.  There was a Python API change/bugfix between 
> 3.7 and 3.8 that directly affects this patch.  The relevant commit is 
> <https://github.com/python/cpython/commit/364f0b0f19c>.  I have applied 
> the workarounds described there to my patch set, and now it works for 
> 3.6 and 3.7 as well.

This has all been committed now.

  9. Re: Use Python "Limited API" in PL/Python

    vignesh C <vignesh21@gmail.com> — 2025-03-17T06:08:08Z

    On Fri, 14 Mar 2025 at 14:11, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 05.03.25 17:40, Peter Eisentraut wrote:
> > On 03.03.25 11:17, Peter Eisentraut wrote:
> >> Update for the hackers list: This patch set was briefly committed but
> >> had to be reverted because it crashed on some older Python versions;
> >> see [0].
> >>
> >> I have locally reproduced the problem with Python 3.6 and 3.7; Python
> >> 3.8 is ok.  This matches the results from the buildfarm.
> >>
> >> I have poked at this a bit more but haven't gotten a good idea where
> >> or how to fix it so far.  Help welcome.
> >
> > I have figured this out.  There was a Python API change/bugfix between
> > 3.7 and 3.8 that directly affects this patch.  The relevant commit is
> > <https://github.com/python/cpython/commit/364f0b0f19c>.  I have applied
> > the workarounds described there to my patch set, and now it works for
> > 3.6 and 3.7 as well.
>
> This has all been committed now.

I felt we can mark this commitfest entry at [1] to commited now.
[1] - https://commitfest.postgresql.org/patch/5416/

Regards,
Vignesh

  10. Re: Use Python "Limited API" in PL/Python

    Bryan Green <dbryan.green@gmail.com> — 2026-01-02T17:40:11Z

    On 1/15/2025 5:28 AM, Jakob Egger wrote:
> On 14.01.2025, at 16:51, Jakob Egger <jakob@eggerapps.at> wrote:
>> I've tried to create a patch with this change. I'm attaching it to this message so that cfbot picks it up. (I was unable to reproduce the issue locally)
> 
> Apologies, please disregard my last patch. It does not work.
> 
> It looks like meson just can't build libraries that link with Python that use the limited API.
> 
> My understanding of Meson is limited, so I am unsure how to best work around this limitation.
> 
> I've spent two days trying to figure it out; I'm at the end of my wits.
> 
> Best regards,
> Jakob
Hi Jakob and Peter,

I've been working on enabling the Python Limited API on MSVC, which was
disabled in the recent commit due to build failures.

The issue is that Meson doesn't automatically link against python3.lib
when using the Limited API. Meson's Python dependency always returns the
version-specific library (python3XX.lib) regardless of whether
limited api is defined. This appears to be a known Meson limitation:
https://github.com/mesonbuild/meson/issues/13824

Initially, I thought that maybe the library wasn't being installed, but
that is not the case and the patch to our CI Windows MSVC image is not
needed.

I have a patch that works around this by manually constructing the
Python dependency on MSVC to use python3.lib instead of python3XX.lib,
using cc.find_library('python3') and building the dependency with the
appropriate include directory.  It passes CI when I push.  I'll see what
cfbot thinks.

Still testing, but wanted to share progress and get feedback on the
approach.

-- 
Bryan Green
EDB: https://www.enterprisedb.com

  11. Re: Use Python "Limited API" in PL/Python

    Bryan Green <dbryan.green@gmail.com> — 2026-01-02T18:12:22Z

    On 1/2/2026 11:40 AM, Bryan Green wrote:
> On 1/15/2025 5:28 AM, Jakob Egger wrote:
>> On 14.01.2025, at 16:51, Jakob Egger <jakob@eggerapps.at> wrote:
>>> I've tried to create a patch with this change. I'm attaching it to this message so that cfbot picks it up. (I was unable to reproduce the issue locally)
>>
>> Apologies, please disregard my last patch. It does not work.
>>
>> It looks like meson just can't build libraries that link with Python that use the limited API.
>>
>> My understanding of Meson is limited, so I am unsure how to best work around this limitation.
>>
>> I've spent two days trying to figure it out; I'm at the end of my wits.
>>
>> Best regards,
>> Jakob
> Hi Jakob and Peter,
> 
> I've been working on enabling the Python Limited API on MSVC, which was
> disabled in the recent commit due to build failures.
> 
> The issue is that Meson doesn't automatically link against python3.lib
> when using the Limited API. Meson's Python dependency always returns the
> version-specific library (python3XX.lib) regardless of whether
> limited api is defined. This appears to be a known Meson limitation:
> https://github.com/mesonbuild/meson/issues/13824
> 
> Initially, I thought that maybe the library wasn't being installed, but
> that is not the case and the patch to our CI Windows MSVC image is not
> needed.
> 
> I have a patch that works around this by manually constructing the
> Python dependency on MSVC to use python3.lib instead of python3XX.lib,
> using cc.find_library('python3') and building the dependency with the
> appropriate include directory.  It passes CI when I push.  I'll see what
> cfbot thinks.
> 
> Still testing, but wanted to share progress and get feedback on the
> approach.
> 

cfbot seems to be happy:
https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F6369
-- 
Bryan Green
EDB: https://www.enterprisedb.com

  12. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2026-01-05T07:53:33Z

    On 02.01.26 18:40, Bryan Green wrote:
> I have a patch that works around this by manually constructing the
> Python dependency on MSVC to use python3.lib instead of python3XX.lib,
> using cc.find_library('python3') and building the dependency with the
> appropriate include directory.  It passes CI when I push.  I'll see what
> cfbot thinks.
> 
> Still testing, but wanted to share progress and get feedback on the
> approach.

This approach seems fine to me.  I suppose one question would be whether 
it would be worth using that same approach on all platforms rather than 
just on windows/msvc.  That way we only have one code path to maintain. 
But I'm not sure how easy that would be to achieve.

  13. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2026-01-05T07:55:08Z

    On 02.01.26 19:12, Bryan Green wrote:
>> I have a patch that works around this by manually constructing the
>> Python dependency on MSVC to use python3.lib instead of python3XX.lib,
>> using cc.find_library('python3') and building the dependency with the
>> appropriate include directory.  It passes CI when I push.  I'll see what
>> cfbot thinks.
>>
>> Still testing, but wanted to share progress and get feedback on the
>> approach.
>>
> 
> cfbot seems to be happy:
> https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F6369

Another thing you could test is whether this actually makes the ABI 
stable.  So for example build PL/Python against Python 3.13 and then 
install Python 3.14 (and uninstall 3.13 just to be sure) and then start 
a server and run some PL/Python stuff.

  14. Re: Use Python "Limited API" in PL/Python

    Peter Eisentraut <peter@eisentraut.org> — 2026-01-14T12:56:46Z

    On 02.01.26 19:12, Bryan Green wrote:
> On 1/2/2026 11:40 AM, Bryan Green wrote:
>> On 1/15/2025 5:28 AM, Jakob Egger wrote:
>>> On 14.01.2025, at 16:51, Jakob Egger <jakob@eggerapps.at> wrote:
>>>> I've tried to create a patch with this change. I'm attaching it to this message so that cfbot picks it up. (I was unable to reproduce the issue locally)
>>>
>>> Apologies, please disregard my last patch. It does not work.
>>>
>>> It looks like meson just can't build libraries that link with Python that use the limited API.
>>>
>>> My understanding of Meson is limited, so I am unsure how to best work around this limitation.
>>>
>>> I've spent two days trying to figure it out; I'm at the end of my wits.
>>>
>>> Best regards,
>>> Jakob
>> Hi Jakob and Peter,
>>
>> I've been working on enabling the Python Limited API on MSVC, which was
>> disabled in the recent commit due to build failures.
>>
>> The issue is that Meson doesn't automatically link against python3.lib
>> when using the Limited API. Meson's Python dependency always returns the
>> version-specific library (python3XX.lib) regardless of whether
>> limited api is defined. This appears to be a known Meson limitation:
>> https://github.com/mesonbuild/meson/issues/13824
>>
>> Initially, I thought that maybe the library wasn't being installed, but
>> that is not the case and the patch to our CI Windows MSVC image is not
>> needed.
>>
>> I have a patch that works around this by manually constructing the
>> Python dependency on MSVC to use python3.lib instead of python3XX.lib,
>> using cc.find_library('python3') and building the dependency with the
>> appropriate include directory.  It passes CI when I push.  I'll see what
>> cfbot thinks.
>>
>> Still testing, but wanted to share progress and get feedback on the
>> approach.
>>
> 
> cfbot seems to be happy:
> https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F6369

I have committed this.  Thanks.