Thread

1. RFC: adding pytest as a supported test framework

   Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-10T18:46:00Z

   Hi all,
   
   For the v18 cycle, I would like to try to get pytest [1] in as a
   supported test driver, in addition to the current offerings.
   
   (I'm tempted to end the email there.)
   
   We had an unconference session at PGConf.dev [2] around this topic.
   There seemed to be a number of nodding heads and some growing
   momentum. So here's a thread to try to build wider consensus. If you
   have a competing or complementary test proposal in mind, heads up!
   
   == Problem Statement(s) ==
   
   1. We'd like to rerun a failing test by itself.
   
   2. It'd be helpful to see _what_ failed without poring over logs.
   
   These two got the most nodding heads of the points I presented. (#1
   received tongue-in-cheek applause.) I think most modern test
   frameworks are going to give you these things, but unfortunately we
   don't have them.
   
   Additionally,
   
   3. Many would like to use modern developer tooling during testing
   (language servers! autocomplete! debuggers! type checking!) and we
   can't right now.
   
   4. It'd be great to split apart client-side tests from server-side
   tests. Driving Postgres via psql all the time is fine for acceptance
   testing, but it becomes a big problem when you need to test how
   clients talk to servers with incompatible feature sets, or how a peer
   behaves when talking to something buggy.
   
   5. Personally, I want to implement security features test-first (for
   high code coverage and other benefits), and our Perl-based tests are
   usually too cumbersome for that.
   
   == Why pytest? ==
   
   From the small and biased sample at the unconference session, it looks
   like a number of people have independently settled on pytest in their
   own projects. In my opinion, pytest occupies a nice space where it
   solves some of the above problems for us, and it gives us plenty of
   tools to solve the other problems without too much pain.
   
   Problem 1 (rerun failing tests): One architectural roadblock to this
   in our Test::More suite is that tests depend on setup that's done by
   previous tests. pytest allows you to declare each test's setup
   requirements via pytest fixtures, letting the test runner build up the
   world exactly as it needs to be for a single isolated test. These
   fixtures may be given a "scope" so that multiple tests may share the
   same setup for performance or other reasons.
   
   Problem 2 (seeing what failed): pytest does this via assertion
   introspection and very detailed failure reporting. If you haven't seen
   this before, take a look at the pytest homepage [1]; there's an
   example of a full log.
   
   Problem 3 (modern tooling): We get this from Python's very active
   developer base.
   
   Problems 4 (splitting client and server tests) and 5 (making it easier
   to write tests first) aren't really Python- or pytest-specific, but I
   have done both quite happily in my OAuth work [3], and I've since
   adapted that suite multiple times to develop and test other proposals
   on this list, like LDAP/SCRAM, client encryption, direct SSL, and
   compression.
   
   Python's standard library has lots of power by itself, with very good
   documentation. And virtualenvs and better package tooling have made it
   much easier, IMO, to avoid the XKCD dependency tangle [4] of the
   2010s. When it comes to third-party packages, which I think we're
   probably going to want in moderation, we would still need to discuss
   supply chain safety. Python is not as mature here as, say, Go.
   
   == A Plan ==
   
   Even if everyone were on board immediately, there's a lot of work to
   do. I'd like to add pytest in a more probationary status, so we can
   iron out the inevitable wrinkles. My proposal would be:
   
   1. Commit bare-bones support in our Meson setup for running pytest, so
   everyone can kick the tires independently.
   2. Add a test for something that we can't currently exercise.
   3. Port a test from a place where the maintenance is terrible, to see
   if we can improve it.
   
   If we hate it by that point, no harm done; tear it back out. Otherwise
   we keep rolling forward.
   
   Thoughts? Suggestions?
   
   Thanks,
   --Jacob
   
   [1] https://docs.pytest.org/
   [2] https://wiki.postgresql.org/wiki/PGConf.dev_2024_Developer_Unconference#New_testing_frameworks
   [3] https://github.com/jchampio/pg-pytest-suite
   [4] https://xkcd.com/1987/
   
   
   
   

2. Re: RFC: adding pytest as a supported test framework

   Alexander Korotkov <aekorotkov@gmail.com> — 2024-06-10T19:26:16Z

   Hi!
   
   On Mon, Jun 10, 2024 at 9:46 PM Jacob Champion
   <jacob.champion@enterprisedb.com> wrote:
   > Thoughts? Suggestions?
   
   Thank you for working on this.
   Do you think you could re-use something from testgres[1] package?
   
   Links.
   1. https://github.com/postgrespro/testgres
   
   ------
   Regards,
   Alexander Korotkov
   Supabase
   
   
   
   

3. Re: RFC: adding pytest as a supported test framework

   Andres Freund <andres@anarazel.de> — 2024-06-10T20:04:11Z

   Hi,
   
   
   Just for context for the rest the email: I think we desperately need to move
   off perl for tests. The infrastructure around our testing is basically
   unmaintained and just about nobody that started doing dev stuff in the last 10
   years learned perl.
   
   
   On 2024-06-10 11:46:00 -0700, Jacob Champion wrote:
   > 4. It'd be great to split apart client-side tests from server-side
   > tests. Driving Postgres via psql all the time is fine for acceptance
   > testing, but it becomes a big problem when you need to test how
   > clients talk to servers with incompatible feature sets, or how a peer
   > behaves when talking to something buggy.
   
   That seems orthogonal to using pytest vs something else?
   
   
   > == Why pytest? ==
   > 
   > From the small and biased sample at the unconference session, it looks
   > like a number of people have independently settled on pytest in their
   > own projects. In my opinion, pytest occupies a nice space where it
   > solves some of the above problems for us, and it gives us plenty of
   > tools to solve the other problems without too much pain.
   
   We might be able to alleviate that by simply abstracting it away, but I found
   pytest's testrunner pretty painful. Oodles of options that are not very well
   documented and that often don't work because they are very specific to some
   situations, without that being explained.
   
   
   > Problem 1 (rerun failing tests): One architectural roadblock to this
   > in our Test::More suite is that tests depend on setup that's done by
   > previous tests. pytest allows you to declare each test's setup
   > requirements via pytest fixtures, letting the test runner build up the
   > world exactly as it needs to be for a single isolated test. These
   > fixtures may be given a "scope" so that multiple tests may share the
   > same setup for performance or other reasons.
   
   OTOH, that's quite likely to increase overall test times very
   significantly. Yes, sometimes that can be avoided with careful use of various
   features, but often that's hard, and IME is rarely done rigiorously.
   
   
   > Problem 2 (seeing what failed): pytest does this via assertion
   > introspection and very detailed failure reporting. If you haven't seen
   > this before, take a look at the pytest homepage [1]; there's an
   > example of a full log.
   
   That's not really different than what the perl tap test stuff allows. We
   indeed are bad at utilizing it, but I'm not sure that switching languages will
   change that.
   
   I think part of the problem is that the information about what precisely
   failed is often much harder to collect when testing multiple servers
   interacting than when doing localized unit tests.
   
   I think we ought to invest a bunch in improving that, I'd hope that a lot of
   that work would be largely independent of the language the tests are written
   in.
   
   
   > Python's standard library has lots of power by itself, with very good
   > documentation. And virtualenvs and better package tooling have made it
   > much easier, IMO, to avoid the XKCD dependency tangle [4] of the
   > 2010s.
   
   Ugh, I think this is actually python's weakest area. There's about a dozen
   package managers and "python distributions", that are at best half compatible,
   and the documentation situation around this is *awful*.
   
   
   > When it comes to third-party packages, which I think we're
   > probably going to want in moderation, we would still need to discuss
   > supply chain safety. Python is not as mature here as, say, Go.
   
   What external dependencies are you imagining?
   
   
   
   > == A Plan ==
   > 
   > Even if everyone were on board immediately, there's a lot of work to
   > do. I'd like to add pytest in a more probationary status, so we can
   > iron out the inevitable wrinkles. My proposal would be:
   > 
   > 1. Commit bare-bones support in our Meson setup for running pytest, so
   > everyone can kick the tires independently.
   > 2. Add a test for something that we can't currently exercise.
   > 3. Port a test from a place where the maintenance is terrible, to see
   > if we can improve it.
   > 
   > If we hate it by that point, no harm done; tear it back out. Otherwise
   > we keep rolling forward.
   
   I think somewhere between 1 and 4 a *substantial* amount of work would be
   required to provide a bunch of the infrastructure that Cluster.pm etc
   provide. Otherwise we'll end up with a lot of copy pasted code between tests.
   
   Greetings,
   
   Andres Freund
   
   
   
   

4. Re: RFC: adding pytest as a supported test framework

   Andrew Dunstan <andrew@dunslane.net> — 2024-06-10T20:46:56Z

   On 2024-06-10 Mo 16:04, Andres Freund wrote:
   > Hi,
   >
   >
   > Just for context for the rest the email: I think we desperately need to move
   > off perl for tests. The infrastructure around our testing is basically
   > unmaintained and just about nobody that started doing dev stuff in the last 10
   > years learned perl.
   
   
   Andres,
   
   I get that you don't like perl. But it's hard for me to take this 
   terribly seriously. "desperately" seems like massive overstatement at 
   best. As for what up and coming developers learn, they mostly don't 
   learn C either, and that's far more critical to what we do.
   
   I'm not sure what part of the testing infrastructure you think is 
   unmaintained. For example, the last release of Test::Simple was all the 
   way back on April 25.
   
   Maybe there are some technical superiorities about what Jacob is 
   proposing, enough for us to add it to our armory. I'll keep an open mind 
   on that.
   
   But let's not throw the baby out with the bathwater. Quite apart from 
   anything else, a wholesale rework of the test infrastructure would make 
   backpatching more painful.
   
   
   cheers
   
   
   andrew
   
   --
   Andrew Dunstan
   EDB:https://www.enterprisedb.com
   

5. Re: RFC: adding pytest as a supported test framework

   Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-10T21:40:39Z

   On Mon, 10 Jun 2024 at 20:46, Jacob Champion
   <jacob.champion@enterprisedb.com> wrote:
   > For the v18 cycle, I would like to try to get pytest [1] in as a
   > supported test driver, in addition to the current offerings.
   
   Huge +1 from me (but I'm definitely biased here)
   
   > Thoughts? Suggestions?
   
   I think the most important thing is that we make it easy for people to
   use this thing, and use it in a "correct" way. I have met very few
   people that actually like writing tests, so I think it's very
   important to make the barrier to do so as low as possible.
   
   For the PgBouncer repo I created my own pytest based test suite more
   ~1.5 years ago now. I tried to make it as easy as possible to write
   tests there, and it has worked out quite well imho. I don't think it
   makes sense to copy all things I did there verbatim, because some of
   it is quite specific to testing PgBouncer. But I do think there's
   quite a few things that could probably be copied (or at least inspire
   what you do). Some examples:
   
   1. helpers to easily run shell commands, most importantly setting
   check=True by default[1]
   2. helper to get a free tcp port[2]
   3. helper to check if the log contains a specific string[3]
   4. automatically show PG logs on test failure[4]
   5. helpers to easily run sql commands (psycopg interface isn't very
   user friendly imho for the common case)[5]
   6. startup/teardown cleanup logic[6]
   
   [1]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L83-L131
   [2]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L210-L233
   [3]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1125-L1143
   [4]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1075-L1103
   [5]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L326-L338
   [6]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L546-L642
   
   
   On Mon, 10 Jun 2024 at 22:04, Andres Freund <andres@anarazel.de> wrote:
   > > Problem 1 (rerun failing tests): One architectural roadblock to this
   > > in our Test::More suite is that tests depend on setup that's done by
   > > previous tests. pytest allows you to declare each test's setup
   > > requirements via pytest fixtures, letting the test runner build up the
   > > world exactly as it needs to be for a single isolated test. These
   > > fixtures may be given a "scope" so that multiple tests may share the
   > > same setup for performance or other reasons.
   >
   > OTOH, that's quite likely to increase overall test times very
   > significantly. Yes, sometimes that can be avoided with careful use of various
   > features, but often that's hard, and IME is rarely done rigiorously.
   
   You definitely want to cache things like initdb and "pg_ctl start".
   But that's fairly easy to do with some startup/teardown logic. For
   PgBouncer I create a dedicated schema for each test that needs to
   create objects and automatically drop that schema at the end of the
   test[6] (including any temporary objects outside of schemas like
   users/replication slots). You can even choose not to clean up certain
   large schemas if they are shared across multiple tests.
   
   [6]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L546-L642
   
   > > Problem 2 (seeing what failed): pytest does this via assertion
   > > introspection and very detailed failure reporting. If you haven't seen
   > > this before, take a look at the pytest homepage [1]; there's an
   > > example of a full log.
   >
   > That's not really different than what the perl tap test stuff allows. We
   > indeed are bad at utilizing it, but I'm not sure that switching languages will
   > change that.
   
   It's not about allowing, it's about doing the thing that you want by
   default. The following code
   
   assert a == b
   
   will show you the actual values of both a and b when the test fails,
   instead of saying something like "false is not true". Ofcourse you can
   provide a message here too, like with perl its ok function, but even
   when you don't the output is helpful.
   
   > I think part of the problem is that the information about what precisely
   > failed is often much harder to collect when testing multiple servers
   > interacting than when doing localized unit tests.
   >
   > I think we ought to invest a bunch in improving that, I'd hope that a lot of
   > that work would be largely independent of the language the tests are written
   > in.
   
   Well, as you already noted no-one that started doing dev stuff in the
   last 10 years knows Perl nor wants to learn it. So a large part of the
   community tries to touch the current perl test suite as little as
   possible. I personally haven't tried to improve anything about our
   perl testing framework, even though I'm normally very much into
   improving developer tooling.
   
   
   > > Python's standard library has lots of power by itself, with very good
   > > documentation. And virtualenvs and better package tooling have made it
   > > much easier, IMO, to avoid the XKCD dependency tangle [4] of the
   > > 2010s.
   >
   > Ugh, I think this is actually python's weakest area. There's about a dozen
   > package managers and "python distributions", that are at best half compatible,
   > and the documentation situation around this is *awful*.
   
   I definitely agree this is Python its weakest area. But since venv is
   part of the python standard library it's much better. I have the
   following short blurb in PgBouncer its test README[7] and it has
   worked for all contributors so far:
   
   # create a virtual environment (only needed once)
   python3 -m venv env
   
   # activate the environment. You will need to activate this environment in
   # your shell every time you want to run the tests. (so it's needed once per
   # shell).
   source env/bin/activate
   
   # Install the dependencies (only needed once, or whenever extra dependencies
   # get added to requirements.txt)
   pip install -r requirements.txt
   
   
   [7]: https://github.com/pgbouncer/pgbouncer/blob/master/test/README.md
   
   > I think somewhere between 1 and 4 a *substantial* amount of work would be
   > required to provide a bunch of the infrastructure that Cluster.pm etc
   > provide. Otherwise we'll end up with a lot of copy pasted code between tests.
   
   Totally agreed, that we should have a fairly decent base to work on
   top of. I think we should at least port a few tests to show that the
   base has at least the most basic functionality.
   
   
   
   

6. Re: RFC: adding pytest as a supported test framework

   Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-10T21:43:22Z

   On Mon, 10 Jun 2024 at 22:47, Andrew Dunstan <andrew@dunslane.net> wrote:
   > As for what up and coming developers learn, they mostly don't learn C either, and that's far more critical to what we do.
   
   I think many up and coming devs have at least touched C somewhere
   (e.g. in university). And because it's more critical to the project
   and also to many other low level projects, they don't mind learning it
   so much if they don't know it yet. But I, for example, try to write as
   few Perl tests as possible, because getting good at Perl has pretty
   much no use to me outside of writing tests for postgres.
   
   (I do personally think that official Rust support in Postgres would
   probably be a good thing, but that is a whole other discussion that
   I'd like to save for some other day)
   
   > But let's not throw the baby out with the bathwater. Quite apart from anything else, a wholesale rework of the test infrastructure would make backpatching more painful.
   
   Backporting test improvements to decrease backporting pain is
   something we don't look badly upon afaict (Citus its test suite breaks
   semi-regularly on minor PG version updates due to some slight output
   changes introduced by e.g. an updated version of the isolationtester).
   
   
   
   

7. Re: RFC: adding pytest as a supported test framework

   Andres Freund <andres@anarazel.de> — 2024-06-11T01:49:11Z

   Hi,
   
   On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
   > 
   > On 2024-06-10 Mo 16:04, Andres Freund wrote:
   > > Hi,
   > > 
   > > 
   > > Just for context for the rest the email: I think we desperately need to move
   > > off perl for tests. The infrastructure around our testing is basically
   > > unmaintained and just about nobody that started doing dev stuff in the last 10
   > > years learned perl.
   
   > Andres,
   > 
   > I get that you don't like perl.
   
   I indeed don't particularly like perl - but that's really not the main
   issue. I've already learned [some of] it. What is the main issue is that I've
   also watched several newer folks try to write tests in it, and it was not
   pretty.
   
   
   > But it's hard for me to take this terribly seriously. "desperately" seems
   > like massive overstatement at best.
   
   Shrug.
   
   
   > As for what up and coming developers learn, they mostly don't learn C
   > either, and that's far more critical to what we do.
   
   C is a a lot more useful to to them than perl. And it's actually far more
   widely known these days than perl. C does teach you some reasonably
   low-level-ish understanding of hardware. There are gazillions of programs
   written in C that we'll have to maintain for decades. I don't think that's
   comparably true for perl.
   
   
   > I'm not sure what part of the testing infrastructure you think is
   > unmaintained. For example, the last release of Test::Simple was all the way
   > back on April 25.
   
   IPC::Run is quite buggy and basically just maintained by Noah these days.
   
   Greetings,
   
   Andres Freund
   
   
   
   

8. Re: RFC: adding pytest as a supported test framework

   Andrew Dunstan <andrew@dunslane.net> — 2024-06-11T12:04:57Z

   On 2024-06-10 Mo 21:49, Andres Freund wrote:
   > Hi,
   >
   > On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
   >> On 2024-06-10 Mo 16:04, Andres Freund wrote:
   >>> Hi,
   >>>
   >>>
   >>> Just for context for the rest the email: I think we desperately need to move
   >>> off perl for tests. The infrastructure around our testing is basically
   >>> unmaintained and just about nobody that started doing dev stuff in the last 10
   >>> years learned perl.
   >> Andres,
   >>
   >> I get that you don't like perl.
   > I indeed don't particularly like perl - but that's really not the main
   > issue. I've already learned [some of] it. What is the main issue is that I've
   > also watched several newer folks try to write tests in it, and it was not
   > pretty.
   
   
   Hmm. I've done webinars in the past about how to write TAP tests for 
   PostgreSQL, maybe I need to beef that up some.
   
   
   >> I'm not sure what part of the testing infrastructure you think is
   >> unmaintained. For example, the last release of Test::Simple was all the way
   >> back on April 25.
   > IPC::Run is quite buggy and basically just maintained by Noah these days.
   >
   
   Yes, that's true. I think the biggest pain point is possibly the 
   recovery tests.
   
   Some time ago I did some work on wrapping libpq using the perl FFI 
   module. It worked pretty well, and would mean we could probably avoid 
   many uses of IPC::Run, and would probably be substantially more 
   efficient (no fork required). It wouldn't avoid all uses of IPC::Run, 
   though.
   
   But my point was mainly that while a new framework might have value, I 
   don't think we need to run out and immediately rewrite several hundred 
   TAP tests. Let's pick the major pain points and address those.
   
   
   cheers
   
   
   andrew
   
   --
   Andrew Dunstan
   EDB:https://www.enterprisedb.com
   

9. Re: RFC: adding pytest as a supported test framework

   Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-11T14:28:23Z

   On Mon, Jun 10, 2024 at 1:04 PM Andres Freund <andres@anarazel.de> wrote:
   > Just for context for the rest the email: I think we desperately need to move
   > off perl for tests. The infrastructure around our testing is basically
   > unmaintained and just about nobody that started doing dev stuff in the last 10
   > years learned perl.
   
   Okay. Personally, I'm going to try to stay out of discussions around
   subtracting Perl and focus on adding Python, for a bunch of different
   reasons:
   
   - Tests aren't cheap, but in my experience, the maintenance-cost math
   for tests is a lot different than the math for implementations.
   - I don't personally care for Perl, but having tests in any form is
   usually better than not having them.
   - Trying to convince people to get rid of X while adding Y is a good
   way to make sure Y never happens.
   
   > On 2024-06-10 11:46:00 -0700, Jacob Champion wrote:
   > > 4. It'd be great to split apart client-side tests from server-side
   > > tests. Driving Postgres via psql all the time is fine for acceptance
   > > testing, but it becomes a big problem when you need to test how
   > > clients talk to servers with incompatible feature sets, or how a peer
   > > behaves when talking to something buggy.
   >
   > That seems orthogonal to using pytest vs something else?
   
   Yes, I think that's fair. It's going to be hard not to talk about
   "things that pytest+Python don't give us directly but are much easier
   to build" in all of this (and I tried to call that out in the next
   section, maybe belatedly). I think I'm going to have to convince both
   a group of people who want to ask "why pytest in particular?" and a
   group of people who ask "why isn't what we have good enough?"
   
   > > == Why pytest? ==
   > >
   > > From the small and biased sample at the unconference session, it looks
   > > like a number of people have independently settled on pytest in their
   > > own projects. In my opinion, pytest occupies a nice space where it
   > > solves some of the above problems for us, and it gives us plenty of
   > > tools to solve the other problems without too much pain.
   >
   > We might be able to alleviate that by simply abstracting it away, but I found
   > pytest's testrunner pretty painful. Oodles of options that are not very well
   > documented and that often don't work because they are very specific to some
   > situations, without that being explained.
   
   Hm. There are a bunch of them, but I've never needed to go through the
   oodles of options. Anything in particular that caused problems?
   
   > > Problem 1 (rerun failing tests): One architectural roadblock to this
   > > in our Test::More suite is that tests depend on setup that's done by
   > > previous tests. pytest allows you to declare each test's setup
   > > requirements via pytest fixtures, letting the test runner build up the
   > > world exactly as it needs to be for a single isolated test. These
   > > fixtures may be given a "scope" so that multiple tests may share the
   > > same setup for performance or other reasons.
   >
   > OTOH, that's quite likely to increase overall test times very
   > significantly. Yes, sometimes that can be avoided with careful use of various
   > features, but often that's hard, and IME is rarely done rigiorously.
   
   Well, scopes are pretty front and center when you start building
   pytest fixtures, and the complicated longer setups will hopefully
   converge correctly early on and be reused everywhere else. I imagine
   no one wants to build cluster setup from scratch.
   
   On a slight tangent, is this not a problem today? I mean... part of my
   personal long-term goal is in increasing test hygiene, which is going
   to take some shifts in practice. As long as review keeps the quality
   of the tests fairly high, I see the inevitable "our tests take too
   long" problem as a good one. That's true no matter what framework we
   use, unless the framework is so bad that no one uses it and the
   runtime is trivial. If we're worried that people will immediately
   start exploding the runtime and no one will notice during review,
   maybe we can have some infrastructure flag how much a patch increased
   it?
   
   > > Problem 2 (seeing what failed): pytest does this via assertion
   > > introspection and very detailed failure reporting. If you haven't seen
   > > this before, take a look at the pytest homepage [1]; there's an
   > > example of a full log.
   >
   > That's not really different than what the perl tap test stuff allows. We
   > indeed are bad at utilizing it, but I'm not sure that switching languages will
   > change that.
   
   Jelte already touched on this, but I wanted to hammer on the point: If
   no one, not even the developers who chose and like Perl, is using
   Test::More in a way that's maintainable, I would prefer to use a
   framework that does maintainable things by default so that you have to
   try really hard to screw it up. It is possible to screw up `assert
   actual == expected`, but it takes more work than doing it the right
   way.
   
   > I think part of the problem is that the information about what precisely
   > failed is often much harder to collect when testing multiple servers
   > interacting than when doing localized unit tests.
   >
   > I think we ought to invest a bunch in improving that, I'd hope that a lot of
   > that work would be largely independent of the language the tests are written
   > in.
   
   We do a lot more acceptance testing than internal testing, which came
   up as a major complaint from me and others during the unconference.
   One of the reasons people avoid writing internal tests in Perl is
   because it's very painful to find a rhythm with Test::More. From
   experience test-driving the OAuth work, I'm *very* happy with the
   development cycle that pytest gave me.
   
   Other languages _could_ do that, sure. It's a simple matter of programming...
   
   > Ugh, I think this is actually python's weakest area. There's about a dozen
   > package managers and "python distributions", that are at best half compatible,
   > and the documentation situation around this is *awful*.
   
   So... don't support the half-compatible stuff? I thought this
   conversation was still going on with Windows Perl (ActiveState?
   Strawberry?) but everyone just seems to pick what works for them and
   move on to better things to do.
   
   Modern CPython includes pip and venv. Done. If someone comes to us
   with some horrible Anaconda setup wanting to know why their duct tape
   doesn't work, can't we just tell them no?
   
   > > When it comes to third-party packages, which I think we're
   > > probably going to want in moderation, we would still need to discuss
   > > supply chain safety. Python is not as mature here as, say, Go.
   >
   > What external dependencies are you imagining?
   
   The OAuth pytest suite makes extensive use of
   - psycopg, to easily drive libpq;
   - construct, for on-the-wire packet representations and manipulation; and
   - pyca/cryptography, for easy generation of certificates and manual
   crypto testing.
   
   I'd imagine each would need considerable discussion, if there is
   interest in doing the same things that I do with them.
   
   > I think somewhere between 1 and 4 a *substantial* amount of work would be
   > required to provide a bunch of the infrastructure that Cluster.pm etc
   > provide. Otherwise we'll end up with a lot of copy pasted code between tests.
   
   Possibly, yes. I think it depends on what you want to test first, and
   there's a green-field aspect of hope/anxiety/ennui, too. Are you
   trying to port the acceptance-test framework that we already have, or
   are you trying to build a framework that can handle the things we
   can't currently test? Will it be easier to refactor duplication into
   shared fixtures when the language doesn't encourage an infinite number
   of ways to do things? Or will we have to keep on top of it to avoid
   pain?
   
   --Jacob
   
   
   
   

10. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-11T14:30:56Z

    On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov
    <aekorotkov@gmail.com> wrote:
    > Thank you for working on this.
    > Do you think you could re-use something from testgres[1] package?
    
    Possibly? I think we're all coming at this with our own bags of tricks
    and will need to carve off pieces to port, contribute, or reimplement.
    Does testgres have something in particular you'd like to see the
    Postgres tests support?
    
    Thanks,
    --Jacob
    
    
    
    

11. Re: RFC: adding pytest as a supported test framework

    Noah Misch <noah@leadboat.com> — 2024-06-11T23:48:29Z

    On Mon, Jun 10, 2024 at 06:49:11PM -0700, Andres Freund wrote:
    > On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
    > > On 2024-06-10 Mo 16:04, Andres Freund wrote:
    > > > Just for context for the rest the email: I think we desperately need to move
    > > > off perl for tests. The infrastructure around our testing is basically
    > > > unmaintained and just about nobody that started doing dev stuff in the last 10
    > > > years learned perl.
    
    > > As for what up and coming developers learn, they mostly don't learn C
    > > either, and that's far more critical to what we do.
    > 
    > C is a a lot more useful to to them than perl. And it's actually far more
    > widely known these days than perl.
    
    If we're going to test in a non-Perl language, I'd pick C over Python.  There
    would be several other unlikely-community-choice languages I'd pick over
    Python (C#, Java, C++).  We'd need a library like today's Perl
    PostgreSQL::Test to make C-language tests nice, but the same would apply to
    any new language.
    
    I also want the initial scope to be the new language coexisting with the
    existing Perl tests.  If a bulk translation ever happens, it should happen
    long after the debut of the new framework.  That said, I don't much trust a
    human-written bulk language translation to go through without some tests
    accidentally ceasing to test what they test in Perl today.
    
    
    
    

12. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-12T10:17:21Z

    On 2024-06-11 Tu 19:48, Noah Misch wrote:
    > On Mon, Jun 10, 2024 at 06:49:11PM -0700, Andres Freund wrote:
    >> On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
    >>> On 2024-06-10 Mo 16:04, Andres Freund wrote:
    >>>> Just for context for the rest the email: I think we desperately need to move
    >>>> off perl for tests. The infrastructure around our testing is basically
    >>>> unmaintained and just about nobody that started doing dev stuff in the last 10
    >>>> years learned perl.
    >>> As for what up and coming developers learn, they mostly don't learn C
    >>> either, and that's far more critical to what we do.
    >> C is a a lot more useful to to them than perl. And it's actually far more
    >> widely known these days than perl.
    > If we're going to test in a non-Perl language, I'd pick C over Python.  There
    > would be several other unlikely-community-choice languages I'd pick over
    > Python (C#, Java, C++).  We'd need a library like today's Perl
    > PostgreSQL::Test to make C-language tests nice, but the same would apply to
    > any new language.
    
    
    Indeed. We've invested quite a lot of effort on that infrastructure. I 
    guess people can learn from what we've done so a second language might 
    be easier to support.
    
    (Java would be my pick from your unlikely set, but I can see the 
    attraction of Python.)
    
    
    >
    > I also want the initial scope to be the new language coexisting with the
    > existing Perl tests.  If a bulk translation ever happens, it should happen
    > long after the debut of the new framework.  That said, I don't much trust a
    > human-written bulk language translation to go through without some tests
    > accidentally ceasing to test what they test in Perl today.
    
    
    +1
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB: https://www.enterprisedb.com
    
    
    
    
    

13. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-12T11:40:30Z

    On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote:
    > If we're going to test in a non-Perl language, I'd pick C over Python.  There
    > would be several other unlikely-community-choice languages I'd pick over
    > Python (C#, Java, C++).
    
    My main goals of this thread are:
    1. Allowing people to quickly write tests
    2. Have those tests do what the writer intended them to do
    3. Have good error reporting by default
    
    Those goals indeed don't necesitate python.
    
    But I think those are really hard to achieve with any C based
    framework, and probably with C++ too. Also manual memory management in
    tests seems to add tons of complexity for basically no benefit.
    
    I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable
    choices for me (and possibly some more). They allow some type of
    introspection, they have a garbage collector, and their general
    tooling is quite good.
    
    But I think a dynamically typed scripting language is much more
    fitting for writing tests like this. I love static typing for
    production code, but imho it really doesn't have much benefit for
    tests.
    
    As scripting languages go, the ones that are still fairly heavily in
    use are Javascript, Python, Ruby, and PHP. I think all of those could
    probably work, but my personal order of preference would be Python,
    Ruby, Javascript, PHP.
    
    Finally, I'm definitely biased towards using Python myself. But I
    think there's good reasons for that:
    1. In the data space, that Postgres in, Python is very heavily used for analysis
    2. Everyone coming out of university these days knows it to some extent
    3. Multiple people in the community have been writing Postgres related
    tests in python and have enjoyed doing so (me[1], Jacob[2],
    Alexander[3])
    
    What language people want to write tests in is obviously very
    subjective. And obviously we cannot allow every language for writing
    tests. But I think if ~25% of the community prefers to write their
    tests in Python. Then that should be enough of a reason to allow them
    to do so.
    
    TO CLARIFY: This thread is not a proposal to replace Perl with Python.
    It's a proposal to allow people to also write tests in Python.
    
    > I also want the initial scope to be the new language coexisting with the
    > existing Perl tests.  If a bulk translation ever happens, it should happen
    > long after the debut of the new framework.  That said, I don't much trust a
    > human-written bulk language translation to go through without some tests
    > accidentally ceasing to test what they test in Perl today.
    
    I definitely don't think we should rewrite all the tests that we have
    in Perl today into some other language. But I do think that whatever
    language we choose, that language should make it as least as easy to
    write tests, as easy to read them and as easy to see that they are
    testing the intended thing, as is currently the case for Perl.
    Rewriting a few Perl tests into the new language, even if not merging
    the rewrite, is a good way of validating that imho.
    
    PS. For PgBouncer I actually hand-rewrote all the tests that we had in
    bash (which is the worst testing language ever) in Python and doing so
    actually found more bugs in PgBouncer code that our bash tests
    wouldn't catch. So it's not necessarily the case that you lose
    coverage by rewriting tests.
    
    [1]: https://github.com/pgbouncer/pgbouncer/tree/master/test
    [2]: https://github.com/jchampio/pg-pytest-suite
    [3]: https://github.com/postgrespro/testgres
    
    
    
    

14. Re: RFC: adding pytest as a supported test framework

    Alexander Korotkov <aekorotkov@gmail.com> — 2024-06-12T11:48:45Z

    On Tue, Jun 11, 2024 at 5:31 PM Jacob Champion
    <jacob.champion@enterprisedb.com> wrote:
    > On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov
    > <aekorotkov@gmail.com> wrote:
    > > Thank you for working on this.
    > > Do you think you could re-use something from testgres[1] package?
    >
    > Possibly? I think we're all coming at this with our own bags of tricks
    > and will need to carve off pieces to port, contribute, or reimplement.
    > Does testgres have something in particular you'd like to see the
    > Postgres tests support?
    
    Generally, testgres was initially designed as Python analogue of what
    we have in src/test/perl/PostgreSQL/Test.  In particular its
    testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster.  It
    comes under PostgreSQL License.  So, I wonder if we could revise it
    and fetch most part of it into our source tree.
    
    ------
    Regards,
    Alexander Korotkov
    Supabase
    
    
    
    

15. Re: RFC: adding pytest as a supported test framework

    Alexander Korotkov <aekorotkov@gmail.com> — 2024-06-12T12:00:54Z

    On Wed, Jun 12, 2024 at 2:48 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
    > On Tue, Jun 11, 2024 at 5:31 PM Jacob Champion
    > <jacob.champion@enterprisedb.com> wrote:
    > > On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov
    > > <aekorotkov@gmail.com> wrote:
    > > > Thank you for working on this.
    > > > Do you think you could re-use something from testgres[1] package?
    > >
    > > Possibly? I think we're all coming at this with our own bags of tricks
    > > and will need to carve off pieces to port, contribute, or reimplement.
    > > Does testgres have something in particular you'd like to see the
    > > Postgres tests support?
    >
    > Generally, testgres was initially designed as Python analogue of what
    > we have in src/test/perl/PostgreSQL/Test.  In particular its
    > testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster.  It
    > comes under PostgreSQL License.  So, I wonder if we could revise it
    > and fetch most part of it into our source tree.
    
    Plus testgres exists from 2016 and already have quite amount of use
    cases.  This is what I quickly found on github.
    
    https://github.com/adjust/pg_querylog
    https://github.com/postgrespro/pg_pathman
    https://github.com/lanterndata/lantern
    https://github.com/orioledb/orioledb
    https://github.com/cbandy/pgtwixt
    https://github.com/OpenNTI/nti.testing
    https://github.com/postgrespro/pg_probackup
    https://github.com/postgrespro/rum
    
    ------
    Regards,
    Alexander Korotkov
    Supabase
    
    
    
    

16. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-12T12:09:21Z

    On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote:
    > If we're going to test in a non-Perl language, I'd pick C over Python.
    > <snip>
    > We'd need a library like today's Perl
    > PostgreSQL::Test to make C-language tests nice, but the same would apply to
    > any new language.
    
    P.P.S. We already write tests in C, we use it for testing libpq[1].
    I'd personally definitely welcome a C library to make those tests
    nicer to write, because I've written a fair bit of those in the past
    and currently it's not very fun to do.
    
    [1]: https://github.com/postgres/postgres/blob/master/src/test/modules/libpq_pipeline/libpq_pipeline.c
    
    
    
    

17. Re: RFC: adding pytest as a supported test framework

    Neil <neil@fairwindsoft.com> — 2024-06-12T13:49:32Z

    > On Jun 12, 2024, at 6:40 AM, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > 
    > I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable
    > choices for me (and possibly some more). They allow some type of
    > introspection, they have a garbage collector, and their general
    > tooling is quite good.
    > 
    
    Having used Python for 15+ years and then abandoned it for all projects I would
    say the single most important points for a long term project like Postgres is,
    not necessarily in order, package stability, package depth, semantic versioning,
    available resources, and multiprocessor support.
    
    The reason I abandoned Python was for the constant API breaks in packages. Yes,
    python is a great language to teach in school for a one-time class project, but
    that is not Postgres’s use-case.  Remember that Fortran and Pascal were the 
    darlings for teaching in school prior to Python and no-one uses them any more.
    
    Yes Python innovates fast and is fashionable.  But again, not Postgres’s use-case.
    
    I believe that anyone coming out of school these days would have a relatively
    easy transition to any of Go, Rust, Kotlin, Swift, etc.  In other words, any of
    the modern languages.  In addition, the language should scale well to
    multiprocessors, because parallel testing is becoming more important every day.
    
    If the Postgres project is going to pick a new language for testing, it should
    pick a language for the next 50 years based on the projects needs.
    
    Python is good for package depth and resource availability, but fails IMO in the
    other categories. My experience with python where the program flow can change
    because of non-visible characters is a terrible way to write robust long term
    maintainable code.  Because of this most of the modern languages are going to be
    closer in style to Postgres’s C code base than Python.
    
    
    

18. Re: RFC: adding pytest as a supported test framework

    Wolfgang Walther <walther@technowledgy.de> — 2024-06-12T14:56:49Z

    Jelte Fennema-Nio:
    > As scripting languages go, the ones that are still fairly heavily in
    > use are Javascript, Python, Ruby, and PHP. I think all of those could
    > probably work, but my personal order of preference would be Python,
    > Ruby, Javascript, PHP.
    > 
    > Finally, I'm definitely biased towards using Python myself. But I
    > think there's good reasons for that:
    > 1. In the data space, that Postgres in, Python is very heavily used for analysis
    > 2. Everyone coming out of university these days knows it to some extent
    > 3. Multiple people in the community have been writing Postgres related
    > tests in python and have enjoyed doing so (me[1], Jacob[2],
    > Alexander[3])
    
    PostgREST also uses pytest for integration tests - and that was a very 
    good decision compared to the bash based tests we had before.
    
    One more argument for Python compared to the other mentioned scripting 
    languages: Python is already a development dependency via meson. None of 
    the other 3 are. In a future where meson will be the only build system, 
    we will have python "for free" already.
    
    Best,
    
    Wolfgang
    
    
    
    

19. Re: RFC: adding pytest as a supported test framework

    Andres Freund <andres@anarazel.de> — 2024-06-12T15:28:12Z

    Hi,
    
    On 2024-06-11 08:04:57 -0400, Andrew Dunstan wrote:
    > Some time ago I did some work on wrapping libpq using the perl FFI module.
    > It worked pretty well, and would mean we could probably avoid many uses of
    > IPC::Run, and would probably be substantially more efficient (no fork
    > required). It wouldn't avoid all uses of IPC::Run, though.
    
    FWIW, I'd *love* to see work on this continue. The reduction in test runtime
    on windows is substantial and would shorten the hack->CI->fail->hack loop a
    good bit shorter. And save money.
    
    
    > But my point was mainly that while a new framework might have value, I don't
    > think we need to run out and immediately rewrite several hundred TAP tests.
    
    Oh, yea. That's not at all feasible to just do in one go.
    
    Greetings,
    
    Andres Freund
    
    
    
    

20. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-12T15:35:22Z

    On Wed, 12 Jun 2024 at 15:49, FWS Neil <neil@fairwindsoft.com> wrote:
    > I believe that anyone coming out of school these days would have a relatively
    > easy transition to any of Go, Rust, Kotlin, Swift, etc.  In other words, any of
    > the modern languages.
    
    Agreed, which is why I said they'd be acceptable to me. But I think
    one important advantage of Python is that it's clear that many people
    in the community are willing to write tests in it. At PGConf.dev there
    were a lot of people in the unconference session about this. Also many
    people already wrote a Postgres testing framework for python, and are
    using it (see list of projects that Alexander shared). I haven't seen
    that level of willingness to write tests for any of those other
    languages (yet).
    
    > In addition, the language should scale well to
    > multiprocessors, because parallel testing is becoming more important every day.
    > <snip>
    > Python is good for package depth and resource availability, but fails IMO in the
    > other categories.
    
    You can easily pin packages or call a different function based on the
    version of the package, so I'm not sure what the problem is with
    package stability. Also chances are we'll pull in very little external
    packages and rely mostly on the stdlib (which is quite stable).
    
    Regarding parallelised running of tests, I agree that's very
    important. And indeed normally parallelism in python can be a pain
    (although async/await makes I/O parallelism a lot better at least).
    But running pytest tests in parallel is extremely easy by using
    pytest-xdist[1], so I don't really think there's an issue for this
    specific Python usecase.
    
    > My experience with python where the program flow can change
    > because of non-visible characters is a terrible way to write robust long term
    > maintainable code.  Because of this most of the modern languages are going to be
    > closer in style to Postgres’s C code base than Python.
    
    I'm assuming this is about spaces vs curly braces for blocks? Now that
    we have auto formatters for every modern programming language I indeed
    prefer curly braces myself too. But honestly that's pretty much a tabs
    vs spaces discussion.
    
    [1]: https://pypi.org/project/pytest-xdist/
    
    On Wed, 12 Jun 2024 at 15:49, FWS Neil <neil@fairwindsoft.com> wrote:
    >
    >
    > > On Jun 12, 2024, at 6:40 AM, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > >
    > > I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable
    > > choices for me (and possibly some more). They allow some type of
    > > introspection, they have a garbage collector, and their general
    > > tooling is quite good.
    > >
    >
    > Having used Python for 15+ years and then abandoned it for all projects I would
    > say the single most important points for a long term project like Postgres is,
    > not necessarily in order, package stability, package depth, semantic versioning,
    > available resources, and multiprocessor support.
    >
    > The reason I abandoned Python was for the constant API breaks in packages. Yes,
    > python is a great language to teach in school for a one-time class project, but
    > that is not Postgres’s use-case.  Remember that Fortran and Pascal were the
    > darlings for teaching in school prior to Python and no-one uses them any more.
    >
    > Yes Python innovates fast and is fashionable.  But again, not Postgres’s use-case.
    >
    > I believe that anyone coming out of school these days would have a relatively
    > easy transition to any of Go, Rust, Kotlin, Swift, etc.  In other words, any of
    > the modern languages.  In addition, the language should scale well to
    > multiprocessors, because parallel testing is becoming more important every day.
    >
    > If the Postgres project is going to pick a new language for testing, it should
    > pick a language for the next 50 years based on the projects needs.
    >
    > Python is good for package depth and resource availability, but fails IMO in the
    > other categories. My experience with python where the program flow can change
    > because of non-visible characters is a terrible way to write robust long term
    > maintainable code.  Because of this most of the modern languages are going to be
    > closer in style to Postgres’s C code base than Python.
    
    
    
    

21. Re: RFC: adding pytest as a supported test framework

    Andres Freund <andres@anarazel.de> — 2024-06-12T15:50:40Z

    Hi,
    
    On 2024-06-11 07:28:23 -0700, Jacob Champion wrote:
    > On Mon, Jun 10, 2024 at 1:04 PM Andres Freund <andres@anarazel.de> wrote:
    > > Just for context for the rest the email: I think we desperately need to move
    > > off perl for tests. The infrastructure around our testing is basically
    > > unmaintained and just about nobody that started doing dev stuff in the last 10
    > > years learned perl.
    > 
    > Okay. Personally, I'm going to try to stay out of discussions around
    > subtracting Perl and focus on adding Python, for a bunch of different
    > reasons:
    
    I think I might have formulated my paragraph above badly - I didn't mean that
    we should move away from perl tests tomorrow, but that we need a path forward
    that allows folks to write tests without perl.
    
    
    > - Tests aren't cheap, but in my experience, the maintenance-cost math
    > for tests is a lot different than the math for implementations.
    
    At the moment they tend to be *more* expensive often, due to spurious
    failures. That's mostly not perl's fault, don't get me wrong, but us not
    having better infrastructure for testing complicated behaviour and/or testing
    things on a more narrow basis.
    
    
    > > > Problem 1 (rerun failing tests): One architectural roadblock to this
    > > > in our Test::More suite is that tests depend on setup that's done by
    > > > previous tests. pytest allows you to declare each test's setup
    > > > requirements via pytest fixtures, letting the test runner build up the
    > > > world exactly as it needs to be for a single isolated test. These
    > > > fixtures may be given a "scope" so that multiple tests may share the
    > > > same setup for performance or other reasons.
    > >
    > > OTOH, that's quite likely to increase overall test times very
    > > significantly. Yes, sometimes that can be avoided with careful use of various
    > > features, but often that's hard, and IME is rarely done rigiorously.
    > 
    > Well, scopes are pretty front and center when you start building
    > pytest fixtures, and the complicated longer setups will hopefully
    > converge correctly early on and be reused everywhere else. I imagine
    > no one wants to build cluster setup from scratch.
    
    One (the?) prime source of state in our tap tests is the
    database. Realistically we can't just tear that one down and reset it between
    tests without causing the test times to explode. So we'll have to live with
    some persistent state.
    
    
    > On a slight tangent, is this not a problem today?
    
    It is, but that doesn't mean making it even bigger is unproblematic :)
    
    
    
    
    > > I think part of the problem is that the information about what precisely
    > > failed is often much harder to collect when testing multiple servers
    > > interacting than when doing localized unit tests.
    > >
    > > I think we ought to invest a bunch in improving that, I'd hope that a lot of
    > > that work would be largely independent of the language the tests are written
    > > in.
    > 
    > We do a lot more acceptance testing than internal testing, which came
    > up as a major complaint from me and others during the unconference.
    > One of the reasons people avoid writing internal tests in Perl is
    > because it's very painful to find a rhythm with Test::More.
    
    What definition of internal tests are you using here?
    
    I think a lot of our tests are complicated, fragile and slow because we almost
    exclusively do end-to-end tests, because with a few exceptions we don't have a
    way to exercise code in a more granular way.
    
    
    > > > When it comes to third-party packages, which I think we're
    > > > probably going to want in moderation, we would still need to discuss
    > > > supply chain safety. Python is not as mature here as, say, Go.
    > >
    > > What external dependencies are you imagining?
    > 
    > The OAuth pytest suite makes extensive use of
    > - psycopg, to easily drive libpq;
    
    That's probably not going to fly. It introduces painful circular dependencies
    between building postgres (for libpq), building psycopg (requiring libpq) and
    testing postgres (requiring psycopg).
    
    You *can* solve such issues, but we've debated that in the past, and I doubt
    we'll find agreement on the awkwardness it introduces.
    
    
    > - construct, for on-the-wire packet representations and manipulation; and
    
    That seems fairly minimal.
    
    
    > - pyca/cryptography, for easy generation of certificates and manual
    > crypto testing.
    
    That's a bit more painful, but I guess maybe not unrealistic?
    
    
    > I'd imagine each would need considerable discussion, if there is
    > interest in doing the same things that I do with them.
    
    One thing worth thinking about is that such dependencies have to work on a
    relatively large number of platforms / architectures. A lot of projects
    don't...
    
    Greetings,
    
    Andres Freund
    
    
    
    

22. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-12T16:08:16Z

    On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote:
    > > The OAuth pytest suite makes extensive use of
    > > - psycopg, to easily drive libpq;
    >
    > That's probably not going to fly. It introduces painful circular dependencies
    > between building postgres (for libpq), building psycopg (requiring libpq) and
    > testing postgres (requiring psycopg).
    >
    > You *can* solve such issues, but we've debated that in the past, and I doubt
    > we'll find agreement on the awkwardness it introduces.
    
    psycopg has a few implementations binary, c, & pure python. The pure
    python one can be linked to a specific libpq.so file at runtime[1]. As
    long as we don't break the libpq API (which we shouldn't), we can just
    point it to the libpq compiled by meson/make. We wouldn't be able to
    use the newest libpq features that way (because psycopg wouldn't know
    about them), but that seems totally fine for most usages (i.e. sending
    a query over a connection). If we really want to use those from the
    python tests we could write our own tiny CFFI layer specifically for
    those.
    
    > One thing worth thinking about is that such dependencies have to work on a
    > relatively large number of platforms / architectures. A lot of projects
    > don't...
    
    Do they really? A bunch of the Perl tests we just skip on windows or
    uncommon platforms. I think we could do the same for these.
    
    
    
    

23. Re: RFC: adding pytest as a supported test framework

    Alexander Korotkov <aekorotkov@gmail.com> — 2024-06-12T16:34:19Z

    On Wed, Jun 12, 2024 at 7:08 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote:
    > > > The OAuth pytest suite makes extensive use of
    > > > - psycopg, to easily drive libpq;
    > >
    > > That's probably not going to fly. It introduces painful circular dependencies
    > > between building postgres (for libpq), building psycopg (requiring libpq) and
    > > testing postgres (requiring psycopg).
    > >
    > > You *can* solve such issues, but we've debated that in the past, and I doubt
    > > we'll find agreement on the awkwardness it introduces.
    >
    > psycopg has a few implementations binary, c, & pure python. The pure
    > python one can be linked to a specific libpq.so file at runtime[1]. As
    > long as we don't break the libpq API (which we shouldn't), we can just
    > point it to the libpq compiled by meson/make. We wouldn't be able to
    > use the newest libpq features that way (because psycopg wouldn't know
    > about them), but that seems totally fine for most usages (i.e. sending
    > a query over a connection). If we really want to use those from the
    > python tests we could write our own tiny CFFI layer specifically for
    > those.
    
    I guess you mean pg8000. Note that pg8000 and psycopg2 have some
    differences in interpretation of datatypes (AFAIR arrays, jsonb...).
    So, it would be easier to chose one particular driver.  However, with
    a bit efforts it's possible to make all the code driver agnostic.
    
    ------
    Regards,
    Alexander Korotkov
    Supabase
    
    
    
    

24. Re: RFC: adding pytest as a supported test framework

    Daniel Gustafsson <daniel@yesql.se> — 2024-06-12T16:45:52Z

    > On 12 Jun 2024, at 18:08, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote:
    
    >> One thing worth thinking about is that such dependencies have to work on a
    >> relatively large number of platforms / architectures. A lot of projects
    >> don't...
    > 
    > Do they really? A bunch of the Perl tests we just skip on windows or
    > uncommon platforms. I think we could do the same for these.
    
    For a project intended to improve on the status quo it seems like a too low bar to just port over the deficincies in the thing we’re trying to improve over.
    
    ./daniel
    
    
    

25. Re: RFC: adding pytest as a supported test framework

    Daniele Varrazzo <daniele.varrazzo@gmail.com> — 2024-06-12T16:46:23Z

    On Wed, 12 Jun 2024 at 18:08, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    >
    > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote:
    > > > The OAuth pytest suite makes extensive use of
    > > > - psycopg, to easily drive libpq;
    > >
    > > That's probably not going to fly. It introduces painful circular dependencies
    > > between building postgres (for libpq), building psycopg (requiring libpq) and
    > > testing postgres (requiring psycopg).
    >
    > psycopg has a few implementations binary, c, & pure python. The pure
    > python one can be linked to a specific libpq.so file at runtime[1]. As
    
    This is true, but [citation needed] :D I assume the pointer wanted to
    be https://www.psycopg.org/psycopg3/docs/api/pq.html#pq-impl
    
    I see the following use cases and how I would use psycopg to implement them:
    
    - by installing 'psycopg[binary]' you would get a binary bundle
    shipping with a stable version of the libpq, so you can test the
    database server regardless of libpq instabilities in the same
    codebase.
    - using the pure Python psycopg (enforced by exporting
    'PSYCOPG_IMPL=python') you would use the libpq found on the
    LD_LIBRARY_PATH, which can be useful to test regressions to the libpq
    itself.
    - if you want to test new libpq functions you can reach them in Python
    by dynamic lookup. See [2] for an example of a function only available
    from libpq v17.
    
    [2]: https://github.com/psycopg/psycopg/blob/2bf7783d66ab239a2fa330a842fd461c4bb17c48/psycopg/psycopg/pq/_pq_ctypes.py#L564-L569
    
    -- Daniele
    
    
    
    

26. Re: RFC: adding pytest as a supported test framework

    Alexander Korotkov <aekorotkov@gmail.com> — 2024-06-12T17:11:55Z

    On Wed, Jun 12, 2024 at 7:34 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
    > On Wed, Jun 12, 2024 at 7:08 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote:
    > > > > The OAuth pytest suite makes extensive use of
    > > > > - psycopg, to easily drive libpq;
    > > >
    > > > That's probably not going to fly. It introduces painful circular dependencies
    > > > between building postgres (for libpq), building psycopg (requiring libpq) and
    > > > testing postgres (requiring psycopg).
    > > >
    > > > You *can* solve such issues, but we've debated that in the past, and I doubt
    > > > we'll find agreement on the awkwardness it introduces.
    > >
    > > psycopg has a few implementations binary, c, & pure python. The pure
    > > python one can be linked to a specific libpq.so file at runtime[1]. As
    > > long as we don't break the libpq API (which we shouldn't), we can just
    > > point it to the libpq compiled by meson/make. We wouldn't be able to
    > > use the newest libpq features that way (because psycopg wouldn't know
    > > about them), but that seems totally fine for most usages (i.e. sending
    > > a query over a connection). If we really want to use those from the
    > > python tests we could write our own tiny CFFI layer specifically for
    > > those.
    >
    > I guess you mean pg8000. Note that pg8000 and psycopg2 have some
    > differences in interpretation of datatypes (AFAIR arrays, jsonb...).
    > So, it would be easier to chose one particular driver.  However, with
    > a bit efforts it's possible to make all the code driver agnostic.
    
    Ops, this is probably outdated due to presence of psycopg3, as pointed
    by Daniele Varrazzo [1].
    
    Links.
    1. https://www.postgresql.org/message-id/CA%2Bmi_8Zj0gpzPKUEcEx2mPOAsm0zPvznhbcnQDA_eeHVnVqg9Q%40mail.gmail.com
    
    ------
    Regards,
    Alexander Korotkov
    Supabase
    
    
    
    

27. Re: RFC: adding pytest as a supported test framework

    Daniel Gustafsson <daniel@yesql.se> — 2024-06-12T17:30:33Z

    > On 12 Jun 2024, at 17:50, Andres Freund <andres@anarazel.de> wrote:
    > On 2024-06-11 07:28:23 -0700, Jacob Champion wrote:
    
    >> The OAuth pytest suite makes extensive use of
    >> - psycopg, to easily drive libpq;
    > 
    > That's probably not going to fly. It introduces painful circular dependencies
    > between building postgres (for libpq), building psycopg (requiring libpq) and
    > testing postgres (requiring psycopg).
    
    I might be missing something obvious, but if we use a third-party libpq driver
    in the testsuite doesn't that imply that a patch adding net new functionality
    to libpq also need to add it to the driver in order to write the tests?  I'm
    thinking about the SCRAM situation a few years back when drivers weren't up to
    date.
    
    --
    Daniel Gustafsson
    
    
    
    
    

28. Re: RFC: adding pytest as a supported test framework

    Daniele Varrazzo <daniele.varrazzo@gmail.com> — 2024-06-12T17:38:54Z

    On Wed, 12 Jun 2024 at 19:30, Daniel Gustafsson <daniel@yesql.se> wrote:
    
    > I might be missing something obvious, but if we use a third-party libpq driver
    > in the testsuite doesn't that imply that a patch adding net new functionality
    > to libpq also need to add it to the driver in order to write the tests?  I'm
    > thinking about the SCRAM situation a few years back when drivers weren't up to
    > date.
    
    As Jelte pointed out, new libpq functions can be tested via CFFI. I
    posted a practical example in a link upthread (pure Python Psycopg is
    entirely implemented on FFI).
    
    -- Daniele
    
    
    
    

29. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-12T19:07:16Z

    On Wed, Jun 12, 2024 at 1:30 PM Daniel Gustafsson <daniel@yesql.se> wrote:
    > > On 12 Jun 2024, at 17:50, Andres Freund <andres@anarazel.de> wrote:
    > > On 2024-06-11 07:28:23 -0700, Jacob Champion wrote:
    >
    > >> The OAuth pytest suite makes extensive use of
    > >> - psycopg, to easily drive libpq;
    > >
    > > That's probably not going to fly. It introduces painful circular dependencies
    > > between building postgres (for libpq), building psycopg (requiring libpq) and
    > > testing postgres (requiring psycopg).
    >
    > I might be missing something obvious, but if we use a third-party libpq driver
    > in the testsuite doesn't that imply that a patch adding net new functionality
    > to libpq also need to add it to the driver in order to write the tests?  I'm
    > thinking about the SCRAM situation a few years back when drivers weren't up to
    > date.
    
    Yeah, I don't think depending on psycopg2 is practical at all. We can
    either shell out to psql like we do now, or we can use something like
    CFFI.
    
    On the overall topic of this thread, I personally find most of the
    rationale articulated in the original message unconvincing. Many of
    those improvements could be made on top of the Perl framework we have
    today, and some of them have been discussed, but nobody's done the
    work. I also don't understand the argument that assert a == b is some
    new and wonderful thing; I mean, you can already do is($a,$b,"test
    name") which *also* shows you the values when they don't match, and
    includes a test name, too! I personally think that most of the
    frustration associated with writing TAP tests has to do with (1)
    Windows behavior being randomly different than on other platforms in
    ways that are severely under-documented, (2)
    PostgreSQL::Test::whatever being somewhat clunky and under-designed,
    and (3) the general difficulty of producing race-free test cases. A
    new test framework isn't going to solve (3), and (1) and (2) could be
    fixed anyway.
    
    However, I understand that a lot of people would prefer to code in
    Python than in Perl. I am not one of them: I learned Perl in the early
    nineties, and I haven't learned Python yet. Nonetheless, Python being
    more popular than Perl is a reasonable reason to consider allowing its
    use in PostgreSQL. But if that's the reason, let's be up front about
    it.
    
    I do think we want a scripting language here i.e. not C.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

30. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-12T19:30:53Z

    On Tue, Jun 11, 2024 at 4:48 PM Noah Misch <noah@leadboat.com> wrote:
    > If we're going to test in a non-Perl language, I'd pick C over Python.
    
    We already test in C, though? If the complaint is that those tests are
    driven by Perl, I agree that something like libcheck or GTest or
    whatever people are using nowadays would be nicer. But that can be
    added at any point; the activation energy for a new C-based test
    runner seems pretty small. IMO, there's no reason to pick it _over_
    another language, when we already support C tests and agree that
    developers need to be fluent in C.
    
    > We'd need a library like today's Perl
    > PostgreSQL::Test to make C-language tests nice, but the same would apply to
    > any new language.
    
    I think the effort required in rebuilding end-to-end tests in C is
    going to be a lot different than in pretty much any other modern
    high-level language, so I don't agree that "the same would apply".
    
    For the five problem statements I put forward, I think C moves the
    needle for zero of them. It neither solves the problems we have nor
    gives us stronger tools to solve them ourselves. And for my personally
    motivating use case of OAuth, where I need to manipulate HTTP and JSON
    and TLS and so on and so forth, implementing all of that in C would be
    an absolute nightmare. Given that choice, I would rather use Perl --
    and that's saying something, because I like C a lot more than I like
    Perl -- because it's the difference between being given a rusty but
    still functional table saw, and being given a box of Legos to build a
    "better" table saw, when all I want to do is cut a 2x4 in half and
    move on with my work.
    
    I will use the rusty saw if I have to. But I want to get a better saw
    -- that somebody else with experience in making saws has constructed,
    and other people are pretty happy with -- as opposed to building my
    own.
    
    > I also want the initial scope to be the new language coexisting with the
    > existing Perl tests.
    
    Strongly agreed.
    
    Thanks,
    --Jacob
    
    
    
    

31. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-12T19:31:23Z

    On Wed, Jun 12, 2024 at 4:40 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable
    > choices for me (and possibly some more). They allow some type of
    > introspection, they have a garbage collector, and their general
    > tooling is quite good.
    >
    > But I think a dynamically typed scripting language is much more
    > fitting for writing tests like this. I love static typing for
    > production code, but imho it really doesn't have much benefit for
    > tests.
    
    +1. I write mostly protocol mocks and glue code in my authn testing,
    to try to set up the system into some initial state and then break it.
    Of the languages mentioned here, I've only used C#, Java, and Go. If I
    had to reimplement my tests, I'd probably reach for Go out of all of
    those, but the glue would still be more painful than it probably needs
    to be.
    
    > As scripting languages go, the ones that are still fairly heavily in
    > use are Javascript, Python, Ruby, and PHP. I think all of those could
    > probably work, but my personal order of preference would be Python,
    > Ruby, Javascript, PHP.
    
    - Python is the easiest language I've personally used to glue things
    together, bar none.
    - I like Ruby as a language but have no experience using it for
    testing. (RSpec did come up during the unconference session and
    subsequent hallway conversations.)
    - Javascript is a completely different mental model from what we're
    used to, IMO. I think we're likely to spend a lot of time fighting the
    engine unless everyone is very clear on how it works.
    - I don't see a use case for PHP here.
    
    > TO CLARIFY: This thread is not a proposal to replace Perl with Python.
    > It's a proposal to allow people to also write tests in Python.
    
    +1. It doesn't need to replace anything. It just needs to help us do
    more things than we're currently doing.
    
    --Jacob
    
    
    
    

32. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-12T19:31:35Z

    On Wed, Jun 12, 2024 at 4:48 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
    > Generally, testgres was initially designed as Python analogue of what
    > we have in src/test/perl/PostgreSQL/Test.  In particular its
    > testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster.  It
    > comes under PostgreSQL License.  So, I wonder if we could revise it
    > and fetch most part of it into our source tree.
    
    Okay. If there's wide interest in a port of PostgreSQL::Test::Cluster,
    that might be something to take a look at. (Since I'm focused on
    testing things that the current Perl suite can't do at all, I would
    probably not be the first to volunteer.)
    
    --Jacob
    
    
    
    

33. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-12T19:33:53Z

    On Wed, Jun 12, 2024 at 8:50 AM Andres Freund <andres@anarazel.de> wrote:
    > I think I might have formulated my paragraph above badly - I didn't mean that
    > we should move away from perl tests tomorrow, but that we need a path forward
    > that allows folks to write tests without perl.
    
    Okay, agreed.
    
    > > - Tests aren't cheap, but in my experience, the maintenance-cost math
    > > for tests is a lot different than the math for implementations.
    >
    > At the moment they tend to be *more* expensive often, due to spurious
    > failures. That's mostly not perl's fault, don't get me wrong, but us not
    > having better infrastructure for testing complicated behaviour and/or testing
    > things on a more narrow basis.
    
    Well, okay, but I'm not sure how to respond to this in the frame of
    this discussion. Bad tests will continue to exist. I am trying to add
    a tool that, in my view, has made it easier for me to test complicated
    behavior than what we currently have. I can't prove that it will solve
    other issues too.
    
    > > Well, scopes are pretty front and center when you start building
    > > pytest fixtures, and the complicated longer setups will hopefully
    > > converge correctly early on and be reused everywhere else. I imagine
    > > no one wants to build cluster setup from scratch.
    >
    > One (the?) prime source of state in our tap tests is the
    > database. Realistically we can't just tear that one down and reset it between
    > tests without causing the test times to explode. So we'll have to live with
    > some persistent state.
    
    Yes? If I've given the impression that I disagree, sorry; I agree.
    
    > > On a slight tangent, is this not a problem today?
    >
    > It is, but that doesn't mean making it even bigger is unproblematic :)
    
    Given all that's been said, I don't understand why you think the
    problem would get bigger. We would cache expensive state that we need,
    including the cluster, and pytest lets us do that, and my test suite
    does that. I've never written a suite that spun up a separate cluster
    for every single test and then immediately threw it away.
    
    (If you want to _enable_ that behavior, to test in extreme isolation,
    then pytest lets you do that too. But it's not something we should do
    by default.)
    
    > > We do a lot more acceptance testing than internal testing, which came
    > > up as a major complaint from me and others during the unconference.
    > > One of the reasons people avoid writing internal tests in Perl is
    > > because it's very painful to find a rhythm with Test::More.
    >
    > What definition of internal tests are you using here?
    
    There's a spectrum from unit-testing unexported functions all the way
    to end-to-end acceptance, and personally I believe that anything
    finer-grained than end-to-end acceptance is unnecessarily painful. My
    OAuth suite sits somewhere in the middle, where it mocks the protocol
    layer and can test the client and server as independent pieces. Super
    useful for OAuth, which is asymmetrical.
    
    I'd like to additionally see better support for unit tests of backend
    internals, but I don't know those seams as well as all of you do and I
    should not be driving that. I don't think Python will necessarily help
    you with it. But it sure helped me break apart the client and the
    server while enjoying the testing process, and other people want to do
    that too, so that's what I'm pushing for.
    
    > I think a lot of our tests are complicated, fragile and slow because we almost
    > exclusively do end-to-end tests, because with a few exceptions we don't have a
    > way to exercise code in a more granular way.
    
    Yep.
    
    > That's probably not going to fly. It introduces painful circular dependencies
    > between building postgres (for libpq), building psycopg (requiring libpq) and
    > testing postgres (requiring psycopg).
    
    I am trying very hard not to drag that, which I understand is
    controversial and is in no way a linchpin of my proposal, into the
    discussion of whether or not we should try supporting pytest.
    
    I get it; I understand that the circular dependency is weird; there
    are alternatives if it's unacceptable; none of that has anything to do
    with Python+pytest.
    
    > One thing worth thinking about is that such dependencies have to work on a
    > relatively large number of platforms / architectures. A lot of projects
    > don't...
    
    Agreed.
    
    Thanks,
    --Jacob
    
    
    
    

34. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-12T19:34:04Z

    On Wed, Jun 12, 2024 at 10:30 AM Daniel Gustafsson <daniel@yesql.se> wrote:
    > I might be missing something obvious, but if we use a third-party libpq driver
    > in the testsuite doesn't that imply that a patch adding net new functionality
    > to libpq also need to add it to the driver in order to write the tests?
    
    I use the third-party driver to perform the "basics" at a high level
    -- connections, queries during cluster setup, things that don't
    involve ABI changes. For new ABI I use ctypes, or as other people have
    mentioned CFFI would work.
    
    --Jacob
    
    
    
    

35. Re: RFC: adding pytest as a supported test framework

    Noah Misch <noah@leadboat.com> — 2024-06-12T20:56:57Z

    On Wed, Jun 12, 2024 at 01:40:30PM +0200, Jelte Fennema-Nio wrote:
    > On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote:
    > > I also want the initial scope to be the new language coexisting with the
    > > existing Perl tests.  If a bulk translation ever happens, it should happen
    > > long after the debut of the new framework.  That said, I don't much trust a
    > > human-written bulk language translation to go through without some tests
    > > accidentally ceasing to test what they test in Perl today.
    > 
    > I definitely don't think we should rewrite all the tests that we have
    > in Perl today into some other language. But I do think that whatever
    > language we choose, that language should make it as least as easy to
    > write tests, as easy to read them and as easy to see that they are
    > testing the intended thing, as is currently the case for Perl.
    > Rewriting a few Perl tests into the new language, even if not merging
    > the rewrite, is a good way of validating that imho.
    
    Agreed.
    
    > PS. For PgBouncer I actually hand-rewrote all the tests that we had in
    > bash (which is the worst testing language ever) in Python and doing so
    > actually found more bugs in PgBouncer code that our bash tests
    > wouldn't catch. So it's not necessarily the case that you lose
    > coverage by rewriting tests.
    
    Yep.
    
    
    
    

36. Re: RFC: adding pytest as a supported test framework

    Sutou Kouhei <kou@clear-code.com> — 2024-06-12T21:40:01Z

    Hi,
    
    (I don't have an opinion which language should be selected
    here.)
    
    In <CAOYmi+mA7-uNqpY-0jNZY=fE-QsbfeM1j5Mc-vu1Xm+=B8NOXA@mail.gmail.com>
      "Re: RFC: adding pytest as a supported test framework" on Wed, 12 Jun 2024 12:31:23 -0700,
      Jacob Champion <jacob.champion@enterprisedb.com> wrote:
    
    > - I like Ruby as a language but have no experience using it for
    > testing. (RSpec did come up during the unconference session and
    > subsequent hallway conversations.)
    
    If we want to select Ruby, I can help. (I'm a Ruby committer
    and a maintainer of a testing framework bundled in Ruby.)
    
    I'm using Ruby for PGroonga's tests that can't be covered by
    pg_regress. For example, streaming replication related
    tests. PGroonga has a small utility for it:
    https://github.com/pgroonga/pgroonga/blob/main/test/helpers/sandbox.rb
    
    Here is a streaming replication test with it:
    https://github.com/pgroonga/pgroonga/blob/main/test/test-streaming-replication.rb
    
    I'm using test-unit as testing framework that is bundled in
    Ruby: https://github.com/test-unit/test-unit/
    
    I don't recommend that we use RSpec as testing framework
    even if we select Ruby. RSpec may change API. (RSpec did it
    several times in the past.) If testing framework changes API, we
    need to rewrite our tests to adapt the change.
    
    I'll never change test-unit API because I don't want to
    rewrite existing tests.
    
    
    Thanks,
    -- 
    kou
    
    
    
    

37. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-12T21:50:15Z

    On Wed, 12 Jun 2024 at 18:46, Daniele Varrazzo
    <daniele.varrazzo@gmail.com> wrote:
    > This is true, but [citation needed] :D I assume the pointer wanted to
    > be https://www.psycopg.org/psycopg3/docs/api/pq.html#pq-impl
    
    Ugh, yes I definitely meant to add a link to that [1]. I meant this one though:
    
    [1]: https://www.psycopg.org/psycopg3/docs/basic/install.html#pure-python-installation
    
    > - using the pure Python psycopg (enforced by exporting
    > 'PSYCOPG_IMPL=python') you would use the libpq found on the
    > LD_LIBRARY_PATH, which can be useful to test regressions to the libpq
    > itself.
    
    This indeed was the main idea I had in mind.
    
    > - if you want to test new libpq functions you can reach them in Python
    > by dynamic lookup. See [2] for an example of a function only available
    > from libpq v17.
    >
    > [2]: https://github.com/psycopg/psycopg/blob/2bf7783d66ab239a2fa330a842fd461c4bb17c48/psycopg/psycopg/pq/_pq_ctypes.py#L564-L569
    
    Yeah, that dynamic lookup would work. But due to the cyclic dependency
    on postgres commit vs psycopg PR we couldn't depend on psycopg for
    those dynamic lookups. So we'd need to have our own dynamic lookup
    code to do this.
    
    I don't see a huge problem with using psycopg for already released
    commonly used features (i.e. connecting to postgres and doing
    queries), but still use our own custom dynamic lookup for the rare
    tests that test newly added features. But I can definitely see people
    making the argument that if we need to write & maintain some dynamic
    lookup code ourselves anyway, we might as well have all the dynamic
    lookup code in our repo to avoid dependencies.
    
    On Wed, 12 Jun 2024 at 18:46, Daniele Varrazzo
    <daniele.varrazzo@gmail.com> wrote:
    >
    > On Wed, 12 Jun 2024 at 18:08, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > >
    > > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote:
    > > > > The OAuth pytest suite makes extensive use of
    > > > > - psycopg, to easily drive libpq;
    > > >
    > > > That's probably not going to fly. It introduces painful circular dependencies
    > > > between building postgres (for libpq), building psycopg (requiring libpq) and
    > > > testing postgres (requiring psycopg).
    > >
    > > psycopg has a few implementations binary, c, & pure python. The pure
    > > python one can be linked to a specific libpq.so file at runtime[1]. As
    >
    > This is true, but [citation needed] :D I assume the pointer wanted to
    > be https://www.psycopg.org/psycopg3/docs/api/pq.html#pq-impl
    >
    > I see the following use cases and how I would use psycopg to implement them:
    >
    > - by installing 'psycopg[binary]' you would get a binary bundle
    > shipping with a stable version of the libpq, so you can test the
    > database server regardless of libpq instabilities in the same
    > codebase.
    > - using the pure Python psycopg (enforced by exporting
    > 'PSYCOPG_IMPL=python') you would use the libpq found on the
    > LD_LIBRARY_PATH, which can be useful to test regressions to the libpq
    > itself.
    > - if you want to test new libpq functions you can reach them in Python
    > by dynamic lookup. See [2] for an example of a function only available
    > from libpq v17.
    >
    > [2]: https://github.com/psycopg/psycopg/blob/2bf7783d66ab239a2fa330a842fd461c4bb17c48/psycopg/psycopg/pq/_pq_ctypes.py#L564-L569
    >
    > -- Daniele
    
    
    
    

38. Re: RFC: adding pytest as a supported test framework

    Melanie Plageman <melanieplageman@gmail.com> — 2024-06-12T22:34:16Z

    On Tue, Jun 11, 2024 at 8:05 AM Andrew Dunstan <andrew@dunslane.net> wrote:
    >
    >
    > On 2024-06-10 Mo 21:49, Andres Freund wrote:
    >
    > Hi,
    >
    > On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
    >
    > I'm not sure what part of the testing infrastructure you think is
    > unmaintained. For example, the last release of Test::Simple was all the way
    > back on April 25.
    >
    > IPC::Run is quite buggy and basically just maintained by Noah these days.
    >
    >
    > Yes, that's true. I think the biggest pain point is possibly the recovery tests.
    >
    > Some time ago I did some work on wrapping libpq using the perl FFI module. It worked pretty well, and would mean we could probably avoid many uses of IPC::Run, and would probably be substantially more efficient (no fork required). It wouldn't avoid all uses of IPC::Run, though.
    >
    > But my point was mainly that while a new framework might have value, I don't think we need to run out and immediately rewrite several hundred TAP tests. Let's pick the major pain points and address those.
    
    FWIW, I felt a lot of pain trying to write recovery TAP tests with
    IPC::Run's pumping functionality. It was especially painful (as
    someone who knows even less Perl than the "street fighting Perl"
    Thomas Munro has described having) before the additional test
    infrastructure was added in BackgroudPsql.pm last year. As an example
    of the "worst case", it took me two full work days to go from a repro
    (with psql sessions on a primary and replica node) of the vacuum hang
    issue being explored in [1] to a sort-of working TAP test which
    demonstrated it - and that was with help from several other
    committers. Granted, this is a complex case.
    
    A small part of the issue is that, as Tristan has said elsewhere,
    there aren't good developer tool integrations that I know about for
    Perl. I use neovim's LSP support for C and Python (in other projects),
    and there is a whole ecosystem of tools I can use for both C and
    Python. I know not everyone likes or needs these, but I find that they
    help me write and debug code faster.
    
    I had offered to take a stab at writing some of the BackgroundPsql
    test infrastructure in Python. I haven't started exploring that yet or
    looking at what Jacob has done so far, but I am optimistic that this
    is an area where it is worth seeing what is available to us outside of
    IPC::Run.
    
    - Melanie
    
    [1] https://www.postgresql.org/message-id/CAAKRu_bXH2g_pchG7rN_4fs-_6_kVbbJ97gYRoN0Zdb9P04Wag%40mail.gmail.com
    
    
    
    

39. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-12T22:43:07Z

    On Wed, 12 Jun 2024 at 21:07, Robert Haas <robertmhaas@gmail.com> wrote:
    > Yeah, I don't think depending on psycopg2 is practical at all. We can
    > either shell out to psql like we do now, or we can use something like
    > CFFI.
    
    Quick clarification I meant psycopg3, not psycopg2. And I'd very much
    like to avoid using psql for sending queries, luckily CFFI in python
    is very good.
    
    > Many of
    > those improvements could be made on top of the Perl framework we have
    > today, and some of them have been discussed, but nobody's done the
    > work.
    
    I agree it's not a technical issue. It is a people issue. There are
    very few people skilled in Perl active in the community. And most of
    those are very senior hackers that have much more important things to
    do that make our Perl testing framework significantly better. And the
    less senior people that might see improving tooling as a way to get
    help out in the community, are try to stay away from Perl with a 10
    foot pole. So the result is, nothing gets improved. Especially since
    very few people outside our community improve this tooling either.
    
     I also don't understand the argument that assert a == b is some
    > new and wonderful thing; I mean, you can already do is($a,$b,"test
    > name") which *also* shows you the values when they don't match, and
    > includes a test name, too!
    
    Sure you can, if you know the function exists. And clearly not
    everyone knows that it exists, as the quick grep below demonstrates:
    
    ❯ grep 'ok(.* == .*' **.pl | wc -l
    41
    
    But apart from the obvious syntax doing what you want, the output is
    also much better when looking at a slightly more complex case. With
    the following code:
    
    def some_returning_func():
        return 1234
    
    def some_func(val):
        if val > 100:
            return 100
        return val
    
    def test_mytest():
        assert some_func(some_returning_func()) == 50
    
    Pytest will show the following output
    
        def test_mytest():
    >       assert some_func(some_returning_func()) == 50
    E       assert 100 == 50
    E        +  where 100 = some_func(1234)
    E        +    where 1234 = some_returning_func()
    
    I have no clue how you could get output that's even close to that
    clear with Perl.
    
    Another problem I run into is that, as you probably know, sometimes
    you need to look at the postgres logs to find out what actually went
    wrong. Currently the only way to find them (for me) is following the
    following steps: hmm, let me figure out what that directory was called
    again... ah okay it is build/testrun/pg_upgrade/001_basic/... okay
    let's start opening log files that all have very similar names until
    find the right one.
    
    When a test in pytest fails it automatically outputs all stdout/stderr
    that was outputted, and hides it on success. So for the PgBouncer test
    suite. I simply send all the relevant log files to stdout, prefixed by
    some capitalized identifying line with a few newlines around it.
    Something like "PG_LOG: /path/to/actual/logfile". Then when a test
    fails in my terminal window I can look at the files related to the
    failed test instantly. This allows me to debug failures much faster.
    
    A related thing that also doesn't help at all is that (afaik) seeing
    any of the perl tap test output in your terminal requires running
    `meson test` with the -v option, and then scrolling up past all the
    super verbose output of successfully passing tests to find out what
    exactly failed in the single test that failed. And if you don't want
    to do that you have to navigate to the magic directory path (
    build/testrun/pg_upgrade/001_basic/) of the specific tests to look at
    the stdout file there... Which then turns out not to even be there if
    you actually had a compilation failure in your perl script (which
    happens very often to anyone that doesn't use perl often). So now you
    have to scroll up anyway.
    
    Pytest instead is very good at only showing output for the tests that
    failed, and hiding pretty much all output for the tests that passed.
    
    
    
    

40. Re: RFC: adding pytest as a supported test framework

    Daniel Gustafsson <daniel@yesql.se> — 2024-06-13T11:27:42Z

    > On 13 Jun 2024, at 00:34, Melanie Plageman <melanieplageman@gmail.com> wrote:
    
    > FWIW, I felt a lot of pain trying to write recovery TAP tests with
    > IPC::Run's pumping functionality. It was especially painful (as
    > someone who knows even less Perl than the "street fighting Perl"
    > Thomas Munro has described having) before the additional test
    > infrastructure was added in BackgroudPsql.pm last year.
    
    A key aspect of this, which isn't specific to Perl or our use of it, is that
    this was done in backbranches which doesn't have the (recently) much improved
    BackgroundPsql.pm.  The quality of our tools and the ease of use they provide
    is directly related to the investment we make into continuously improving our
    testharness.  Regardless of which toolset we adopt, if we don't make this
    investment (taking learnings from the past years and toolsets into account)
    we're bound to repeat this thread in a few years advocating for toolset X+1.
    
    --
    Daniel Gustafsson
    
    
    
    
    

41. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T13:38:24Z

    On Wed, Jun 12, 2024 at 6:43 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > I agree it's not a technical issue. It is a people issue. There are
    > very few people skilled in Perl active in the community. And most of
    > those are very senior hackers that have much more important things to
    > do that make our Perl testing framework significantly better. And the
    > less senior people that might see improving tooling as a way to get
    > help out in the community, are try to stay away from Perl with a 10
    > foot pole. So the result is, nothing gets improved. Especially since
    > very few people outside our community improve this tooling either.
    
    I agree with you, but I'm skeptical that solving it will be as easy as
    switching to Python. For whatever reason, it seems like every piece of
    infrastructure that the PostgreSQL community has suffers from severe
    neglect. Literally everything I know of either has one or maybe two
    very senior hackers maintaining it, or no maintainer at all. Andrew
    maintains the buildfarm and it evolves quite slowly. Andres did all
    the work on meson, with some help from Peter. Thomas maintains cfbot
    as a skunkworks. The Perl-based TAP test framework gets barely any
    love at all. The CommitFest application is pretty much totally
    stagnant, and in fact is a great example of what I'm talking about
    here: I wrote an original version in Perl and somebody -- I think
    Magnus -- rewrote it in a more maintainable framework -- and then the
    development pace went to basically zero. All of this stuff is critical
    project infrastructure and yet it feels like nobody wants to work on
    it.
    
    Now, this case may prove to be an exception to that rule and that will
    be great. But what I think is a lot more likely is that we'll get a
    lot of pressure to commit something as soon as parity with the Perl
    TAP test system has been achieved, or maybe even before that, and then
    the rate of further improvements will slow to a trickle. That's not to
    say that sticking with Perl is better. A quick Google search finds a
    web page that says Python is two orders of magnitude more popular than
    Perl, and that's not something we should just ignore. But I still
    think it's fair to question whether the preference of many developers
    for Python over Perl will translate into sustained investment in
    improving the infrastructure. Again, I will be thrilled if it does,
    but that just doesn't seem to be the way that things go around here,
    and I bet the reasons go well beyond choice of programming language.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

42. Re: RFC: adding pytest as a supported test framework

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-06-13T15:19:47Z

    Robert Haas <robertmhaas@gmail.com> writes:
    > On Wed, Jun 12, 2024 at 6:43 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    >> I agree it's not a technical issue. It is a people issue. There are
    >> very few people skilled in Perl active in the community. And most of
    >> those are very senior hackers that have much more important things to
    >> do that make our Perl testing framework significantly better. And the
    >> less senior people that might see improving tooling as a way to get
    >> help out in the community, are try to stay away from Perl with a 10
    >> foot pole. So the result is, nothing gets improved. Especially since
    >> very few people outside our community improve this tooling either.
    
    > I agree with you, but I'm skeptical that solving it will be as easy as
    > switching to Python. For whatever reason, it seems like every piece of
    > infrastructure that the PostgreSQL community has suffers from severe
    > neglect.
    
    Yeah.  In this case it's perhaps more useful to look at our external
    dependencies, the large majority of which are suffering from age
    and neglect:
    
      * autoconf & gmake (although meson may get us out from under these)
      * bison
      * flex
      * perl
      * tcl
      * regex library (basically from tcl)
      * libxml2
      * kerberos
      * ldap
      * pam
      * uuid library
    
    I think the basic problem is inherent in being a successful long-lived
    project.  Or maybe we're just spectacularly bad at picking which
    things to depend on.  Whichever it is, we'd better have a 10- or 20-
    year perspective when thinking about adopting new major dependencies.
    
    In the case at hand, I share Robert's doubts about Python.  Sure it's
    more popular than Perl, but I don't think it's actually better, and
    in some ways it's worse.  (The moving-target package collection was
    mentioned as a problem, for instance.)  Is it going to age better
    than Perl?  Doubt it.
    
    I wonder if we should be checking out some of the other newer
    languages that were mentioned upthread.  It feels like going to
    Python here will lead to having two testing infrastructures with
    mas-o-menos the same capabilities, leaving us with a situation
    where people have to know both languages in order to make sense of
    our test suite.  I find it hard to picture that as an improvement
    over the status quo.
    
    			regards, tom lane
    
    
    
    

43. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-13T17:07:58Z

    On Thu, 13 Jun 2024 at 15:38, Robert Haas <robertmhaas@gmail.com> wrote:
    > For whatever reason, it seems like every piece of
    > infrastructure that the PostgreSQL community has suffers from severe
    > neglect. Literally everything I know of either has one or maybe two
    > very senior hackers maintaining it, or no maintainer at all. Andrew
    > maintains the buildfarm and it evolves quite slowly. Andres did all
    > the work on meson, with some help from Peter. Thomas maintains cfbot
    > as a skunkworks. The Perl-based TAP test framework gets barely any
    > love at all. The CommitFest application is pretty much totally
    > stagnant, and in fact is a great example of what I'm talking about
    > here: I wrote an original version in Perl and somebody -- I think
    > Magnus -- rewrote it in a more maintainable framework -- and then the
    > development pace went to basically zero. All of this stuff is critical
    > project infrastructure and yet it feels like nobody wants to work on
    > it.
    
    Overall, I agree with the sentiment of us not maintaining our tooling
    well (although I think meson maintenance has been pretty decent so
    far). I think there's a bunch of reasons for this (not all apply to
    each of the tools):
    1. pretty much solely maintained by senior community members who don't
    have time to maintain it
    2. no clear way to contribute. e.g. where should I send a patch/PR for
    the commitfest app, or the cfbot?
    3. (related to 1) unresponsive when somehow contributions are actually
    sent in (I have two open PRs on the cfbot app from 3 years ago without
    any response)
    
    I think 1 & 3 could be addressed by more easily giving commit/merge
    access to these tools than to the main PG repo. And I think 2 could be
    addressed by writing on the relevant wiki page where to go, and
    probably putting a link to the wiki page on the actual website of the
    tool.
    
    But Perl is at the next level of unmaintained infrastructure. It is
    actually clear how you can contribute to it, but still no new
    community members actually want to contribute to it. Also, it's not
    only unmaintained by us but it's also pretty much unmaintained by the
    upstream community.
    
    > But I still
    > think it's fair to question whether the preference of many developers
    > for Python over Perl will translate into sustained investment in
    > improving the infrastructure. Again, I will be thrilled if it does,
    > but that just doesn't seem to be the way that things go around here,
    > and I bet the reasons go well beyond choice of programming language.
    
    As you said, no one in our community wants to maintain our testsuite
    full time. But our test suite consists partially of upstream
    dependencies and partially of our own code. Right now pretty much
    no-one improves the ustream code, and pretty much no-one improves our
    own code. Using a more modern language gives up much more frequent
    upstream improvements for free, and it will allow new community
    members to contribute to our own test suite.
    
    And I understand you are sceptical that people will contribute to our
    own test suite, just because it's Python. But as a counterpoint:
    people are currently already doing exactly that, just outside of the
    core postgres repo[1][2][3]. I don't see why those people would
    suddenly stop doing that if we include such a suite in the official
    repo. Apparently many people hate writing tests in Perl so much that
    they'd rather build Python test frameworks to test their extensions,
    than to use/improve the Perl testing framework included in Postgres.
    
    [1]: https://github.com/pgbouncer/pgbouncer/tree/master/test
    [2]: https://github.com/jchampio/pg-pytest-suite
    [3]: https://github.com/postgrespro/testgres
    
    
    PS. I don't think it makes sense to host our tooling like the
    commitfest app on our own git server instead of github/gitlab. That
    only makes it harder for community members to contribute and also much
    harder to set up CI. I understand the reasons why we use mailing lists
    for the development of core postgres, but I don't think those apply
    nearly as much to our tooling repos. And honestly also not to stuff
    like the website.
    
    
    
    

44. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-13T17:28:00Z

    On Thu, 13 Jun 2024 at 17:19, Tom Lane <tgl@sss.pgh.pa.us> wrote:
    > I wonder if we should be checking out some of the other newer
    > languages that were mentioned upthread.
    
    If this is actually something that we want to seriously evaluate, I
    think that's a significant effort. And I think the people that want a
    language would need to step in to make that effort. So far Jacob[1],
    Alexander[2] and me[3] seem to be doing that for Python, and Sutou has
    done that for Ruby[4].
    
    [1]: https://github.com/pgbouncer/pgbouncer/tree/master/test
    [2]: https://github.com/jchampio/pg-pytest-suite
    [3]: https://github.com/postgrespro/testgres
    [4]: https://github.com/pgroonga/pgroonga/blob/main/test/test-streaming-replication.rb
    
    > It feels like going to
    > Python here will lead to having two testing infrastructures with
    > mas-o-menos the same capabilities, leaving us with a situation
    > where people have to know both languages in order to make sense of
    > our test suite.  I find it hard to picture that as an improvement
    > over the status quo.
    
    You don't have to be fluent in writing Python to be able to read and
    understand tests written in it. As someone familiar with Python I can
    definitely read our test suite, and I expect everyone smart enough to
    be fluent in Perl to be able to read and understand Python with fairly
    little effort too.
    
    I think having significantly more tests being written, and those tests
    being written faster and more correctly, is definitely worth the
    slight mental effort of learning to read two very similarly looking
    scripting languages (they both pretty much looking like pseudo code).
    
    
    
    

45. Re: RFC: adding pytest as a supported test framework

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-06-13T17:47:53Z

    Jelte Fennema-Nio <postgres@jeltef.nl> writes:
    > You don't have to be fluent in writing Python to be able to read and
    > understand tests written in it.
    
    [ shrug... ]  I think the same can be said of Perl, with about as
    much basis.  It matters a lot if you have previous experience with
    the language.
    
    			regards, tom lane
    
    
    
    

46. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T17:51:11Z

    On Thu, Jun 13, 2024 at 11:19 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
    > I wonder if we should be checking out some of the other newer
    > languages that were mentioned upthread.  It feels like going to
    > Python here will lead to having two testing infrastructures with
    > mas-o-menos the same capabilities, leaving us with a situation
    > where people have to know both languages in order to make sense of
    > our test suite.  I find it hard to picture that as an improvement
    > over the status quo.
    
    As I see it, one big problem is that if you pick a language that's too
    new, it's more likely to fade away. Python is very well-established,
    e.g. see
    
    https://www.tiobe.com/tiobe-index/
    
    That gives Python a rating of 15.39%; vs. Perl at 0.69%. There are
    other things that you could pick, for sure, like Javascript, but if
    you want a scripting language that's popular now, Python is hard to
    beat. And that means it's more likely to still have some life in it 10
    or 20 years from now than many other things.
    
    Not all sites agree on which programming languages are actually the
    most popular and I'm not strongly against considering other
    possibilities, but Python seems to be pretty high on most lists, often
    #1, and that does matter.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

47. Re: RFC: adding pytest as a supported test framework

    Melanie Plageman <melanieplageman@gmail.com> — 2024-06-13T18:09:01Z

    On Thu, Jun 13, 2024 at 7:27 AM Daniel Gustafsson <daniel@yesql.se> wrote:
    >
    > > On 13 Jun 2024, at 00:34, Melanie Plageman <melanieplageman@gmail.com> wrote:
    >
    > > FWIW, I felt a lot of pain trying to write recovery TAP tests with
    > > IPC::Run's pumping functionality. It was especially painful (as
    > > someone who knows even less Perl than the "street fighting Perl"
    > > Thomas Munro has described having) before the additional test
    > > infrastructure was added in BackgroudPsql.pm last year.
    >
    > A key aspect of this, which isn't specific to Perl or our use of it, is that
    > this was done in backbranches which doesn't have the (recently) much improved
    > BackgroundPsql.pm.  The quality of our tools and the ease of use they provide
    > is directly related to the investment we make into continuously improving our
    > testharness.  Regardless of which toolset we adopt, if we don't make this
    > investment (taking learnings from the past years and toolsets into account)
    > we're bound to repeat this thread in a few years advocating for toolset X+1.
    
    True. And thank you for committing BackgroundPsql.pm (and Andres for
    starting that work). My specific case is likely one of a poor work
    person blaming her tools :)
    
    - Melanie
    
    
    
    

48. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T18:11:02Z

    On Thu, Jun 13, 2024 at 1:08 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > I think 1 & 3 could be addressed by more easily giving commit/merge
    > access to these tools than to the main PG repo. And I think 2 could be
    > addressed by writing on the relevant wiki page where to go, and
    > probably putting a link to the wiki page on the actual website of the
    > tool.
    
    +1.
    
    > But Perl is at the next level of unmaintained infrastructure. It is
    > actually clear how you can contribute to it, but still no new
    > community members actually want to contribute to it. Also, it's not
    > only unmaintained by us but it's also pretty much unmaintained by the
    > upstream community.
    
    I feel like I already agreed to this in a previous email and you're
    continuing to argue with me as if I were disagreeing.
    
    > As you said, no one in our community wants to maintain our testsuite
    > full time. But our test suite consists partially of upstream
    > dependencies and partially of our own code. Right now pretty much
    > no-one improves the ustream code, and pretty much no-one improves our
    > own code. Using a more modern language gives up much more frequent
    > upstream improvements for free, and it will allow new community
    > members to contribute to our own test suite.
    
    I also agree with this. I'm just not super optimistic about how much
    of that will actually happen. And I'd like to hear you acknowledge
    that concern and think about whether it can be addressed in some way,
    instead of just repeating that we should do it anyway. Because I agree
    we probably should do it anyway, but that doesn't mean I wouldn't like
    to see the downsides mitigated as much as we can. In particular, if
    the proposal is exactly "let's add the smallest possible patch that
    enables people to write tests in Python and then add a few new tests
    in Python while leaving almost everything else in Perl, with no
    migration plan and no clear vision of how the Python support ever gets
    any better than the minimum stub that is proposed for initial commit,"
    then I don't know that I can vote for that plan. Honestly, that sounds
    like very little work for the person proposing that minimal patch and
    a whole lot of work for the rest of the community later on, and the
    evidence is not in favor of volunteers showing up to take care of that
    work. The plan should be more front-loaded than that: enough initial
    development should get done by the people making the proposal that if
    the work stops after, we don't have another big mess on our hands.
    
    Or so I think, anyway.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

49. Re: RFC: adding pytest as a supported test framework

    Daniel Gustafsson <daniel@yesql.se> — 2024-06-13T18:39:16Z

    > On 13 Jun 2024, at 20:09, Melanie Plageman <melanieplageman@gmail.com> wrote:
    > 
    > On Thu, Jun 13, 2024 at 7:27 AM Daniel Gustafsson <daniel@yesql.se> wrote:
    >> 
    >>> On 13 Jun 2024, at 00:34, Melanie Plageman <melanieplageman@gmail.com> wrote:
    >> 
    >>> FWIW, I felt a lot of pain trying to write recovery TAP tests with
    >>> IPC::Run's pumping functionality. It was especially painful (as
    >>> someone who knows even less Perl than the "street fighting Perl"
    >>> Thomas Munro has described having) before the additional test
    >>> infrastructure was added in BackgroudPsql.pm last year.
    >> 
    >> A key aspect of this, which isn't specific to Perl or our use of it, is that
    >> this was done in backbranches which doesn't have the (recently) much improved
    >> BackgroundPsql.pm.  The quality of our tools and the ease of use they provide
    >> is directly related to the investment we make into continuously improving our
    >> testharness.  Regardless of which toolset we adopt, if we don't make this
    >> investment (taking learnings from the past years and toolsets into account)
    >> we're bound to repeat this thread in a few years advocating for toolset X+1.
    > 
    > True. And thank you for committing BackgroundPsql.pm (and Andres for
    > starting that work). My specific case is likely one of a poor work
    > person blaming her tools :)
    
    I don't think it is since the tools we had then were really hard to use.  I
    wrote very similar tests to yours for the online checksums patch and they were
    quite complicated to get right.  The point is that the complexity was greatly
    reduced by the community, and that kind of work will be equally important
    regardless of toolset.
    
    --
    Daniel Gustafsson
    
    
    
    
    

50. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-13T18:52:20Z

    On Thu, 13 Jun 2024 at 20:11, Robert Haas <robertmhaas@gmail.com> wrote:
    > > But Perl is at the next level of unmaintained infrastructure. It is
    > > actually clear how you can contribute to it, but still no new
    > > community members actually want to contribute to it. Also, it's not
    > > only unmaintained by us but it's also pretty much unmaintained by the
    > > upstream community.
    >
    > I feel like I already agreed to this in a previous email and you're
    > continuing to argue with me as if I were disagreeing.
    
    Sorry about that.
    
    > I also agree with this. I'm just not super optimistic about how much
    > of that will actually happen. And I'd like to hear you acknowledge
    > that concern and think about whether it can be addressed in some way,
    > instead of just repeating that we should do it anyway. Because I agree
    > we probably should do it anyway, but that doesn't mean I wouldn't like
    > to see the downsides mitigated as much as we can.
    
    I'm significantly more optimistic than you, but I also definitely
    understand and agree with the concern. I also agree that mitigating
    that concern beforehand would be a good thing.
    
    > In particular, if
    > the proposal is exactly "let's add the smallest possible patch that
    > enables people to write tests in Python and then add a few new tests
    > in Python while leaving almost everything else in Perl, with no
    > migration plan and no clear vision of how the Python support ever gets
    > any better than the minimum stub that is proposed for initial commit,"
    > then I don't know that I can vote for that plan. Honestly, that sounds
    > like very little work for the person proposing that minimal patch and
    > a whole lot of work for the rest of the community later on, and the
    > evidence is not in favor of volunteers showing up to take care of that
    > work. The plan should be more front-loaded than that: enough initial
    > development should get done by the people making the proposal that if
    > the work stops after, we don't have another big mess on our hands.
    >
    > Or so I think, anyway.
    
    I understand and agree with your final stated goal of not ending up in
    another big mess. It's also clear to me that you don't think the
    current proposal achieves that goal. So I assume you have some
    additional ideas for the proposal to help achieve that goal and/or
    some specific worries that you'd like to get addressed better in the
    proposal. But currently it's not really clear to me what either of
    those are. Could you clarify?
    
    
    
    

51. Re: RFC: adding pytest as a supported test framework

    Greg Sabino Mullane <htamfids@gmail.com> — 2024-06-13T19:16:54Z

    On Thu, Jun 13, 2024 at 9:38 AM Robert Haas <robertmhaas@gmail.com> wrote:
    
    > I agree with you, but I'm skeptical that solving it will be as easy as
    > switching to Python. For whatever reason, it seems like every piece of
    > infrastructure that the PostgreSQL community has suffers from severe
    > neglect. Literally everything I know of either has one or maybe two
    > very senior hackers maintaining it, or no maintainer at all.
    
    ...
    
    > All of this stuff is critical project infrastructure and yet it feels like
    > nobody wants to work on
    > it.
    
    
    I feel at least some of this is a visibility / marketing problem. I've not
    seen any dire requests for help come across on the lists, nor things on the
    various todos/road maps/ blog posts people make from time to time. If I
    had, I would have jumped in. And for the record, I'm very proficient with
    Perl.
    
    Cheers,
    Greg
    

52. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T19:20:29Z

    On Thu, Jun 13, 2024 at 2:52 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > I understand and agree with your final stated goal of not ending up in
    > another big mess. It's also clear to me that you don't think the
    > current proposal achieves that goal. So I assume you have some
    > additional ideas for the proposal to help achieve that goal and/or
    > some specific worries that you'd like to get addressed better in the
    > proposal. But currently it's not really clear to me what either of
    > those are. Could you clarify?
    
    Hmm, I don't know that I have what you're hoping I have, or at least
    not any more than what I've said already.
    
    I interpreted Jacob's original email as articulating a goal ("For the
    v18 cycle, I would like to try to get pytest [1] in as a supported
    test driver, in addition to the current offerings") rather than a
    plan. There's no patch set yet and, as I understand it, no detailed
    plan for a patch set: that email seemed to focus on the question of
    desirability, rather than on outlining a plan of work, which I assume
    is still to come. Some things I'd like to see when a patch set does
    show up are:
    
    - good documentation for people who have no previous experience with
    Python and/or pytest e.g. here's how to set up your environment on
    Linux, Windows, macOS, *BSD so you can run the tests, here's how to
    run the tests, here's how it's different from the Perl framework we
    have now
    
    - no external dependencies on PostgreSQL connectors. psql or libpq
    foreign function interface. the latter would be a cool increment of
    progress over the status quo.
    
    - at least as much in-tree support for writing tests as we have today
    with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention
    of the stuff we have now, and documentation of those facilities that
    is as good or, ideally, better than what we have today.
    
    - high overall code quality and level of maturity, not just something
    someone threw together for parity with the Perl system.
    
    - enough tests written for or converted to the new system to give
    reviewers confidence that it's truly usable and fit for purpose.
    
    The important thing to me here (as it so often is) is to think like a
    maintainer. Imagine that immediately after the patches for this
    feature are committed, the developers who did the work all disappear
    from the community and are never heard from again. How much pain does
    that end us causing? The answer doesn't need to be zero; that is
    unrealistic. But it also shouldn't be "well, if that happens we're
    going to have to rip the feature out" or "well, a bunch of committers
    who didn't want to write tests in Python in the first place are now
    going to have to do a lot of work in Python to stabilize the work
    already committed."
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

53. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T19:21:09Z

    On Thu, Jun 13, 2024 at 3:17 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:
    > I feel at least some of this is a visibility / marketing problem. I've not seen any dire requests for help come across on the lists, nor things on the various todos/road maps/ blog posts people make from time to time. If I had, I would have jumped in. And for the record, I'm very proficient with Perl.
    
    I agree with all of that!
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

54. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-13T19:28:42Z

    On Thu, Jun 13, 2024 at 11:11 AM Robert Haas <robertmhaas@gmail.com> wrote:
    > I feel like I already agreed to this in a previous email and you're
    > continuing to argue with me as if I were disagreeing.
    
    I also think that maybe arguments are starting to sail past each
    other, and the temperature is starting to climb. (And Jelte may be
    arguing to all readers of the thread, rather than just a single
    individual. It's hard to tell with the list format.) And now I see
    that there's another email that came in while I was writing this, but
    I think I'm going to have to send this as-is because I can't write
    emails that fast.
    
    > I also agree with this. I'm just not super optimistic about how much
    > of that will actually happen. And I'd like to hear you acknowledge
    > that concern and think about whether it can be addressed in some way,
    > instead of just repeating that we should do it anyway. Because I agree
    > we probably should do it anyway, but that doesn't mean I wouldn't like
    > to see the downsides mitigated as much as we can.
    
    Okay, +1.
    
    > In particular, if
    > the proposal is exactly "let's add the smallest possible patch that
    > enables people to write tests in Python and then add a few new tests
    > in Python while leaving almost everything else in Perl, with no
    > migration plan and no clear vision of how the Python support ever gets
    > any better than the minimum stub that is proposed for initial commit,"
    > then I don't know that I can vote for that plan.
    
    (that's not the proposal and I know/think you know that but having my
    original email twisted into that is making me feel a bit crispy)
    
    I do not want to migrate, and I have stated so multiple times, which
    is why I have not proposed a migration plan. Other committers have
    already expressed resistance to the idea that we would rewrite the
    Perl stuff. I think they're right. I think we should not. I think we
    should accept the cost of both Perl and something else for the
    near-to-medium future, as long as the "something else" gives us value
    offsetting the additional cost.
    
    > Honestly, that sounds
    > like very little work for the person proposing that minimal patch and
    > a whole lot of work for the rest of the community later on, and the
    > evidence is not in favor of volunteers showing up to take care of that
    > work.
    
    Okay, cool. Here: as the person who is 100% signing himself up to do
    that, time for me to jump in.
    
    I have an entire 6000-something-line suite of protocol tests that has
    been linked like four times above. It does something fundamentally
    different from the end-to-end Perl suite; it is not a port. It is far
    from perfect and I do not want to pressure people to adopt it as-is,
    which is why I have not. In this thread, I am offering it solely as
    evidence that I have follow-up intent.
    
    But I might get hit by a bus. Or, as far as anyone except me knows, I
    might lose interest after things get hard, which would be sad. Which
    is why my very first proposal was to add an entry point that can be
    reverted. The suite is not going to infect the codebase any more than
    the Perl does. A revert will involve pulling the Meson test entry
    code, and deleting all pytest subdirectories (of which there is only
    one, IIRC, in my OAuth suite).
    
    > The plan should be more front-loaded than that: enough initial
    > development should get done by the people making the proposal that if
    > the work stops after, we don't have another big mess on our hands.
    
    For me personally, the problem is the opposite. I have done _so much_
    initial development by myself that there's no way it could ever be
    reviewed and accepted. But I had to do that to get meaningful
    development done in my style of work, which is focused on security and
    testability and verifiable implementation.
    
    I am trying to carve off pieces of that and say "hey, does this look
    nice to anyone else?" That will take time, probably over multiple
    different threads. In the meantime, I don't want to be a serialization
    point for other people who are excited about trying new testing
    methods, because very few people are currently doing the exact kind of
    work I am doing. They may want to do other things, as evidenced by the
    thread contents. At least one committer would have to sign up to be a
    serialization point, unfortunately, but I think that's going to be
    true regardless of plan, if we want multiple non-committer members of
    the community to be involved instead of just one torch-bearer.
    
    Because of how many moving parts and competing interests and personal
    disagreements there are, I am firmly in the camp of "try something
    that many people think *might* work better, that can be undone if it
    sucks, and iterate on it." I want to build community momentum, because
    I think that's a pretty effective way to change the cultural norms
    that you're saying you're frustrated with. That doesn't mean I want to
    do this without a plan; it just means that the plan can involve saying
    "this is not working and we can undo it" which makes the uncertainty
    easier to take.
    
    --Jacob
    
    
    
    

55. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T20:04:04Z

    On Thu, Jun 13, 2024 at 3:28 PM Jacob Champion
    <jacob.champion@enterprisedb.com> wrote:
    > (that's not the proposal and I know/think you know that but having my
    > original email twisted into that is making me feel a bit crispy)
    
    I definitely did not mean to imply that. I took your original email as
    a goal, rather than a proposal or plan. My statement was strictly
    intended as a hypothetical because I didn't think any plan had been
    proposed - I only meant to say that *if* the plan were to do X, that
    would be a hard sell for me.
    
    > I do not want to migrate, and I have stated so multiple times, which
    > is why I have not proposed a migration plan. Other committers have
    > already expressed resistance to the idea that we would rewrite the
    > Perl stuff. I think they're right. I think we should not. I think we
    > should accept the cost of both Perl and something else for the
    > near-to-medium future, as long as the "something else" gives us value
    > offsetting the additional cost.
    
    I agree. It's not terribly pretty, IMHO, but it's hard to see doing
    things any other way.
    
    > For me personally, the problem is the opposite. I have done _so much_
    > initial development by myself that there's no way it could ever be
    > reviewed and accepted. But I had to do that to get meaningful
    > development done in my style of work, which is focused on security and
    > testability and verifiable implementation.
    
    I admire this attitude. I think a lot of people who go off and do a
    ton of initial work outside core show up and are like "ok, now take
    all of my code." As you say, that's not realistic. One caveat here,
    perhaps, is that the focus of the work you've done up until now and
    the things that I and other community members may want as a condition
    of merging stuff may be somewhat distinct. You will have naturally
    been focused on your goals rather than other people's goals, or so I
    assume.
    
    > I am trying to carve off pieces of that and say "hey, does this look
    > nice to anyone else?" That will take time, probably over multiple
    > different threads.
    
    This makes sense, but I would be a bit wary of splitting it up over
    too many different threads. It may well make sense to split it up, but
    it will probably be easier to review if the core work to enable this
    is one patch set on one thread where someone can read just that one
    thread and understand the situation, rather than many threads where
    you have to read them all.
    
    > Because of how many moving parts and competing interests and personal
    > disagreements there are, I am firmly in the camp of "try something
    > that many people think *might* work better, that can be undone if it
    > sucks, and iterate on it." I want to build community momentum, because
    > I think that's a pretty effective way to change the cultural norms
    > that you're saying you're frustrated with. That doesn't mean I want to
    > do this without a plan; it just means that the plan can involve saying
    > "this is not working and we can undo it" which makes the uncertainty
    > easier to take.
    
    As a community, we're really bad at this. Once something gets
    committed, getting a consensus to revert it is really hard, especially
    if a major release has happened meanwhile, but most of the time even
    if it hasn't. It might be a little easier in this case, since after
    all it's not a directly user-visible feature. But historically what
    happens if somebody says "hey, there are six unfixed problems with
    this feature!" is that everybody says "well, you're free to fix the
    problems if you want, but you're not allowed to revert the feature."
    And that is *exactly* how we end up with stuff like the current TAP
    test framework: ripping that out would mean removing all the TAP tests
    that depend on it, and that wouldn't have achieved consensus two
    months after the feature went in, let alone today.
    
    Now, it has been suggested to me by at least one other person involved
    with the project that we need to be more open to the kind of thing
    that you propose here: add experimental things and take them out if it
    doesn't work out. I can definitely understand that this might be a
    culturally better approach than what we currently do. So maybe that's
    the way forward, but it is hard (at least for me) to get past the fear
    of being the one left holding the bag, and I suspect that other
    committers have similar fears. What exactly we should do about that,
    I'm not sure.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

56. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-13T20:06:13Z

    On 2024-06-12 We 11:28, Andres Freund wrote:
    > Hi,
    >
    > On 2024-06-11 08:04:57 -0400, Andrew Dunstan wrote:
    >> Some time ago I did some work on wrapping libpq using the perl FFI module.
    >> It worked pretty well, and would mean we could probably avoid many uses of
    >> IPC::Run, and would probably be substantially more efficient (no fork
    >> required). It wouldn't avoid all uses of IPC::Run, though.
    > FWIW, I'd *love* to see work on this continue. The reduction in test runtime
    > on windows is substantial and would shorten the hack->CI->fail->hack loop a
    > good bit shorter. And save money.
    
    
    OK, I will put it high on my list. I just did some checking and it seems 
    to be feasible on Windows. StrawberryPerl at least has FFI::Platypus out 
    of the box, so we would not need to turn any great handsprings to make 
    progress on this on a fairly wide variety of platforms.
    
    What seems a good place to start would be a simple 
    PostgreSQL::Test::Session object that would allow us to get rid of a 
    whole heap of start/pump_until/kill cycles and deal with the backend in 
    a much more straightforward and comprehensible way, not to mention the 
    possibility of removing lots of $node->{safe_}psql calls.
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB: https://www.enterprisedb.com
    
    
    
    
    

57. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-13T20:06:44Z

    On Thu, Jun 13, 2024 at 12:20 PM Robert Haas <robertmhaas@gmail.com> wrote:
    > I interpreted Jacob's original email as articulating a goal ("For the
    > v18 cycle, I would like to try to get pytest [1] in as a supported
    > test driver, in addition to the current offerings") rather than a
    > plan.
    
    That's the first part of it.
    
    > There's no patch set yet and, as I understand it, no detailed
    > plan for a patch set: that email seemed to focus on the question of
    > desirability, rather than on outlining a plan of work, which I assume
    > is still to come.
    
    There was a four-step plan sketch at the end of that email, titled "A
    Plan". That was not intended to be "the final detailed plan", because
    I was soliciting feedback on the exact pieces people wanted to try to
    implement first, and I am not the God Emperor of Pytest. But it was
    definitely A Plan.
    
    > Some things I'd like to see when a patch set does
    > show up are:
    >
    > - good documentation for people who have no previous experience with
    > Python and/or pytest e.g. here's how to set up your environment on
    > Linux, Windows, macOS, *BSD so you can run the tests, here's how to
    > run the tests, here's how it's different from the Perl framework we
    > have now
    
    +1
    
    > - no external dependencies on PostgreSQL connectors. psql or libpq
    > foreign function interface. the latter would be a cool increment of
    > progress over the status quo.
    
    If this is a -1 for psycopg, then I will cast my vote for ctypes/CFFI
    and against psql.
    
    > - at least as much in-tree support for writing tests as we have today
    > with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention
    > of the stuff we have now, and documentation of those facilities that
    > is as good or, ideally, better than what we have today.
    
    I think this is way too much expectation for a v1 patch. If you were
    committing this by yourself, would you agree to develop the entirety
    of PostgreSQL::Test in a single commit, without the benefit of the
    buildfarm checking you as you went, and other people trying to write
    tests with it?
    
    > - high overall code quality and level of maturity, not just something
    > someone threw together for parity with the Perl system.
    
    +1
    
    > - enough tests written for or converted to the new system to give
    > reviewers confidence that it's truly usable and fit for purpose.
    
    This is that "know everything up front" tax that I think is not
    reasonable for a test framework. If the thing you're trying to avoid
    is the foot-in-the-door phenomenon, I would agree with you for a
    Postgres feature. But these are tests; we don't ship them, we have
    different rules for backporting them, they are developed in a very
    different way.
    
    > The important thing to me here (as it so often is) is to think like a
    > maintainer. Imagine that immediately after the patches for this
    > feature are committed, the developers who did the work all disappear
    > from the community and are never heard from again. How much pain does
    > that end us causing? The answer doesn't need to be zero; that is
    > unrealistic. But it also shouldn't be "well, if that happens we're
    > going to have to rip the feature out"
    
    Can you elaborate on why that's not an okay outcome?
    
    > or "well, a bunch of committers
    > who didn't want to write tests in Python in the first place are now
    > going to have to do a lot of work in Python to stabilize the work
    > already committed."
    
    Is it that? If the problem is that, we should address that. Because if
    that is truly the fear, I cannot assuage that fear without showing you
    something, and I cannot show you something you do not want to see, if
    you don't want to write tests in Python in the first place.
    
    --Jacob
    
    
    
    

58. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-13T20:19:21Z

    On 2024-06-12 We 18:34, Melanie Plageman wrote:
    > On Tue, Jun 11, 2024 at 8:05 AM Andrew Dunstan <andrew@dunslane.net> wrote:
    >>
    >> On 2024-06-10 Mo 21:49, Andres Freund wrote:
    >>
    >> Hi,
    >>
    >> On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
    >>
    >> I'm not sure what part of the testing infrastructure you think is
    >> unmaintained. For example, the last release of Test::Simple was all the way
    >> back on April 25.
    >>
    >> IPC::Run is quite buggy and basically just maintained by Noah these days.
    >>
    >>
    >> Yes, that's true. I think the biggest pain point is possibly the recovery tests.
    >>
    >> Some time ago I did some work on wrapping libpq using the perl FFI module. It worked pretty well, and would mean we could probably avoid many uses of IPC::Run, and would probably be substantially more efficient (no fork required). It wouldn't avoid all uses of IPC::Run, though.
    >>
    >> But my point was mainly that while a new framework might have value, I don't think we need to run out and immediately rewrite several hundred TAP tests. Let's pick the major pain points and address those.
    > FWIW, I felt a lot of pain trying to write recovery TAP tests with
    > IPC::Run's pumping functionality. It was especially painful (as
    > someone who knows even less Perl than the "street fighting Perl"
    > Thomas Munro has described having) before the additional test
    > infrastructure was added in BackgroudPsql.pm last year. As an example
    > of the "worst case", it took me two full work days to go from a repro
    > (with psql sessions on a primary and replica node) of the vacuum hang
    > issue being explored in [1] to a sort-of working TAP test which
    > demonstrated it - and that was with help from several other
    > committers. Granted, this is a complex case.
    
    
    The pump stuff is probably the least comprehensible and most fragile 
    part of the whole infrastructure. As I just mentioned to Andres, I'm 
    hoping to make a substantial improvement in that area.
    
    
    >
    > A small part of the issue is that, as Tristan has said elsewhere,
    > there aren't good developer tool integrations that I know about for
    > Perl. I use neovim's LSP support for C and Python (in other projects),
    > and there is a whole ecosystem of tools I can use for both C and
    > Python. I know not everyone likes or needs these, but I find that they
    > help me write and debug code faster.
    
    
    You might find this useful: 
    <https://climatechangechat.com/setting_up_lsp_nvim-lspconfig_and_perl_in_neovim.html>
    
    (I don't use neovim - I'm an old emacs dinosaur.)
    
    
    >
    > I had offered to take a stab at writing some of the BackgroundPsql
    > test infrastructure in Python. I haven't started exploring that yet or
    > looking at what Jacob has done so far, but I am optimistic that this
    > is an area where it is worth seeing what is available to us outside of
    > IPC::Run.
    
    
    Yeah, like I said, I'm working on reducing our reliance on especially 
    the more fragile parts of IPC::Run.
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB: https://www.enterprisedb.com
    
    
    
    
    

59. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T20:27:35Z

    On Thu, Jun 13, 2024 at 4:06 PM Jacob Champion
    <jacob.champion@enterprisedb.com> wrote:
    > There was a four-step plan sketch at the end of that email, titled "A
    > Plan". That was not intended to be "the final detailed plan", because
    > I was soliciting feedback on the exact pieces people wanted to try to
    > implement first, and I am not the God Emperor of Pytest. But it was
    > definitely A Plan.
    
    Well, OK, now I feel a bit dumb. I guess I missed that or forgot about it.
    
    > > - at least as much in-tree support for writing tests as we have today
    > > with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention
    > > of the stuff we have now, and documentation of those facilities that
    > > is as good or, ideally, better than what we have today.
    >
    > I think this is way too much expectation for a v1 patch. If you were
    > committing this by yourself, would you agree to develop the entirety
    > of PostgreSQL::Test in a single commit, without the benefit of the
    > buildfarm checking you as you went, and other people trying to write
    > tests with it?
    
    Eh... I'm confused. PostgreSQL::Test::Cluster is more than half of the
    code in that directory, and without it you wouldn't be able to write
    most of the TAP tests that we have today. You would really want to
    call this project done without having an equivalent?
    
    > > The important thing to me here (as it so often is) is to think like a
    > > maintainer. Imagine that immediately after the patches for this
    > > feature are committed, the developers who did the work all disappear
    > > from the community and are never heard from again. How much pain does
    > > that end us causing? The answer doesn't need to be zero; that is
    > > unrealistic. But it also shouldn't be "well, if that happens we're
    > > going to have to rip the feature out"
    >
    > Can you elaborate on why that's not an okay outcome?
    
    Well, you just argued that it should be an okay outcome, and I do sort
    of see your point, but I refer you to my earlier reply about the
    difficulty of getting anything reverted in the culture as it stands.
    
    > > or "well, a bunch of committers
    > > who didn't want to write tests in Python in the first place are now
    > > going to have to do a lot of work in Python to stabilize the work
    > > already committed."
    >
    > Is it that? If the problem is that, we should address that. Because if
    > that is truly the fear, I cannot assuage that fear without showing you
    > something, and I cannot show you something you do not want to see, if
    > you don't want to write tests in Python in the first place.
    
    I have zero desire to write tests in Python. If I could convince
    everyone here to spend their time and energy improving the stuff we
    have in Perl instead of introducing a whole new test framework, I
    would 100% do that. But I'm pretty sure that I can't, and I think the
    project needs to pick from among realistic options rather than
    theoretical ones. Said differently, it's not all about me.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

60. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-13T20:32:53Z

    On 2024-06-12 We 11:50, Andres Freund wrote:
    > Hi,
    >
    > On 2024-06-11 07:28:23 -0700, Jacob Champion wrote:
    >> On Mon, Jun 10, 2024 at 1:04 PM Andres Freund<andres@anarazel.de>  wrote:
    >>> Just for context for the rest the email: I think we desperately need to move
    >>> off perl for tests. The infrastructure around our testing is basically
    >>> unmaintained and just about nobody that started doing dev stuff in the last 10
    >>> years learned perl.
    >> Okay. Personally, I'm going to try to stay out of discussions around
    >> subtracting Perl and focus on adding Python, for a bunch of different
    >> reasons:
    > I think I might have formulated my paragraph above badly - I didn't mean that
    > we should move away from perl tests tomorrow,
    
    
    OK, glad we're on the same page there. Let's move on.
    
    
    > but that we need a path forward
    > that allows folks to write tests without perl.
    
    
    OK, although to be honest I'm more interested in fixing some of the 
    things that have made testing with perl a pain, especially the IPC::Run 
    pump stuff.
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB:https://www.enterprisedb.com
    

61. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-13T20:41:54Z

    On 2024-06-13 Th 15:16, Greg Sabino Mullane wrote:
    > I'm very proficient with Perl.
    >
    >
    
    Yes you are, and just as well!
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB:https://www.enterprisedb.com
    

62. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-13T20:54:56Z

    On 2024-06-12 We 18:43, Jelte Fennema-Nio wrote:
    >
    > I agree it's not a technical issue. It is a people issue. There are
    > very few people skilled in Perl active in the community. And most of
    > those are very senior hackers that have much more important things to
    > do that make our Perl testing framework significantly better. And the
    > less senior people that might see improving tooling as a way to get
    > help out in the community, are try to stay away from Perl with a 10
    > foot pole. So the result is, nothing gets improved. Especially since
    > very few people outside our community improve this tooling either.
    >
    >   
    
    
    FTR, I have put a lot of effort into maintaining and improving the 
    infrastructure over the years. And I don't think there is anything much 
    more important. So I'm going to put more effort in. And I'm not alone. 
    Andres, Alvaro, Noah and Thomas are some of those who have spent a lot 
    of effort on extending and improving our testing.
    
    People tend to get a bit hung up about languages. I lost count of the 
    various languages I had learned when it got somewhere north of 30.
    
    Still, I understand that perl has a few oddities that make people 
    scratch their heads (as do most languages). It's probably losing market 
    share, along with some of the other things we rely on. Not sure that 
    alone is a reason to move away from it.
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB: https://www.enterprisedb.com
    
    
    
    
    

63. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-13T21:23:48Z

    On Thu, Jun 13, 2024 at 4:54 PM Andrew Dunstan <andrew@dunslane.net> wrote:
    > FTR, I have put a lot of effort into maintaining and improving the
    > infrastructure over the years. And I don't think there is anything much
    > more important. So I'm going to put more effort in. And I'm not alone.
    > Andres, Alvaro, Noah and Thomas are some of those who have spent a lot
    > of effort on extending and improving our testing.
    
    I appreciate the work you've done, and the work others have done, and
    I'm sorry if my comments about the state of the project's
    infrastructure came across as a personal attack. Some of what is wrong
    here is completely outside of our control e.g. Perl is less popular.
    And even there, some people have done heroic work, like Noah stepping
    up to help maintain IPC::Run. And even with the stuff that is under
    our control, it's not that I don't like what you're doing. It's rather
    that I think we need more people doing it. For example, the fact that
    nobody's helping Thomas maintain this cfbot that we all have come to
    rely on, or helping him get that integrated into
    commitfest.postgresql.org, is a problem. You're not on the hook to do
    that, nor is anyone else. Likewise, the PostgreSQL::Test::whatever
    modules are mostly evolving when it's absolutely necessary to get some
    other patch committed, rather than anyone looking to improve them very
    much for their own sake. Maybe part of the problem, as Greg said, is
    that we don't do a good enough job advertising what the problems are
    or how people can help, but whatever the cause, it's not a very
    enjoyable experience, at least for me.
    
    But again, I don't blame you for any of that. You're clearly a big
    part of why it's going as well as it is!
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

64. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-13T21:35:52Z

    On 2024-06-13 Th 17:23, Robert Haas wrote:
    > On Thu, Jun 13, 2024 at 4:54 PM Andrew Dunstan <andrew@dunslane.net> wrote:
    >> FTR, I have put a lot of effort into maintaining and improving the
    >> infrastructure over the years. And I don't think there is anything much
    >> more important. So I'm going to put more effort in. And I'm not alone.
    >> Andres, Alvaro, Noah and Thomas are some of those who have spent a lot
    >> of effort on extending and improving our testing.
    > I appreciate the work you've done, and the work others have done, and
    > I'm sorry if my comments about the state of the project's
    > infrastructure came across as a personal attack. Some of what is wrong
    > here is completely outside of our control e.g. Perl is less popular.
    > And even there, some people have done heroic work, like Noah stepping
    > up to help maintain IPC::Run. And even with the stuff that is under
    > our control, it's not that I don't like what you're doing. It's rather
    > that I think we need more people doing it. For example, the fact that
    > nobody's helping Thomas maintain this cfbot that we all have come to
    > rely on, or helping him get that integrated into
    > commitfest.postgresql.org, is a problem. You're not on the hook to do
    > that, nor is anyone else. Likewise, the PostgreSQL::Test::whatever
    > modules are mostly evolving when it's absolutely necessary to get some
    > other patch committed, rather than anyone looking to improve them very
    > much for their own sake. Maybe part of the problem, as Greg said, is
    > that we don't do a good enough job advertising what the problems are
    > or how people can help, but whatever the cause, it's not a very
    > enjoyable experience, at least for me.
    >
    > But again, I don't blame you for any of that. You're clearly a big
    > part of why it's going as well as it is!
    >
    
    Thank you, I'm not offended by anything you or anyone else has said. 
    Clearly there are areas we can improve, and we need to be somewhat more 
    proactive about it.
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB: https://www.enterprisedb.com
    
    
    
    
    

65. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-13T22:03:12Z

    On Thu, 13 Jun 2024 at 23:35, Andrew Dunstan <andrew@dunslane.net> wrote:
    > Clearly there are areas we can improve, and we need to be somewhat more
    > proactive about it.
    
    To follow that great suggestion, I updated meson wiki[1] after I
    realized some of the major gripes I had with the Perl tap test output
    were not actually caused by Perl but by meson:
    
    The main changes I made was using "-q --print-errorlogs" instead "-v",
    to reduce the enormous clutter in the output of the commands in the
    wiki to something much more reasonable.
    
    As well as adding examples on how to run specific tests
    
    [1]: https://wiki.postgresql.org/wiki/Meson#Test_related_commands
    
    
    
    

66. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-14T00:02:11Z

    On Thu, Jun 13, 2024 at 1:04 PM Robert Haas <robertmhaas@gmail.com> wrote:
    > One caveat here,
    > perhaps, is that the focus of the work you've done up until now and
    > the things that I and other community members may want as a condition
    > of merging stuff may be somewhat distinct. You will have naturally
    > been focused on your goals rather than other people's goals, or so I
    > assume.
    
    Right. That's a risk I knew I was taking when I wrote it, so it's not
    going to offend me when I need to rewrite things.
    
    > I would be a bit wary of splitting it up over
    > too many different threads. It may well make sense to split it up, but
    > it will probably be easier to review if the core work to enable this
    > is one patch set on one thread where someone can read just that one
    > thread and understand the situation, rather than many threads where
    > you have to read them all.
    
    I'll try to avoid too many threads. But right now there is indeed just
    one thread (OAUTHBEARER) and it's way too much:
    
    - the introduction of pytest
    - a Construct-based manipulation of the wire protocol, including
    Wireshark-style network traces on failure
    - pytest fixtures which spin up libpq and the server in isolation from
    each other, relying on the Construct implementation to complete the
    seam
    - OAuth, which was one of the motivating use cases (but not the only
    one) for all of the previous items
    
    I really don't want to derail this thread with those. There are other
    people here with their own hopes and dreams (see: unconference notes),
    and I want to give them a platform too.
    
    > > That doesn't mean I want to
    > > do this without a plan; it just means that the plan can involve saying
    > > "this is not working and we can undo it" which makes the uncertainty
    > > easier to take.
    >
    > As a community, we're really bad at this. [...]
    
    I will carry the response to this to the next email.
    
    Thanks,
    --Jacob
    
    
    
    

67. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-14T00:12:15Z

    On Thu, Jun 13, 2024 at 1:27 PM Robert Haas <robertmhaas@gmail.com> wrote:
    >
    > On Thu, Jun 13, 2024 at 4:06 PM Jacob Champion
    > <jacob.champion@enterprisedb.com> wrote:
    > > There was a four-step plan sketch at the end of that email, titled "A
    > > Plan". That was not intended to be "the final detailed plan", because
    > > I was soliciting feedback on the exact pieces people wanted to try to
    > > implement first, and I am not the God Emperor of Pytest. But it was
    > > definitely A Plan.
    >
    > Well, OK, now I feel a bit dumb. I guess I missed that or forgot about it.
    
    No worries. It's a really long thread. :D
    
    But also: do you have opinions on what to fill in as steps 2
    (something we have no ability to test today) and 3 (something we do
    test today, but hate)?
    
    My vote for step 2 is "client and server separation", perhaps by
    testing libpq fallback against a server that claims support for
    different build-time options. I don't want to have a vote in step 3,
    because part of that step is proving that this framework can provide
    value for a part of the project I don't really know much about.
    
    > > I think this is way too much expectation for a v1 patch. If you were
    > > committing this by yourself, would you agree to develop the entirety
    > > of PostgreSQL::Test in a single commit, without the benefit of the
    > > buildfarm checking you as you went, and other people trying to write
    > > tests with it?
    >
    > Eh... I'm confused. PostgreSQL::Test::Cluster is more than half of the
    > code in that directory, and without it you wouldn't be able to write
    > most of the TAP tests that we have today.
    
    Well, in my defense, you said "PostgreSQL::Test::whatever", which I
    assumed meant all of it, including Kerberos.pm and SSL::Server and
    AdjustUpgrade and... That seemed like way too much to me (and still
    does!), but if that's not what you were arguing then never mind.
    
    Yes, Cluster.pm seems like a pretty natural thing to ask for. I
    imagine it's one of the first things we're going to need. And yet...
    
    > You would really want to
    > call this project done without having an equivalent?
    
    ...I have this really weird sneaking suspicion that, if a replacement
    of the end-to-end Perl acceptance tests can be made an explicit
    anti-goal in the short term, we might not necessarily need an
    "equivalent" for v1. I realize that seems bizarre, because of course
    we need a way to start the server if we want to test the server. But
    frankly, starting a server is Pretty Easy (tm), and Cluster.pm has to
    do a lot more than that because IMO it's designed for a variety of
    acceptance-oriented tasks. 3000+ lines!
    
    If there's widespread interest (as opposed to being just my own
    personal fever dream) in testing Postgres components as individual
    pieces rather than setting up the world, then I wonder if the
    functionality from Cluster.pm couldn't be pared down a lot. Maybe you
    don't need a centralized ->psql() or a ->command_ok() helper, because
    you're usually not trying to test psql and other utilities during your
    server-only tests.
    
    Maybe you can just stand up a standby without a primary and drive it
    via mock replication. Do you need quite as many "poll and wait for
    some asynchronous result" type things when you're not waiting for a
    result to cascade through a multinode system? Does something like (for
    example) ->pg_recvlogical_upto() really have to be implemented in our
    "core" fixtures or can it be done more easily by whoever needs that in
    the future? Maybe You Ain't Gonna Need It.
    
    If (he said, atop his naive idealistic soapbox) we can find a way to
    put off writing utilities until we write the tests that need them,
    without procrastinating, and without putting all of the negative
    externalities of that approach on the committers with low-quality
    copy-paste proliferation, and I'd like a pony while I'm at it, then I
    think the result might end up being pretty coherent and maintainable.
    Then not having "at least as much in-tree support for writing tests as
    we have today" for the very first commit would be a feature and not a
    bug.
    
    Now, maybe if the collective ability to do that existed, we would have
    done it already with Perl, but I do actually wonder whether that's
    true or not.
    
    Or, maybe, the very first suggestion for Step 3 will be something that
    needs absolutely everything in Cluster.pm. So be it; I can live
    without a pony.
    
    > You would really want to
    > call this project done without having an equivalent?
    
    (A cop-out but not-really-cop-out alternative answer to this question
    is that this project is not going to be "done" any more than Postgres
    will ever be "done", and that's part of what I'm arguing should be
    considered natural and okay. I understand that it is easier for me to
    take that stance when I am not on the hook for maintaining it, so I
    don't expect us to necessarily see eye-to-eye on it.)
    
    > > Can you elaborate on why that's not an okay outcome?
    >
    > Well, you just argued that it should be an okay outcome, and I do sort
    > of see your point, but I refer you to my earlier reply about the
    > difficulty of getting anything reverted in the culture as it stands.
    
    Earlier reply was:
    
    > As a community, we're really bad at this. Once something gets
    > committed, getting a consensus to revert it is really hard, especially
    > if a major release has happened meanwhile, but most of the time even
    > if it hasn't. It might be a little easier in this case, since after
    > all it's not a directly user-visible feature. But historically what
    > happens if somebody says "hey, there are six unfixed problems with
    > this feature!" is that everybody says "well, you're free to fix the
    > problems if you want, but you're not allowed to revert the feature."
    > And that is *exactly* how we end up with stuff like the current TAP
    > test framework: ripping that out would mean removing all the TAP tests
    > that depend on it, and that wouldn't have achieved consensus two
    > months after the feature went in, let alone today.
    
    Well... I don't know how to fix that. Here's a draft proposal after a
    few minutes of thought, which may need to be discarded after a few
    more minutes of thought.
    
    If there's agreement that New Tests -- not necessarily written in
    Python, but I selfishly hope they are -- exist on a probationary
    status, then maybe part of that is going to have to be an agreement:
    New features have to be able to have some minimum maintainability
    level *on the basis of the Perl tests only*, while the probationary
    period is in effect. It can't be the equivalent maintainability level,
    because that's either proof that the New Tests are giving us nothing,
    or proof that everyone is being forced to implement the exact same
    tests in both Perl and New Test. Neither is good.
    
    Since we're currently focused on end-to-end acceptance with Perl, that
    is probably a lower bar than what we'd maybe prefer, but I think that
    is the bar we have right now. It also exists as a forcing function to
    make sure that the additional tests are adding value over what we get
    with the Perl, which may paradoxically increase the chances of New
    Test success. (I can't tell if this is magical thinking or not.)
    
    So if a committer doesn't want responsibility for the feature if the
    new tests were deleted, they don't commit. Maybe that's unrealistic
    and too painful. It does increase the review requirements of
    committers quite a bit. It might disqualify my OAuth work (which is
    maybe evidence in its favor?). Maybe it increases the foot-in-the-door
    effect too much. Maybe there would have to be some trust-building
    where right now there is not? Not sure.
    
    > Now, it has been suggested to me by at least one other person involved
    > with the project that we need to be more open to the kind of thing
    > that you propose here: add experimental things and take them out if it
    > doesn't work out. I can definitely understand that this might be a
    > culturally better approach than what we currently do. So maybe that's
    > the way forward, but it is hard (at least for me) to get past the fear
    > of being the one left holding the bag, and I suspect that other
    > committers have similar fears. What exactly we should do about that,
    > I'm not sure.
    
    Yeah.
    
    > I have zero desire to write tests in Python. If I could convince
    > everyone here to spend their time and energy improving the stuff we
    > have in Perl instead of introducing a whole new test framework, I
    > would 100% do that. But I'm pretty sure that I can't, and I think the
    > project needs to pick from among realistic options rather than
    > theoretical ones. Said differently, it's not all about me.
    
    Then, for what it's worth: I really do want to make sure that your
    life, and the life of all the other committers, does not get
    significantly harder if this goes in. I don't think it will, but if
    I'm wrong, I want it to come back out, and then we can regroup or
    pivot entirely and move forward together.
    
    --Jacob
    
    
    
    

68. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-14T12:10:01Z

    On Thu, Jun 13, 2024 at 8:12 PM Jacob Champion
    <jacob.champion@enterprisedb.com> wrote:
    > But also: do you have opinions on what to fill in as steps 2
    > (something we have no ability to test today) and 3 (something we do
    > test today, but hate)?
    >
    > My vote for step 2 is "client and server separation", perhaps by
    > testing libpq fallback against a server that claims support for
    > different build-time options. I don't want to have a vote in step 3,
    > because part of that step is proving that this framework can provide
    > value for a part of the project I don't really know much about.
    
    I mean, both Perl and Python are Turing-complete. Anything you can do
    in one, you can do in the other, especially when you consider that
    we're not likely to accept too many dependencies on external Perl or
    Python modules. That's why I see this as nothing more or less than
    exercise in letting people use the programming language they prefer.
    We've talked about a libpq FFI interface, but it hasn't been done; now
    we're talking about maybe having a Python one. Perhaps we'll end up
    with both. Then you can imagine porting tests from one language to the
    other and the only difference is whether you'd rather have line noise
    before all of your variable names or semantically significant
    whitespace.
    
    I just don't believe in the idea that we're going to write one
    category of tests in one language and another category in another
    language. As soon as we open the door to Python tests, people are
    going to start writing the TAP tests that they would have written in
    Perl in Python instead. And if the test utilities that we have for
    Perl are not there for Python, then they'll either open code things
    for which they would have used a module, or they'll write a
    stripped-down version of the module that will then get built-up patch
    by patch until, 50 or 100 or 200 hours of committer-review later, it
    resembles the existing Perl module. And, if the committer pushes back
    and says, hey, why not write this test in Perl which already has all
    of this test infrastructure in place already, then the submitter will
    wander off muttering about how PostgreSQL committers are hidebound
    backward individuals who just try to ruin everybody's day. So as I see
    it, the only reasonable plan here if we want to introduce testing in
    Python (or C#, or Ruby, or Go, or JavaScript, or Lua, or LOLCODE) is
    to try to achieve a reasonable degree of parity between that language
    and Perl. Because then we can at least review the new infrastructure
    all at once, instead of incrementally spread across many patches
    written, reviewed, and committed by many different people.
    
    Now, I completely understand if you're not excited about getting
    sucked down that rabbit-hole, and maybe some other committer is going
    to see this differently than I do, and that's fine. But my view is
    that if you're not interested in doing all the work to let people do
    more or less the kind of stuff that they currently do in Perl in
    Python instead, then your alternative is to take the tests that you
    want to add and rewrite them in Perl. And I am fairly cetain that if
    you choose that option, it will save me, and a bunch of other
    committers, a whole lot of work, at least in the short term. If we add
    support for Python, we are going to end up having to do a lot of
    things twice for the next let's say ten to twenty years until somebody
    rewrites the remaining Perl tests in Python or whatever language is
    hot and cool by then. My opinion is that we need to be open to
    enduring that pain because we can't indefinitely hold our breath and
    insist on using obsolete tools for everything, but that doesn't mean
    that I don't think it will be painful.
    
    Consider the meson build system project. To get that committed, Andres
    had to make it do pretty much everything MSVC could do and mostly
    everything that configure could do, and the places where he didn't
    make it do everything configure could do remain sore spots that I, at
    least, am not happy about. And in that case, he also managed to get
    MSVC removed entirely, so that we do not have a larger number of build
    systems in total than we had before. Furthermore, the amount of code
    in buildsystem files (makefiles, meson.build) in a typical patch
    needing review is usually none or very little, whereas the amount of
    test code in a patch is sometimes quite large. I've come around to the
    belief that the meson work was worthwhile -- running tests is so much
    faster and nicer now -- but it was a ton of work to get done and
    impacted everyone in the development community, and I think the blast
    radius for this change is likely to be larger for the reasons
    suggested earlier in this paragraph.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

69. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-14T15:15:04Z

    On 2024-06-14 Fr 08:10, Robert Haas wrote:
    > We've talked about a libpq FFI interface, but it hasn't been done;
    
    
    Hold my beer :-)
    
    
    I just posted a POC for that.
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB:https://www.enterprisedb.com
    

70. Re: RFC: adding pytest as a supported test framework

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-06-14T15:49:29Z

    Robert Haas <robertmhaas@gmail.com> writes:
    > I mean, both Perl and Python are Turing-complete. Anything you can do
    > in one, you can do in the other, especially when you consider that
    > we're not likely to accept too many dependencies on external Perl or
    > Python modules. That's why I see this as nothing more or less than
    > exercise in letting people use the programming language they prefer.
    
    I think that's an oversimplified analysis.  Sure, the languages are
    both Turing-complete, but for our purposes here they are both simply
    glue languages around some set of testing facilities.  Some of those
    facilities will be provided by the base languages (plus whatever
    extension modules we choose to require) and some by code we write.
    The overall experience of writing tests will be determined by the
    testing facilities far more than by the language used for glue.
    
    That being the case, I do agree with your point that Python
    equivalents to most of PostgreSQL::Test will need to be built up PDQ.
    Maybe they can be better than the originals, in features or ease of
    use, but "not there at all" is not better.
    
    But what I'd really like to see is some comparison of the
    language-provided testing facilities that we're proposing
    to depend on.  Why is pytest (or whatever) better than Test::More?
    
    I also wonder about integration of python-based testing with what
    we already have.  A significant part of what you called the meson
    work had to do with persuading pg_regress, isolationtester, etc
    to output test results in the common format established by TAP.
    Am I right in guessing that pytest will have nothing to do with that?
    Can we even manage to dump perl and python test scripts into the same
    subdirectory and sort things out automatically?  I'm definitely going
    to be -1 for a python testing feature that cannot integrate with what
    we have because it demands its own control and result-collection
    infrastructure.
    
    			regards, tom lane
    
    
    
    

71. Re: RFC: adding pytest as a supported test framework

    Andres Freund <andres@anarazel.de> — 2024-06-14T16:11:11Z

    Hi,
    
    On 2024-06-14 11:49:29 -0400, Tom Lane wrote:
    > I also wonder about integration of python-based testing with what
    > we already have.  A significant part of what you called the meson
    > work had to do with persuading pg_regress, isolationtester, etc
    > to output test results in the common format established by TAP.
    
    FWIW, meson's testrunner doesn't require TAP, the lowest common denominator is
    just an exit code. However, for things that run many "sub" tests, it's a lot
    nicer if the failures can be reported more granularly than just "the entire
    testsuite failed".
    
    Meson currently supports:
    
        exitcode: the executable's exit code is used by the test harness to record the outcome of the test).
    
        tap: Test Anything Protocol.
    
        gtest (since 0.55.0): for Google Tests.
    
        rust (since 0.56.0): for native rust tests
    
    
    > Am I right in guessing that pytest will have nothing to do with that?
    
    Looks like there's a plugin for pytest to support tap as output:
    https://pypi.org/project/pytest-tap/
    
    However, it's not available as a debian package. I know that some folks just
    advocate installing dependencies via venv, but I personally don't think
    that'll fly. For one, it'll basically prevent tests being run by packagers.
    
    
    > Can we even manage to dump perl and python test scripts into the same
    > subdirectory and sort things out automatically?
    
    That shouldn't be much of a problem.
    
    
    > I'm definitely going to be -1 for a python testing feature that cannot
    > integrate with what we have because it demands its own control and
    > result-collection infrastructure.
    
    Dito.
    
    Greetings,
    
    Andres Freund
    
    
    
    

72. Re: RFC: adding pytest as a supported test framework

    Andres Freund <andres@anarazel.de> — 2024-06-14T16:17:25Z

    On 2024-06-14 09:11:11 -0700, Andres Freund wrote:
    > On 2024-06-14 11:49:29 -0400, Tom Lane wrote:
    > > Am I right in guessing that pytest will have nothing to do with that?
    > 
    > Looks like there's a plugin for pytest to support tap as output:
    > https://pypi.org/project/pytest-tap/
    > 
    > However, it's not available as a debian package. I know that some folks just
    > advocate installing dependencies via venv, but I personally don't think
    > that'll fly. For one, it'll basically prevent tests being run by packagers.
    
    If this were the blocker, I think we could just ship an output adapter
    ourselves. pytest-tap is not a lot of code:
    https://github.com/python-tap/pytest-tap/blob/main/src/pytest_tap/plugin.py
    
    So either vendoring it or just writing an even simpler version ourselves seems
    entirely feasible.
    
    
    
    

73. Re: RFC: adding pytest as a supported test framework

    Greg Sabino Mullane <htamfids@gmail.com> — 2024-06-14T20:33:00Z

    On Thu, Jun 13, 2024 at 1:08 PM Jelte Fennema-Nio <postgres@jeltef.nl>
    wrote:
    
    > But Perl is at the next level of unmaintained infrastructure. It is
    > actually clear how you can contribute to it, but still no new
    > community members actually want to contribute to it. Also, it's not
    > only unmaintained by us but it's also pretty much unmaintained by the
    > upstream community.
    
    
    I am not happy with the state of Perl, as it has made some MAJOR missteps
    along the way, particularly in the last 5 years. But can we dispel this
    strawman? There is a difference between "unpopular" and "unmaintained". The
    latest version of Perl was released May 20, 2024. The latest release of
    Test::More was April 25, 2024. Both are heavily used. Just not as heavily
    as they used to be. :)
    
    Cheers,
    Greg
    

74. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-14T21:08:57Z

    On Fri, 14 Jun 2024 at 22:33, Greg Sabino Mullane <htamfids@gmail.com> wrote:
    > I am not happy with the state of Perl, as it has made some MAJOR missteps along the way, particularly in the last 5 years. But can we dispel this strawman? There is a difference between "unpopular" and "unmaintained". The latest version of Perl was released May 20, 2024. The latest release of Test::More was April 25, 2024. Both are heavily used. Just not as heavily as they used to be. :)
    
    Sorry, yes I exaggerated here. Looking at the last Perl changelog[1]
    it's definitely getting more new features and improvements than I had
    thought.
    
    Test::More on the other hand, while indeed still maintained, it's
    definitely not getting significant new feature development or
    improvements[2]. Especially when comparing it to pytest[3].
    
    [1]: https://perldoc.perl.org/perldelta
    [2]: https://github.com/Test-More/test-more/blob/master/Changes
    [3]: https://docs.pytest.org/en/stable/changelog.html
    
    
    
    

75. Re: RFC: adding pytest as a supported test framework

    Thomas Munro <thomas.munro@gmail.com> — 2024-06-14T21:24:00Z

    On Fri, Jun 14, 2024 at 9:24 AM Robert Haas <robertmhaas@gmail.com> wrote:
    > For example, the fact that
    > nobody's helping Thomas maintain this cfbot that we all have come to
    > rely on, or helping him get that integrated into
    > commitfest.postgresql.org, is a problem.
    
    I've been talking to Magnus and Jelte about cfbot and we're hoping to
    have some good news soon...
    
    
    
    

76. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-14T22:11:04Z

    On Fri, 14 Jun 2024 at 17:49, Tom Lane <tgl@sss.pgh.pa.us> wrote:
    > But what I'd really like to see is some comparison of the
    > language-provided testing facilities that we're proposing
    > to depend on.  Why is pytest (or whatever) better than Test::More?
    
    Some advantages of pytest over Test::More:
    
    1. It's much easier to debug failing tests using the output that
    pytest gives. A good example of this is on pytest its homepage[1]
    (i.e. it shows the value given to the call to inc in the error)
    2. No need to remember a specific comparison DSL
    (is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of
    a boolean expression and your output is great (if you want to add a
    message too for clarity you can use: assert a == b, "the world is
    ending")
    3. Very easy to postgres log files on stderr/stdout when a test fails.
    This might be possible/easy with Perl too, but we currently don't do
    that. So right now for many failures you're forced to traverse the
    build/testrun/... directory tree to find the logs.
    4. Pytest has autodiscovery of test files and functions, so we
    probably wouldn't have to specify all of the exact test files anymore
    in the meson.build files.
    
    Regarding 2, there are ~150 checks that are using a suboptimal way of
    testing for a comparison. Mostly a lot that could use "like(..., ...)"
    instead of "ok(... ~= ...)"
    ❯ grep '\bok(.*=' **.pl | wc -l
    151
    
    [1]: https://docs.pytest.org/en/8.2.x/#a-quick-example
    
    
    
    

77. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-15T14:45:16Z

    On 2024-06-14 Fr 18:11, Jelte Fennema-Nio wrote:
    > On Fri, 14 Jun 2024 at 17:49, Tom Lane<tgl@sss.pgh.pa.us>  wrote:
    >> But what I'd really like to see is some comparison of the
    >> language-provided testing facilities that we're proposing
    >> to depend on.  Why is pytest (or whatever) better than Test::More?
    > Some advantages of pytest over Test::More:
    >
    > 1. It's much easier to debug failing tests using the output that
    > pytest gives. A good example of this is on pytest its homepage[1]
    > (i.e. it shows the value given to the call to inc in the error)
    > 2. No need to remember a specific comparison DSL
    > (is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of
    > a boolean expression and your output is great (if you want to add a
    > message too for clarity you can use: assert a == b, "the world is
    > ending")
    > 3. Very easy to postgres log files on stderr/stdout when a test fails.
    > This might be possible/easy with Perl too, but we currently don't do
    > that. So right now for many failures you're forced to traverse the
    > build/testrun/... directory tree to find the logs.
    
    
    I see the fact that we stash the output in a file as a feature. Without 
    it, capturing failure information in the buildfarm client would be more 
    difficult, especially if there are multiple failures. So this is 
    actually something I think we would need for any alternative framework.
    
    Maybe we need an environment setting that would output the 
    regress_log_00whatever file to stderr on failure.  That should be pretty 
    easy to arrange in the END handler for PostgreSQL::Test::Utils.
    
    
    > 4. Pytest has autodiscovery of test files and functions, so we
    > probably wouldn't have to specify all of the exact test files anymore
    > in the meson.build files.
    
    
    I find this comment a bit ironic. We don't need to do that with the 
    Makefiles, and the requirement to do so was promoted as a meson feature 
    rather than a limitation, ISTR.
    
    
    > Regarding 2, there are ~150 checks that are using a suboptimal way of
    > testing for a comparison. Mostly a lot that could use "like(..., ...)"
    > instead of "ok(... ~= ...)"
    > ❯ grep '\bok(.*=' **.pl | wc -l
    > 151
    
    
    Well, let's fix those. I would be tempted to use cmp_ok() for just about 
    all of them.
    
    But the fact that Test::More has a handful of test primitives rather 
    than just one strikes me as a relatively minor complaint.
    
    
    
    cheers
    
    
    andrew
    
    
    -- 
    
    Andrew Dunstan
    EDB: https://www.enterprisedb.com
    
    
    
    
    

78. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-15T16:48:33Z

    On Sat, 15 Jun 2024 at 16:45, Andrew Dunstan <andrew@dunslane.net> wrote:
    > I see the fact that we stash the output in a file as a feature. Without
    > it, capturing failure information in the buildfarm client would be more
    > difficult, especially if there are multiple failures. So this is
    > actually something I think we would need for any alternative framework.
    
    I indeed heard that the current behaviour was somehow useful to the
    buildfarm client.
    
    > Maybe we need an environment setting that would output the
    > regress_log_00whatever file to stderr on failure.  That should be pretty
    > easy to arrange in the END handler for PostgreSQL::Test::Utils.
    
    That sounds awesome! But I'm wondering: do we really need a setting to
    enable/disable that? Can't we always output it to stderr on failure?
    If we output the log both to stderr and as a file, would that be fine
    for the build farm? If not, a setting should work. (but I'd prefer the
    default for that setting to be on in that case, it seems much easier
    to turn it off in the buildfarm client, instead of asking every
    developer to turn the feature on)
    
    > > 4. Pytest has autodiscovery of test files and functions, so we
    > > probably wouldn't have to specify all of the exact test files anymore
    > > in the meson.build files.
    >
    >
    > I find this comment a bit ironic. We don't need to do that with the
    > Makefiles, and the requirement to do so was promoted as a meson feature
    > rather than a limitation, ISTR.
    
    Now, I'm very curious why that would be considered a feature. I
    certainly have had many cases where I forgot to add the test file to
    the meson.build file.
    
    > > Regarding 2, there are ~150 checks that are using a suboptimal way of
    > > testing for a comparison. Mostly a lot that could use "like(..., ...)"
    > > instead of "ok(... ~= ...)"
    > > ❯ grep '\bok(.*=' **.pl | wc -l
    > > 151
    >
    >
    > Well, let's fix those. I would be tempted to use cmp_ok() for just about
    > all of them.
    
    Sounds great to me.
    
    > But the fact that Test::More has a handful of test primitives rather
    > than just one strikes me as a relatively minor complaint.
    
    It is indeed a minor paper cut, but paper-cuts add up.
    
    Honestly, my primary *objective* complaint about our current test
    suite, is that when a test fails, it's very often impossible for me to
    understand why the test failed, by only looking at the output of
    "meson test". I think logging the postgres log to stderr for Perl, as
    you proposed, would significantly improve that situation. I think the
    only thing that we cannot get from Perl Test::More that we can from
    pytest, is the fancy recursive introspection of the expression that
    pytest shows on error.
    
    
    Apart from that my major *subjective* complaint is that I very much
    dislike writing Perl code. I'm slow at writing it and I don't (want
    to) improve at it because I don't have reasons to use it except for
    Postgres tests. So currently I'm not really incentivised to write more
    tests than the bare minimum, help improve the current test tooling, or
    add new testing frameworks for things we currently cannot test.
    Afaict, there's a significant part of our current community who feel
    the same way (and I'm pretty sure every sub-30 year old person who
    newly joins the community would feel the exact same way too).
    
    As a project I think we would like to have more tests, and to have
    more custom tooling to test things that we currently cannot (e.g.
    oauth or manually messing with the wire-protocol). I think the only
    way to achieve that is by encouraging more people to work on these
    things. I very much appreciate that you and others are improving our
    Perl tooling, because that makes our current tests easier to work
    with. But I don't think it significantly increases the willingness to
    write tests or test-tooling for people that don't want to write Perl
    in the first place.
    
    So I think the only way to get more people involved in contributing
    tests and test-tooling is by allowing testing in another language than
    Perl (but also still allow writing tests in Perl). Even if that means
    that we have two partially-overlapping test frameworks, that are both
    easy to use for different things. In my view that's even a positive
    thing, because that means we are able to test more with two languages
    than we would be able to with either one (and it's thus useful to have
    both).
    
    And I agree with Robbert that Python seems like the best choice for
    this other language, given its current popularity level. But as I said
    before, I'm open to other languages as well.
    
    
    
    

79. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-15T17:26:57Z

    On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    > Honestly, my primary *objective* complaint about our current test
    > suite, is that when a test fails, it's very often impossible for me to
    > understand why the test failed, by only looking at the output of
    > "meson test". I think logging the postgres log to stderr for Perl, as
    > you proposed, would significantly improve that situation. I think the
    > only thing that we cannot get from Perl Test::More that we can from
    > pytest, is the fancy recursive introspection of the expression that
    > pytest shows on error.
    
    This surprises me. I agree that the current state of affairs is kind
    of annoying, but the contents of regress_log_whatever are usually
    quite long. Printing all of that out to standard output seems like
    it's just going to flood the terminal with output. I don't think I'd
    be a fan of that change.
    
    I think I basically agree with all the nearby comments about how the
    advantages you cite for Python aren't, I don't know, entirely
    compelling. Switching from ok() to is() or cmp_ok() or like() is minor
    stuff. Where the output goes is minor stuff. The former can be fixed,
    and the latter can be worked around with scripts and aliases. The one
    thing I know about that *I* think is a pretty big problem about Perl
    is that IPC::Run is not really maintained. But I wonder if the
    solution to that is to do something ourselves instead of depending on
    IPC::Run. Beyond that, I think this is just a language popularity
    contest.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

80. Re: RFC: adding pytest as a supported test framework

    Andres Freund <andres@anarazel.de> — 2024-06-15T17:33:20Z

    Hi,
    
    On 2024-06-15 10:45:16 -0400, Andrew Dunstan wrote:
    > > 4. Pytest has autodiscovery of test files and functions, so we
    > > probably wouldn't have to specify all of the exact test files anymore
    > > in the meson.build files.
    > 
    > 
    > I find this comment a bit ironic. We don't need to do that with the
    > Makefiles, and the requirement to do so was promoted as a meson feature
    > rather than a limitation, ISTR.
    
    The reason its good to have the list of tests somewhere explicit is that we
    have different types of test. With make, there is a single target for all tap
    tests. If you want to run tests concurrently, make can only schedule the tap
    tests at the granularity of a directory. If you want concurrency below that,
    you need to use concurrency on the prove level. But that means that you have
    extremely varying concurrency, depending on whether make runs targets that
    have no internal concurrency or make runs e.g. the recovery tap tests.
    
    I don't think we should rely on global test discovery via pytest. That'll lead
    to uncontrollable concurrency again, which means much longer test times. We'll
    always have different types of tests, just scheduling running them via
    "top-level" tools for different test types just won't work well. That's not
    true for many projects where tests have vastly lower resource usage.
    
    Greetings,
    
    Andres Freund
    
    
    
    

81. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-15T17:39:57Z

    On Sat, 15 Jun 2024 at 19:27, Robert Haas <robertmhaas@gmail.com> wrote:
    > This surprises me. I agree that the current state of affairs is kind
    > of annoying, but the contents of regress_log_whatever are usually
    > quite long. Printing all of that out to standard output seems like
    > it's just going to flood the terminal with output. I don't think I'd
    > be a fan of that change.
    
    I think at the very least the locations of the different logs should
    be listed in the output.
    
    
    
    

82. Re: RFC: adding pytest as a supported test framework

    Greg Sabino Mullane <htamfids@gmail.com> — 2024-06-15T21:52:32Z

    On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl>
    wrote:
    
    > Afaict, there's a significant part of our current community who feel the
    > same way (and I'm pretty sure every sub-30 year old person who
    > newly joins the community would feel the exact same way too).
    >
    
    Those young-uns are also the same group who hold their nose when coding in
    C, and are always clamoring for rewriting Postgres in Rust. And before
    that, C++. And next year, some other popular language that is clearly
    better and more popular than C.
    
    And I agree with Robbert that Python seems like the best choice for this
    > other language, given its current popularity level. But as I said
    > before, I'm open to other languages as well.
    >
    
    Despite my previous posts, I am open to other languages too, including
    Python, but the onus is really on the new language promoters to prove that
    the very large amount of time and trouble is worth it, and worth it for
    language X.
    
    Cheers,
    Greg
    

83. Re: RFC: adding pytest as a supported test framework

    Greg Sabino Mullane <htamfids@gmail.com> — 2024-06-15T21:57:29Z

    On Fri, Jun 14, 2024 at 5:09 PM Jelte Fennema-Nio <postgres@jeltef.nl>
    wrote:
    
    > Test::More on the other hand, while indeed still maintained, it's
    > definitely not getting significant new feature development or
    > improvements[2]. Especially when comparing it to pytest[3].
    >
    
    That's fair, although it's a little hard to tell if the lack of new
    features is because they are not needed for a stable, mature project, or
    because few people are asking for and developing new features. Probably a
    bit of both. But I'll be the first to admit Perl is dying; I just don't
    know what should replace it (or how - or when). Python has its quirks, but
    all languages do, and your claim that it will encourage more and easier
    test writing by developers is a good one.
    
    Cheers,
    Greg
    

84. Re: RFC: adding pytest as a supported test framework

    Melanie Plageman <melanieplageman@gmail.com> — 2024-06-15T22:00:43Z

    On Sat, Jun 15, 2024 at 5:53 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:
    >
    > On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
    >>
    >> Afaict, there's a significant part of our current community who feel the same way (and I'm pretty sure every sub-30 year old person who
    >> newly joins the community would feel the exact same way too).
    >
    >
    > Those young-uns are also the same group who hold their nose when coding in C, and are always clamoring for rewriting Postgres in Rust. And before that, C++. And next year, some other popular language that is clearly better and more popular than C.
    
    Writing a new test framework in a popular language that makes it more
    likely that more people will write more tests and test infrastructure
    is such a completely different thing than suggesting we rewrite
    Postgres in Rust that I feel that this comparison is unfair and,
    frankly, a distraction from the discussion at hand.
    
    - Melanie
    
    
    
    

85. Re: RFC: adding pytest as a supported test framework

    Robert Haas <robertmhaas@gmail.com> — 2024-06-16T21:04:41Z

    On Sat, Jun 15, 2024 at 6:00 PM Melanie Plageman
    <melanieplageman@gmail.com> wrote:
    > > Those young-uns are also the same group who hold their nose when coding in C, and are always clamoring for rewriting Postgres in Rust. And before that, C++. And next year, some other popular language that is clearly better and more popular than C.
    >
    > Writing a new test framework in a popular language that makes it more
    > likely that more people will write more tests and test infrastructure
    > is such a completely different thing than suggesting we rewrite
    > Postgres in Rust that I feel that this comparison is unfair and,
    > frankly, a distraction from the discussion at hand.
    
    I don't really agree with this. We have been told before that we would
    attract more developers to our community if only we allowed backend
    code to be written in C++ or Rust, and that is not altogether a
    different thing than saying that we would attract more test developers
    if only we allowed test code to be written in Python or whatever. The
    difference is one of degree rather than of kind. We have a lot more
    backend code than we do test code, I'm fairly sure, and our tests are
    more self-contained: it's not *as* problematic if some tests are
    written in one language and others in another as it would be if
    different parts of the backend used different languages, and it
    wouldn't be *as* hard if at some point we decided we wanted to convert
    all remaining code to the new language. So, I have a much harder time
    imagining that we would start allowing a new language for backend code
    than that we would start allowing a new language for tests, but I
    don't think the issues are fundamentally different.
    
    But that said, I'm not sure the programming language is the real
    issue. If I really wanted to participate in an open source project,
    I'd probably be willing to learn a new programming language to do
    that. Maybe some people wouldn't, but I had to learn a whole bunch of
    them in college, and learning one more doesn't sound like the biggest
    of deals. But, would I feel respected and valued as a participant in
    that project? Would I have to use weird tools and follow arcane and
    frustrating processes? If I did, *that* would make me give up. I don't
    want to say that the choice of programming language doesn't matter at
    all, but it seems to me that it might matter more because it's a
    symptom of being unwilling to modernize things rather than for its own
    sake.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    

86. Re: RFC: adding pytest as a supported test framework

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2024-06-17T08:27:09Z

    Hi Greg, Jelte,
    
    On Sat, 15 Jun 2024 at 23:53, Greg Sabino Mullane <htamfids@gmail.com>
    wrote:
    >
    > On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl>
    wrote:
    >>
    >> Afaict, there's a significant part of our current community who feel the
    same way (and I'm pretty sure every sub-30 year old person who
    >> newly joins the community would feel the exact same way too).
    
    I feel I'm still relatively new (started in the past 4 years) and I have
    quite some time left before I hit 30 years of age.
    
    I don't specifically feel the way you describe, nor do I think I've ever
    really felt like that in the previous nearly 4 years of hacking. Then
    again, I'm not interested in testing frameworks, so I don't feel much at
    all about which test frameworks we use.
    
    > Those young-uns are also the same group who hold their nose when coding
    in C, and are always clamoring for rewriting Postgres in Rust.
    
    Could you point me to one occasion I have 'always' clamored for this, or
    any of "those young-uns" in the community? I may not be a huge fan of C,
    but rewriting PostgreSQL in [other language] is not on the list of things
    I'm clamoring for. I may have given off-hand mentions that [other language]
    would've helped in certain cases, sure, but I'd hardly call that clamoring.
    
    Kind regards,
    
    Matthias van de Meent
    

87. Re: RFC: adding pytest as a supported test framework

    Aleksander Alekseev <aleksander@timescale.com> — 2024-06-17T12:19:05Z

    Hi Jacob,
    
    > For the v18 cycle, I would like to try to get pytest [1] in as a
    > supported test driver, in addition to the current offerings.
    >
    > (I'm tempted to end the email there.)
    
    Huge +1 from me and many thanks for working on this.
    
    Two cents from me.
    
    I spent a fair part of my career writing in Perl. Personally I like
    the language and often find it more convenient for the tasks I'm
    working on than Python.
    
    This being said, there were several projects I was involved in where
    we had to choose a scripting language. In all the cases there was a
    strong push-back against Perl and Python always seemed to be a common
    denominator for everyone. So I ended up with the rule of thumb to use
    Perl for projects I'm working on alone and Python otherwise. Although
    the objective reality in the entire industry is unknown to me I spoke
    to many people whose observations were similar.
    
    We could speculate about the reasons why people seem to prefer Python
    (good IDE support*, unique** libraries like Matplotlib / NumPy /
    SciPy, ...) but honestly I don't think they are extremely relevant in
    this discussion.
    
    I believe supporting Python in our test infrastructure will attract
    more contributors and thus would be a good step for the project in the
    long run.
    
    * including PyTest integration
    ** citation needed
    
    -- 
    Best regards,
    Aleksander Alekseev
    
    
    
    

88. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-06-17T13:39:47Z

    On Sun, 16 Jun 2024 at 23:04, Robert Haas <robertmhaas@gmail.com> wrote:
    >
    > On Sat, Jun 15, 2024 at 6:00 PM Melanie Plageman
    > <melanieplageman@gmail.com> wrote:
    > > Writing a new test framework in a popular language that makes it more
    > > likely that more people will write more tests and test infrastructure
    > > is such a completely different thing than suggesting we rewrite
    > > Postgres in Rust that I feel that this comparison is unfair and,
    > > frankly, a distraction from the discussion at hand.
    >
    > I don't really agree with this.
    > <snip>
    > it's not *as* problematic if some tests are
    > written in one language and others in another as it would be if
    > different parts of the backend used different languages, and it
    > wouldn't be *as* hard if at some point we decided we wanted to convert
    > all remaining code to the new language.
    
    Honestly, it sounds like you actually do agree with each other. It
    seems you interpreted Melanie her use of "thing" as "people wanting to
    use Rust/Python in the Postgres codebase" while I believe she probably
    meant "all the problems and effort involved in the task making that
    possible''. And afaict from your response, you definitely agree that
    making it possible to use Rust in our main codebase is a lot more
    difficult than for Python for our testing code.
    
    > But, would I feel respected and valued as a participant in
    > that project? Would I have to use weird tools and follow arcane and
    > frustrating processes? If I did, *that* would make me give up. I don't
    > want to say that the choice of programming language doesn't matter at
    > all, but it seems to me that it might matter more because it's a
    > symptom of being unwilling to modernize things rather than for its own
    > sake.
    
    I can personally definitely relate to this (although I wouldn't frame
    it as strongly as you did). Postgres development definitely requires
    weird tools and arcane processes (imho) when compared to most other
    open source projects. The elephant in the room is of course the
    mailing list development flow. But we have some good reasons for using
    that[^1]. But most people have some limit on the amount of weirdness
    they are willing to accept when wanting to contribute, and the mailing
    list pushes us quite close to that limit for a bunch of people
    already. Any additional weird tools/arcane processes might push some
    people over that limit.
    
    We've definitely made big improvements in modernizing our development
    workflow over the last few years though: We now have CI (cfbot), a
    modern build system (meson), and working autoformatting (requiring
    pgindent on commit). These improvements have been very noticeable to
    me, and I think we should continue such efforts. I think allowing
    people to write tests in Python is one of the easier improvements that
    we can make.
    
    [^1]: Although I think those reasons apply much less to the
    documentation, maybe we could allow github contributions for just
    those.
    
    
    
    

89. Re: RFC: adding pytest as a supported test framework

    Andrew Dunstan <andrew@dunslane.net> — 2024-06-17T16:21:10Z

    On 2024-06-17 Mo 4:27 AM, Matthias van de Meent wrote:
    > Hi Greg, Jelte,
    >
    > On Sat, 15 Jun 2024 at 23:53, Greg Sabino Mullane <htamfids@gmail.com> 
    > wrote:
    >
    > > Those young-uns are also the same group who hold their nose when 
    > coding in C, and are always clamoring for rewriting Postgres in Rust.
    >
    > Could you point me to one occasion I have 'always' clamored for this, 
    > or any of "those young-uns" in the community? I may not be a huge fan 
    > of C, but rewriting PostgreSQL in [other language] is not on the list 
    > of things I'm clamoring for. I may have given off-hand mentions that 
    > [other language] would've helped in certain cases, sure, but I'd 
    > hardly call that clamoring.
    >
    >
    
    Greg was being a but jocular here. I didn't take him seriously. But 
    there's maybe a better case to make the point he was making. Back in the 
    dark ages we used a source code control system called CVS. It's quite 
    unlike git and has a great many limitations and uglinesses, and there 
    was some pressure for us to move off it. If we had done so when it was 
    first suggested, we would probably have moved to using Subversion, which 
    is rather like CVS with many of the warts knocked off. Before long, some 
    distributed systems like Mercurial and git came along, and we, like most 
    of the world, chose git. Thus by waiting and not immediately doing what 
    was suggested we got a better solution. Moving twice would have been ... 
    painful.
    
    I have written Python in the past. Not a huge amount, but it doesn't 
    feel like a foreign country to me, just the next town over instead of my 
    immediate neighbourhood. We even have a python script in the buildfarm 
    server code (not written by me). I'm sure if we started writing tests in 
    Python I would adjust. But I think we need to know what the advantages 
    are, beyond simple language preference. And to get to an equivalent 
    place for Python that we are at with perl will involve some work.
    
    
    cheers
    
    
    andrew
    
    --
    Andrew Dunstan
    EDB: https://www.enterprisedb.com
    
    
    
    
    

90. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-18T14:34:17Z

    (slowly catching up from the weekend email backlog)
    
    On Fri, Jun 14, 2024 at 5:10 AM Robert Haas <robertmhaas@gmail.com> wrote:
    > I mean, both Perl and Python are Turing-complete.
    
    Tom responded to this better than I could have, but I don't think this
    is a helpful statement. In fact I opened the unconference session with
    it and immediately waved it away as not-the-point.
    
    > I just don't believe in the idea that we're going to write one
    > category of tests in one language and another category in another
    > language.
    
    You and I will probably not agree, then, because IMO we already do
    that. SQL behavior is tested in SQL via pg_regress characterization
    tests. End-to-end tests are written in Perl. Lower-level tests are
    often written in C (and, unfortunately, then driven in Perl instead of
    C; see above mail to Noah).
    
    I'm fundamentally a polyglot tester by philosophy, so I don't see
    careful use of multiple languages as an inherent problem to be solved.
    They increase complexity (considerably so!) but generally provide
    sufficient value to offset the cost.
    
    > As soon as we open the door to Python tests, people are
    > going to start writing the TAP tests that they would have written in
    > Perl in Python instead.
    
    There's a wide spectrum of opinions between yours (which I will
    cheekily paraphrase as "people will love testing in Python so much
    they'll be willing to reinvent all of the wheels" -- which in the
    short term would increase maintenance cost but in the long term sounds
    like a very good problem to have), and people who seem to think that
    new test suite infrastructure would sit unused because no one wants to
    write tests anyway (to pull a strawman out of some hallway
    conversations at PGConf.dev). I think the truth is probably somewhere
    in the middle?
    
    My prior mail was an attempt to bridge the gap between today and the
    medium term, by introducing a series of compromises and incremental
    steps in response to specific fears. We can jump forward to the end
    state and try to talk about it, but I don't control the end state and
    I don't have a crystal ball.
    
    > So as I see
    > it, the only reasonable plan here if we want to introduce testing in
    > Python (or C#, or Ruby, or Go, or JavaScript, or Lua, or LOLCODE) is
    > to try to achieve a reasonable degree of parity between that language
    > and Perl. Because then we can at least review the new infrastructure
    > all at once, instead of incrementally spread across many patches
    > written, reviewed, and committed by many different people.
    
    I don't at all believe that a test API which is ported en masse as a
    horizontal layer, without motivating vertical slices of test
    functionality, is going to be fit for purpose.
    
    And "written, reviewed, and committed by many different people" is a
    feature for me, not a bug. One of the goals of the thread is to
    encourage more community test writing than we currently have.
    Otherwise, I could have kept silent (I am very happy with my personal
    suite and have been comfortably maintaining it for a while). I am
    trying to build community momentum around a pain point that is
    currently rusted in place.
    
    > Consider the meson build system project. To get that committed, Andres
    > had to make it do pretty much everything MSVC could do and mostly
    > everything that configure could do
    
    I think some lessons can be pulled from that, but fundamentally that's
    a port of the build infrastructure done by a person with a commit bit.
    There are some pretty considerable differences. (And even then, it
    wasn't "done" with the first volley of patches, right?)
    
    Thanks,
    --Jacob
    
    
    
    

91. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-06-18T14:35:35Z

    On Fri, Jun 14, 2024 at 8:49 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
    > I think that's an oversimplified analysis.  Sure, the languages are
    > both Turing-complete, but for our purposes here they are both simply
    > glue languages around some set of testing facilities.  Some of those
    > facilities will be provided by the base languages (plus whatever
    > extension modules we choose to require) and some by code we write.
    > The overall experience of writing tests will be determined by the
    > testing facilities far more than by the language used for glue.
    
    +1. As an example, the more extensive (and high-quality) a language's
    standard library, the more tests you'll be able to write. Convincing
    committers to adopt a new third-party dependency is hard (for good
    reason); the strength of the standard library should be considered as
    a point of technical comparison.
    
    > That being the case, I do agree with your point that Python
    > equivalents to most of PostgreSQL::Test will need to be built up PDQ.
    > Maybe they can be better than the originals, in features or ease of
    > use, but "not there at all" is not better.
    
    There's a wide gulf between "not there at all" and "reimplement it all
    as a prerequisite for v1" as Robert proposed. I'm arguing for a middle
    ground.
    
    > But what I'd really like to see is some comparison of the
    > language-provided testing facilities that we're proposing
    > to depend on.  Why is pytest (or whatever) better than Test::More?
    
    People are focusing a lot on failure reporting, and I agree with them,
    but I did try to include more than just that in my OP.
    
    I'll requote what I personally think is the #1 killer feature of
    pytest, which prove and Test::More appear to be unable to accomplish
    on their own: configurable isolation of tests from each other via
    fixtures [1].
    
        Problem 1 (rerun failing tests): One architectural roadblock to this
        in our Test::More suite is that tests depend on setup that's done by
        previous tests. pytest allows you to declare each test's setup
        requirements via pytest fixtures, letting the test runner build up the
        world exactly as it needs to be for a single isolated test. These
        fixtures may be given a "scope" so that multiple tests may share the
        same setup for performance or other reasons.
    
    When I'm doing red-to-green feature development (e.g. OAuth) or
    green-to-green refactoring (e.g. replacement of libiddawc with libcurl
    in OAuth), quick cycle time and reduction of noise is extremely
    important. I want to be able to rerun just the single red test I care
    about before moving on.
    
    (Tests may additionally be organized with custom attributes. My OAuth
    suite contains tests that must run slowly due to mandatory timeouts;
    I've marked them as slow, and they can be easily skipped from the test
    runner.)
    
    2. The ability to break into a test case with the built-in debugger
    [2] is also fantastic for quick red-green work. Much better than
    print() statements.
    
    (Along similar lines, even the ability to use Python's built-in REPL
    increases velocity. Python users understand that they can execute
    `python3` and be dropped into a sandbox to try out syntax or some
    unfamiliar library. Searching for how to do this in Perl results in a
    handful of custom-built scripts; people here may know which to use as
    a Perl monk, sure, but the point is to make it easy for newcomers to
    write tests.)
    
    > I also wonder about integration of python-based testing with what
    > we already have.  A significant part of what you called the meson
    > work had to do with persuading pg_regress, isolationtester, etc
    > to output test results in the common format established by TAP.
    > Am I right in guessing that pytest will have nothing to do with that?
    
    Andres covered this pretty well. I will note that I had problems with
    pytest-tap itself [3], and I'm unclear whether that represents a bug
    in pytest-tap or a bug in pytest.
    
    Thanks,
    --Jacob
    
    [1] https://docs.pytest.org/en/stable/how-to/fixtures.html
    [2] https://docs.pytest.org/en/stable/how-to/failures.html
    [3] https://github.com/python-tap/pytest-tap/issues/30
    
    
    
    

92. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-09-09T21:50:31Z

    Hi everyone, it's been a while for this thread.
    
    = Status =
    
    Since the last email, quite a bit of offline chatter has happened,
    particularly at FOSDEM [1] and PGConf.dev [2]. The dev-meeting
    audiences, at least, appear to have reached a critical mass of nodding
    heads and cautious optimism around the use of a pytest suite. Some are
    more optimistic than others, some are quite skeptical, some are
    completely silent, but no one has -1'd it yet.
    
    I think I've been solidly outvoted in my attempt to support Python and
    Perl suites side-by-side in perpetuity. Many, many people have
    expressed that they'd feel disappointed if that were a permanent
    situation, so I've been trying to adapt my approach to the idea that
    we would need to port existing Perl tests at _some_ point in the
    timeline.
    
    Simultaneously, I've been approached by some people offline who are
    pretty clearly waiting for me to make the first move here. I'm a bit
    uncomfortable with that -- no one needs my permission to post patches
    in this vein! -- but I get that this entire topic is a balancing act,
    so it makes some sense. Thank you all for your patience. Here is a
    patchset.
    
    = Patches =
    
    This is dev-quality, not targeted for commit yet, but hopefully enough
    to spark some opinions and conversation.
    
    0001: Prepares for pytest support, by adding Meson summary lines
    indicating when the Perl TAP tests are enabled.
    
    Peter E mentioned to me that it was hard to figure out which suites
    had been enabled, so hopefully this helps a bit. I'm not sure what to
    do on the Autoconf side.
    
    0002: Adds Meson and Autoconf support for running pytest, via
    --enable-pytest or -Dpytest=enabled.
    
    This is a skeleton. There is no Postgres-specific test logic in this
    patch. So if you like pytest, but hate how I write tests in it, you
    have the option of basing your alternative on top of this patch to
    show us. I also added a failing test so that you can all see what that
    looks like in the CI. (I'll put this in the Draft CF [3].)
    
    When enabling the feature, the check_pytest.py script checks that the
    configured `PYTHON` executable has all of pytest-requirements.txt
    installed. Peter pointed out that this is incorrect: what we actually
    want to check is that the interpreter used by pytest has all of the
    required packages, and the two could be different.
    
    One way to solve that problem is to just get rid of the check script,
    and assume that if pytest is installed, we're good to go. But I ran
    into enough installation weirdness in the CI (especially for MinGW,
    ugh) that it was really helpful to have a script say "no, you're still
    missing this." I'd like to help buildfarm operators in a similar way.
    Unfortunately, check_pytest.py is pretty complex, and it doesn't make
    as much sense to pay that cost unless and until we add more package
    dependencies, as 0003 does later in the set. Thoughts?
    
    For Autoconf, I'm just running pytest directly rather than translating
    its output through pgtap + prove, mostly because I think that's a
    better dev experience. It has color and everything. Is that okay?
    
    0003: Ports a tiny subset of the SSL client tests.
    
    This is enough to show off
    - pytest fixtures and caching via scopes
    - FFI binding to libpq via ctypes
    - generation of TLS certs via the py-cryptography package
    - low-level protocol testing without a "real" server
    
    Andres requested a while back that we control concurrency via Meson
    rather than pytest's global discovery. So there is no use of
    pytest-xdist here. (I don't think we're in much danger of having CPUs
    go idle during a `meson test` run, anyway.)
    
    0004: Ports a tiny subset of the SSL server tests.
    
    This additionally shows off
    - more complicated test parameterization
    - my vision of how we can cache and configure a running server
    instance via pytest fixtures
    
    Basically, server configuration changes are pushed and popped off of a
    stack as fixtures are entered and exited. (C++ folks will be
    completely unimpressed, but I'm still proud of it.) We'd need to
    discuss the proper use of scopes and establish project-wide fixture
    conventions if we go in this direction, but I hope we do something
    like it; I really want to be able to run each and every test in
    isolation.
    
    There are some Win32 syscall examples as well, for those who are
    interested, but there's nothing that uses that code at the moment.
    
    0005: Adds a new variable to Cirrus to control the default Meson suites.
    
    0006: Sets that new variable to just the pytest suites, so that I
    don't burn too much Cirrus time for this proof of concept, and turns
    on all OSes.
    
    = Other General Thoughts =
    
    You might immediately complain that we don't want to open-code every
    single byte in the protocol for a test, as I have done here. I don't
    want to, either; I want to use Construct [4] for that. I also want us
    to have end-to-end tests that combine the fixtures in 0003 and 0004 --
    but I want everyone to be able to choose between end-to-end tests and
    mocked client/server tests, and I'm most excited about the latter, so
    that's what I focused on here.
    
    The distinction between the fixture scopes "session" and "module" is
    useful in the Autoconf case (which runs all tests in a pyt/ folder at
    once) and useless in the Meson case (which runs each pyt/*.py test in
    an individual pytest instance). I suspect this will cause some
    friction, but I don't know how much. I haven't been very careful in my
    use of "module" scope yet.
    
    I think devs should be able to run pytest directly from the source
    root. So I've put settings specific to Autoconf/Meson into
    Makefile.global/meson.build instead of pytest.ini. (For instance, I
    don't imagine anyone wants to read the pgtap output directly.)
    
    Formatting of the patchset has been provided by isort + black. I hear
    ruff is the new kid on the block, but I don't see raw speed being
    incredibly important for us. And for this particular use case, I
    really like the idea of an "unconfigurable" PEP8 formatter so we can
    do other things with our time.
    
    Huge thanks to Andres and Bilal for helping me debug Windows problems in the CI.
    
    Wow, thanks for reading this far. WDYT?
    
    --Jacob
    
    [1] https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2025_Developer_Meeting#Additional_toolchains_in_the_test_suite
    [2] https://wiki.postgresql.org/wiki/PGConf.dev_2025_Developer_Meeting
    [3] https://commitfest.postgresql.org/54/
    [4] https://construct.readthedocs.io/en/latest/
    

93. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-09-18T18:22:48Z

    On Tue, Sep 9, 2025 at 2:50 PM Jacob Champion
    <jacob.champion@enterprisedb.com> wrote:
    > When enabling the feature, the check_pytest.py script checks that the
    > configured `PYTHON` executable has all of pytest-requirements.txt
    > installed. Peter pointed out that this is incorrect: what we actually
    > want to check is that the interpreter used by pytest has all of the
    > required packages, and the two could be different.
    
    Turns out we've already solved this exact problem for Perl and Prove
    [1], and I should probably choose a similar solution for Python and
    pytest. In other words: make the requirements check into a test.
    
    --Jacob
    
    [1] http://postgr.es/c/c4fe3199a
    
    
    
    

94. Re: RFC: adding pytest as a supported test framework

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-09-22T20:30:04Z

    On Thu, Sep 18, 2025 at 11:22 AM Jacob Champion
    <jacob.champion@enterprisedb.com> wrote:
    > Turns out we've already solved this exact problem for Perl and Prove
    > [1], and I should probably choose a similar solution for Python and
    > pytest. In other words: make the requirements check into a test.
    
    Done this way in v2-0002. pytest and the linked `PYTHON` can now be
    independent of each other. This adds some scaffolding complexity, to
    get the configure script and pytest to talk to each other nicely, but
    I've gotten rid of some architectural complexity in check_pytest.py to
    make up for it.
    
    Thanks,
    --Jacob
    

95. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-10-22T12:44:15Z

    On Mon, 22 Sept 2025 at 22:30, Jacob Champion <jacob.champion@enterprisedb.com> wrote:
    > Done this way in v2-0002
    
    Okay I finally managed to do some testing of this patchset while working
    on a patchset of mine where I'm adding a GoAway message to the protocol
    (should be ready to be published soon)
    
    First of all: THANK YOU! It's a great base to start from and I hope we
    can relatively soon have something merged, that we can gradually
    improve.
    
    I had some problems using it for my own tests though. The primary
    reasons for that were:
    1. It was missing functionality to send queries and get results.
    2. A lot of the fixtures I wanted to use were located in the ssl tests
       directory instead of the shared fixtures module.
    3. When running pytest manually I had to configure LD_LIBRARY_PATH
    
    So here's your patchset with an additional commit on top that does a
    bunch of refactoring/renaming and adding features. I hope you like it. I
    tried to make the most common actions easy to do.
    
    The primary features it adds are:
    - A `sql` method on `PGconn`: It takes a query and returns the results
        as native python types.
    - A `conn` fixture: Which is a libpq based connection to the default
        Postgres server.
    - Use the `pg_config` binary to find the libdir and bindir (can be
        overridden by setting PG_CONFIG). Otherwise I had to use
        LD_LIBRARY_PATH when running pytest manually.
    
    The refactoring it does:
    - Rename `pg_server` fixture to `pg` since it'll likely be one of the
        most commonly used ones.
    - Rename `pg` module to `pypg` to avoid naming conflict/shadowing
        problems with the newly renamed `pg` fixture
    - Move class definitions outside of fixtures to separate modules (either
        in the `pypg` module or the new `libpq` module)
    - Move all "general" fixtures to the `pypg.fixtures` module, instead of
        having them be defined in the ssl module.
    

96. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-11-10T21:11:50Z

    On Wed Oct 22, 2025 at 2:44 PM CEST, Jelte Fennema-Nio wrote:
    > So here's your patchset with an additional commit on top that does a
    > bunch of refactoring/renaming and adding features. 
    
    Rebased to fix conflicts.
    

97. Re: RFC: adding pytest as a supported test framework

    Andres Freund <andres@anarazel.de> — 2025-12-17T16:10:01Z

    On 2025-11-10 22:11:50 +0100, Jelte Fennema-Nio wrote:
    > On Wed Oct 22, 2025 at 2:44 PM CEST, Jelte Fennema-Nio wrote:
    > > So here's your patchset with an additional commit on top that does a
    > > bunch of refactoring/renaming and adding features.
    > 
    > Rebased to fix conflicts.
    
    I assume this intentionally doesn't pass CI:
    https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F6045
    
    
    > From f6823405eb994d457f8123df0d417ca2340e4c71 Mon Sep 17 00:00:00 2001
    > From: Jacob Champion <jacob.champion@enterprisedb.com>
    > Date: Fri, 5 Sep 2025 16:39:08 -0700
    > Subject: [PATCH v3 01/10] meson: Include TAP tests in the configuration
    >  summary
    > 
    > ...to make it obvious when they've been enabled. prove is added to the
    > executables list for good measure.
    > 
    > TODO: does Autoconf need something similar?
    
    I agree with adding tap to the configuration summary, but I don't understand
    the prove part, that just seems like a waste of vertical space.
    
    
    > From 5a27976496db53d8e9b88ab59e6c71f0f42dedcd Mon Sep 17 00:00:00 2001
    > From: Jacob Champion <jacob.champion@enterprisedb.com>
    > Date: Wed, 13 Aug 2025 10:58:56 -0700
    > Subject: [PATCH v3 02/10] Add support for pytest test suites
    > 
    > Specify --enable-pytest/-Dpytest=enabled at configure time. This
    > contains no Postgres test logic -- it is just a "vanilla" pytest
    > skeleton.
    > 
    > I've written a custom pgtap output plugin, used by the Meson mtest
    > runner, to fully control what we see during CI test failures. The
    > pytest-tap plugin would have been preferable, but it's now in
    > maintenance mode, and it has problems with accidentally suppressing
    > important collection failures.
    > 
    > test_something.py is intended to show a sample failure in the CI.
    > 
    > TODOs:
    > - OpenBSD has an ANSI-related terminal bug, but I'm not sure if the bug
    >   is in Cirrus, the image, pytest, Python, or readline. The TERM envvar
    >   is unset to work around it. If this workaround is removed, a bad ANSI
    >   escape is inserted into the pgtap output and mtest is unable to parse
    >   it.
    > - The Chocolatey CI setup is subpar. Need to find a way to bless the
    >   dependencies in use rather than pulling from pip... or maybe that will
    >   be done by the image baker.
    
    Yes, that needs to be baked into the image. Chocolatey is catastrophically
    slow and unreliable. It's also just bad form to hit any service with such
    repeated downloads.
    
    This is true for *all* of the platforms.
    
    
    > +###############################################################
    > +# Library: pytest
    > +###############################################################
    > +
    > +pytest_enabled = false
    > +pytest = not_found_dep
    > +
    > +pytestopt = get_option('pytest')
    > +if not pytestopt.disabled()
    > +  pytest = find_program(get_option('PYTEST'), native: true, required: pytestopt)
    > +  if pytest.found()
    > +    pytest_check = run_command(pytest,
    > +                               '-c', 'pytest.ini',
    > +                               '--confcutdir=config',
    > +                               '--capture=no',
    > +                               'config/check_pytest.py',
    > +                               '--requirements', 'config/pytest-requirements.txt',
    > +                               check: false)
    > +    if pytest_check.returncode() != 0
    > +      message(pytest_check.stderr())
    > +      if pytestopt.enabled()
    > +        error('Additional Python packages are required to run the pytest suites.')
    > +      else
    > +        warning('Additional Python packages are required to run the pytest suites.')
    > +      endif
    > +    else
    > +      pytest_enabled = true
    > +    endif
    > +  endif
    > +endif
    
    Why do we need pytest the program at all?  Running the tests one-by-one with
    pytest as a runner doesn't seem to make a whole lot of sense to me.
    
    
    > diff --git a/src/test/Makefile b/src/test/Makefile
    > index 511a72e6238..0be9771d71f 100644
    > --- a/src/test/Makefile
    > +++ b/src/test/Makefile
    > @@ -12,7 +12,16 @@ subdir = src/test
    >  top_builddir = ../..
    >  include $(top_builddir)/src/Makefile.global
    >  
    > -SUBDIRS = perl postmaster regress isolation modules authentication recovery subscription
    > +SUBDIRS = \
    > +	authentication \
    > +	isolation \
    > +	modules \
    > +	perl \
    > +	postmaster \
    > +	pytest \
    > +	recovery \
    > +	regress \
    > +	subscription
    
    I'm onboard with that, but we should do it separately and probably check for
    other cases where we should do it at the same time.
    
    I think it'd be a seriously bad idea to start with no central infrastructure,
    we'd be force to duplicate that all over. Eventually we'll be forced to
    introduce some central infrastructure, but we'll probably not go around and
    carefully go through the existing tests for stuff that should now use the
    common infrastructure.
    
    Greetings,
    
    Andres Freund
    
    
    
    

98. Re: RFC: adding pytest as a supported test framework

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-18T10:27:05Z

    On Wed, 17 Dec 2025 at 17:10, Andres Freund <andres@anarazel.de> wrote:
    > I assume this intentionally doesn't pass CI:
    > https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F6045
    
    Yeah it was, but turns out was also actually broken because of that.
    Attached is a new version that actually passes all tests. It also adds
    logic to convert postgres errors into python exceptions. I also moved
    the commits around a bit, so the SSL tests from Jacob are now built on
    top of my improvements to the test infra.
    
    > Why do we need pytest the program at all?  Running the tests one-by-one with
    > pytest as a runner doesn't seem to make a whole lot of sense to me.
    
    Do you mean use "python -m pytest" instead of "pytest"? 
    
    Or do you mean running python files manually somehow? Because that's not
    possible. There's only functions defined in those test files, they're
    not executable by themselves. pytest is still needed to find those in
    each file, as well as the fixtures they require. And ofcourse to make
    assertion errors show up as nicely.
    
    > I think it'd be a seriously bad idea to start with no central infrastructure,
    > we'd be force to duplicate that all over. Eventually we'll be forced to
    > introduce some central infrastructure, but we'll probably not go around and
    > carefully go through the existing tests for stuff that should now use the
    > common infrastructure.
    
    The infra to do query execution on a single postgres server is there
    (patch 0004). That one seemed the most important to me. I'm currently
    still working on some infrastructure to be able to spawn multiple
    postgres servers (I'm validating that by converting the libpq load
    balance TAP tests that I wrote in the past). Is there other
    infrastructure that you think is needed?
    

99. Re: RFC: adding pytest as a supported test framework

    Xuneng Zhou <xunengzhou@gmail.com> — 2025-12-19T14:18:41Z

    Hi Jelte,
    
    Thanks for working on this. I’ve done an initial review of patch 4 and
    here are some comments below.
    
    1) Test infra: tmp_check() fixture looks wrong / unused variable
    def tmp_check(tmp_path_factory):
        d = os.getenv("TESTDATADIR")
        if d:
            d = pathlib.Path(d)
        else:
            d = tmp_path_factory.mktemp("tmp_check")
    
        return tmp_path_factory.mktemp("check_tmp")
    
    You compute d then ignore it and always return a new temp dir
    "check_tmp".  It should probably return d (or d / "check_tmp").
    
    2) PQexec NULL handling is missing
    +    def exec(self, query: str):
    +        ...
    +        res = self._lib.PQexec(self._handle, query.encode())
    +        return self._stack.enter_context(PGresult(self._lib, res))
    
    No NULL check. If PQexec returns NULL (OOM, connection lost), the
    PGresult wrapper will pass it to PQresultStatus etc., causing
    undefined behavior or crash.
    
    3) array parsing may be too simple
    
     _parse_array() does inner.split(","). That will break for valid
    Postgres arrays containing:
    quoted elements with commas: {"a,b","c"}
    escapes / quotes: {"a\"b"} or {"a\\b"}
    nested arrays: {{1,2},{3,4}}
    
    Should we document the constraint here or implement a more complete
    state-machine?
    
    4) Type conversion: timestamp/timestamptz conversion could be wrong
    datetime.datetime.fromisoformat for both timestamp and timestamptz.
    
    - timestamptz output formatting from Postgres is not always
    fromisoformat() friendly (can include timezone formats that differ).
    
    - fromisoformat() yields timezone-aware datetimes only if the string
    has an offset; but Postgres output depends on DateStyle and timezone
    settings.
    
    This could make tests brittle across environments. Should we use
    dateutil.parser (if allowed) or document that the server uses ISO
    settings for pytest.
    
    5) Server init: pg_ctl init vs initdb
    
    In non-Windows branch:
    run(pg_ctl, "-D", datadir, "init")
    
    initdb seems to be a more common way here.
    
    6) Logging config: log_connections = all seems wrong
    print("log_connections = all", file=f)
    
    I don't see an option "all" for this parameter
    https://postgresqlco.nf/doc/en/param/log_connections/
    
    7) UX: error message handling and query attachment
    raise_error() builds message with primary + optional Query: ....
    
    Should we include SQLSTATE and severity in the message string by
    default, because it helps when reading CI logs.
    
    --
    Best,
    Xuneng
    
    
    
    

100. Re: RFC: adding pytest as a supported test framework

     Xuneng Zhou <xunengzhou@gmail.com> — 2025-12-20T03:42:29Z

     Hi,
     
     Thanks for the significant work and effort invested in this project. I
     believe it has the potential to be valuable for PostgreSQL over the
     next 10 years or many more.
     
     I noticed that this topic has gone through a long series of
     discussions in the past. Have we reached consensus on the necessity of
     the project and its overall direction? As we’ve seen before, many
     ambitious initiatives are started but don’t always make it to
     completion. Projects with a clearly articulated roadmap and milestones
     tend to attract and sustain contributor interest more effectively.
     
     Assuming there is interest in continuing and contributing to this
     work, the amount of historical context to absorb could be intimidating
     for new contributors. Do we currently have a wiki page or similar
     document that summarizes the project’s current state and future
     direction? If not, it might be worthwhile to create one—not only to
     lower the barrier to entry for new contributors, but also to support
     long-term maintainability.
     
     -- 
     Best,
     Xuneng
     
     
     
     

101. Re: RFC: adding pytest as a supported test framework

     Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-27T17:26:55Z

     On Fri Dec 19, 2025 at 3:18 PM CET, Xuneng Zhou wrote:
     > Thanks for working on this. I’ve done an initial review of patch 4 and
     > here are some comments below.
     
     Attached is a version where I addressed all of those comemnts (the few
     that I didn't or did in non-obvious ways are discussed at the end). I
     also made a lot of improvements to the patchset:
     1. Includes infrastructure to create multiple Postgres clusters in a
        single test or module.
     2. Basic documentation for the pytest testing infrastructure.
     3. Configure pytest and dependencies in pyproject.toml instead of a
        requirements file.
     4. Set up the environment using "uv"[1] if no pytest binary can be
        found and uv is found.
     5. Use INITDB_TEMPLATE to speed up cluster creation.
     6. Don't enable fsync during tests to speed them up.
     7. I added a patch where I've rewritten the libpq load_balance_hosts
        tests in python as a validation that the new infrastructure and APIs
        work well. (the perl ones were also written by me)
     8. Postgres logs are now included as a separate pytest "section" in the
        output.
     9. The pgtap output now includes all of the pytest sections. For an
        example of what the failure output looks like, take a look here[2]
     
     
     I also *removed* a few things that Jacob had initially added:
     1. The SCRAM based windows auth. I think it's a good idea, but it
        doesn't work with INITDB_TEMPLATE. I think that logic should be made
        part of a separate patchset that stops use trust auth when creating
        the INITDB_TEMPLATE. That way also the perl tests can benefit from it.
        So seems good to do, but separate from the whole pytest work.
     2. I removed the current_windows_user() function. This was dead code (as
        also written in Jacob's comment) and python has built-in ways to get the current user.
     3. I removed the fancy missing/incorrect dependency detection script. I
        think (as Jacob also suggested in his code comments) that
        importorskip is a better fit for this. Especially since we only have
        pytest as a dependency for the core framework, and only the
        cryptography package for the ssl tests.
     
     Finally, I prepared a PR for our images to include the pytest
     dependencies, so in a future version of the patchset we don't need to
     ad-hoc install the required packages.
     
     IMO patch 4 now serves as a good enough central infrastructure base for
     people to develop tests on top of (which would possibly add some more
     infrastructure as needed).
     
     In regards to your second message, related to consensus on the
     necessity of the project: Yes, based on the in-person conversations
     about this people are either in favor of a pytest based test framework,
     or neutral. There were no strong objections. We were now at the point
     where "someone" now actually needs to do the work of getting some decent
     infrastructure in place. Which is what Jacob and me have been trying to
     do.
     
     [1]: https://github.com/astral-sh/uv
     [2]: https://cirrus-ci.com/task/4805772426608640?logs=test_world#L16
     [3]: https://github.com/anarazel/pg-vm-images/pull/130
     
     > 4) Type conversion: timestamp/timestamptz conversion could be wrong
     > datetime.datetime.fromisoformat for both timestamp and timestamptz.
     
     I now configured datestyle to ISO and UTC in postgresql.conf. If that
     turns out not to be enough at some point (e.g. because a test sets a
     different datestyle), we can revisit this.
     
     > 6) Logging config: log_connections = all seems wrong
     > print("log_connections = all", file=f)
     >
     > I don't see an option "all" for this parameter
     > https://postgresqlco.nf/doc/en/param/log_connections/
     
     That's a new value since PG18:
     https://postgresqlco.nf/doc/en/param/log_connections/18/
     
     > 7) UX: error message handling and query attachment
     > raise_error() builds message with primary + optional Query: ....
     >
     > Should we include SQLSTATE and severity in the message string by
     > default, because it helps when reading CI logs.
     
     I don't think adding this additional info helps much anymore, now that
     the postgres logs (item 8) are part of the output too. So I left it like
     this, and even removed the query from the error message. By only
     including the actual error message it's easier to match on it for errors
     that a test expects. Matching on the SQLSTATE can be also done by
     matching on the error type.
     

102. Re: RFC: adding pytest as a supported test framework

     Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-28T23:25:22Z

     On Sat Dec 27, 2025 at 6:26 PM CET, Jelte Fennema-Nio wrote:
     > Attached is a version where I addressed all of those comemnts (the few
     > that I didn't or did in non-obvious ways are discussed at the end). I
     > also made a lot of improvements to the patchset:
     
     Rebased to resolve conflict with master.