Thread

Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-02T02:01:33Z

There has been periodic discussion here about allowing psql to deal
with multiple sessions, or possibly creating another tool to allow
this sort of test.  Is anyone working on this?
 
It's very soon going to be critical that I be able to test particular
interleavings of statements in particular concurrent transaction sets
to be able to make meaningful progress on the serializable
transaction work.  It would be wonderful if some of these scripts
could be integrated into the PostgreSQL 'make check' scripts,
although that's not an absolute requirement.  I'm not really
concerned about performance tests for a while, just testing the
behavior of particular interleavings of statements in multiple
sessions.  If psql isn't expected to support that soon, any
suggestions?  Is pgTAP suited to this?
 
-Kevin

Re: Testing with concurrent sessions

David E. Wheeler <david@kineticode.com> — 2010-01-03T18:53:34Z

On Jan 1, 2010, at 6:01 PM, Kevin Grittner wrote:

> It's very soon going to be critical that I be able to test particular
> interleavings of statements in particular concurrent transaction sets
> to be able to make meaningful progress on the serializable
> transaction work.  It would be wonderful if some of these scripts
> could be integrated into the PostgreSQL 'make check' scripts,
> although that's not an absolute requirement.  I'm not really
> concerned about performance tests for a while, just testing the
> behavior of particular interleavings of statements in multiple
> sessions.  If psql isn't expected to support that soon, any
> suggestions?  Is pgTAP suited to this?

We've discussed it a bit in the past with regard to testing replication and such. I think the consensus was, failing support for concurrent sessions in psql, to use a Perl script to control multiple psql sessions and perhaps use Test::More to do the testing. Although pgTAP might make sense, too, if the tests ought to run in the database.

Best,

David

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-04T23:10:01Z

"David E. Wheeler" <david@kineticode.com> wrote:

> I think the consensus was, failing support for concurrent sessions
> in psql, to use a Perl script to control multiple psql sessions
> and perhaps use Test::More to do the testing.

Are there any examples of that?  While I can hack my way through
regular expressions when I need them, perl as a language is
something I don't know at all; with an example I might be able to
come up to speed quickly, though.

> Although pgTAP might make sense, too, if the 
> tests ought to run in the database.

I need to run statements against a database; I don't particularly
need any special features of psql for this.  Can anyone confirm that
pgTAP can let you interleave specific statements against specific
connections in a specific sequence?  (The answer to that didn't leap
out at me in a quick scan of the docs.)

-Kevin

Re: Testing with concurrent sessions

Peter Eisentraut <peter_e@gmx.net> — 2010-01-04T23:29:37Z

On mån, 2010-01-04 at 17:10 -0600, Kevin Grittner wrote:
> "David E. Wheeler" <david@kineticode.com> wrote:
>  
> > I think the consensus was, failing support for concurrent sessions
> > in psql, to use a Perl script to control multiple psql sessions
> > and perhaps use Test::More to do the testing.
>  
> Are there any examples of that?  While I can hack my way through
> regular expressions when I need them, perl as a language is
> something I don't know at all; with an example I might be able to
> come up to speed quickly, though.

If you're not deep into Perl, perhaps ignore the Test::More comment for
the moment and just use DBI to connect to several database sessions,
execute your queries and check if the results are what you want.  Once
you have gotten somewhere with that, wrapping a test harness around it
is something others will be able to help with.

> > Although pgTAP might make sense, too, if the 
> > tests ought to run in the database.
>  
> I need to run statements against a database; I don't particularly
> need any special features of psql for this.  Can anyone confirm that
> pgTAP can let you interleave specific statements against specific
> connections in a specific sequence?  (The answer to that didn't leap
> out at me in a quick scan of the docs.)

pgTAP isn't really going to help you here, as it runs with *one*
database session, and its main functionality is to format the result of
SQL functions into TAP output, which is not very much like what you
ought to be doing.

Re: Testing with concurrent sessions

David E. Wheeler <david@kineticode.com> — 2010-01-04T23:40:36Z

On Jan 4, 2010, at 3:29 PM, Peter Eisentraut wrote:

> If you're not deep into Perl, perhaps ignore the Test::More comment for
> the moment and just use DBI to connect to several database sessions,
> execute your queries and check if the results are what you want.  Once
> you have gotten somewhere with that, wrapping a test harness around it
> is something others will be able to help with.

Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low).

> pgTAP isn't really going to help you here, as it runs with *one*
> database session, and its main functionality is to format the result of
> SQL functions into TAP output, which is not very much like what you
> ought to be doing.

Right, exactly.

Best,

David

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-05T12:08:41Z

Hi,

Kevin Grittner wrote:
> It's very soon going to be critical that I be able to test particular
> interleavings of statements in particular concurrent transaction sets
> to be able to make meaningful progress on the serializable
> transaction work.

I've something in place for Postgres-R, as I also need to test
concurrent transactions there. It's based on python/twisted and is able
to start multiple Postgres instances (as required for testing
replication) and query them concurrently (as you seem to need as well).
It uses an asynchronous event loop (from twisted) and basically controls
processes, issues queries and checks results and ordering constraints
(e.g. transaction X must commit and return a result before transaction Y).

I'm still under the impression that this testing framework needs
cleanup. However, others already showed interest as well...

Regards

Markus Wanner

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-05T16:48:11Z

Markus Wanner <markus@bluegap.ch> wrote:
 
> Kevin Grittner wrote:
>> It's very soon going to be critical that I be able to test
>> particular interleavings of statements in particular concurrent
>> transaction sets to be able to make meaningful progress on the
>> serializable transaction work.
> 
> I've something in place for Postgres-R, as I also need to test
> concurrent transactions there. It's based on python/twisted and is
> able to start multiple Postgres instances (as required for testing
> replication) and query them concurrently (as you seem to need as
> well).  It uses an asynchronous event loop (from twisted) and
> basically controls processes, issues queries and checks results
> and ordering constraints (e.g. transaction X must commit and
> return a result before transaction Y).
 
Where would I find this (and any related documentation)?
 
-Kevin

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-05T19:06:37Z

Hi,

Kevin Grittner wrote:
> Where would I find this (and any related documentation)?

Sorry, if that didn't get clear. I'm trying to put together something I 
can release real soon now (tm). I'll keep you informed.

Regards

Markus Wanner

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-06T21:52:51Z

"David E. Wheeler" <david@kineticode.com> wrote:

> Last I heard, Andrew was willing to require Test::More for
> testing, so that a Perl script could handle multiple psql
> connections (perhaps forked) and output test results based on
> them. But he wasn't as interested in requiring DBI and DBD::Pg,
> neither of which are in the Perl core and are more of a PITA to
> install (not huge, but the barrier might as well stay low).

OK, I've gotten familiar with Perl as a programming language and
tinkered with Test::More.  What's not clear to me yet is what would
be considered good technique for launching several psql sessions
from that environment, interleaving commands to each of them, and
checking results from each of them as the test plan progresses.  Any
code snippets or URLs to help me understand that are welcome.  (It
seems clear enough with DBI, but I'm trying to avoid that per the
above.)

-Kevin

Re: Testing with concurrent sessions

Peter Eisentraut <peter_e@gmx.net> — 2010-01-06T22:08:29Z

On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote:
> "David E. Wheeler" <david@kineticode.com> wrote:
>  
> > Last I heard, Andrew was willing to require Test::More for
> > testing, so that a Perl script could handle multiple psql
> > connections (perhaps forked) and output test results based on
> > them. But he wasn't as interested in requiring DBI and DBD::Pg,
> > neither of which are in the Perl core and are more of a PITA to
> > install (not huge, but the barrier might as well stay low).
>  
> OK, I've gotten familiar with Perl as a programming language and
> tinkered with Test::More.  What's not clear to me yet is what would
> be considered good technique for launching several psql sessions
> from that environment, interleaving commands to each of them, and
> checking results from each of them as the test plan progresses.  Any
> code snippets or URLs to help me understand that are welcome.  (It
> seems clear enough with DBI, but I'm trying to avoid that per the
> above.)

Then I don't see much of a point in using Perl.  You might as well fire
up a few psqls from a shell script.

Re: Testing with concurrent sessions

David E. Wheeler <david@kineticode.com> — 2010-01-06T22:09:39Z

On Jan 6, 2010, at 1:52 PM, Kevin Grittner wrote:

>> Last I heard, Andrew was willing to require Test::More for
>> testing, so that a Perl script could handle multiple psql
>> connections (perhaps forked) and output test results based on
>> them. But he wasn't as interested in requiring DBI and DBD::Pg,
>> neither of which are in the Perl core and are more of a PITA to
>> install (not huge, but the barrier might as well stay low).
> 
> OK, I've gotten familiar with Perl as a programming language and
> tinkered with Test::More.  What's not clear to me yet is what would
> be considered good technique for launching several psql sessions
> from that environment, interleaving commands to each of them, and
> checking results from each of them as the test plan progresses.  Any
> code snippets or URLs to help me understand that are welcome.  (It
> seems clear enough with DBI, but I'm trying to avoid that per the
> above.)

Probably the simplest way is to use the core IPC::Open3 module:

    http://search.cpan.org/perldoc?IPC::Open3

IPC::Run might be easier to use if it's available, but it's not  in Perl core, so YMMV. Really it's up to andrew what modules he requires test servers to have.

Best,

David

Re: Testing with concurrent sessions

David E. Wheeler <david@kineticode.com> — 2010-01-06T22:10:15Z

On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote:

> Then I don't see much of a point in using Perl.  You might as well fire
> up a few psqls from a shell script

If you're more comfortable with shell, then yes. Although then it won't run on Windows, will it?

Best,

David

Re: Testing with concurrent sessions

Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> — 2010-01-06T22:13:22Z

On 2010-01-07 00:08 +0200, Peter Eisentraut wrote:
> On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote:
>> "David E. Wheeler"<david@kineticode.com>  wrote:
>>
>>> Last I heard, Andrew was willing to require Test::More for
>>> testing, so that a Perl script could handle multiple psql
>>> connections (perhaps forked) and output test results based on
>>> them. But he wasn't as interested in requiring DBI and DBD::Pg,
>>> neither of which are in the Perl core and are more of a PITA to
>>> install (not huge, but the barrier might as well stay low).
>>
>> OK, I've gotten familiar with Perl as a programming language and
>> tinkered with Test::More.  What's not clear to me yet is what would
>> be considered good technique for launching several psql sessions
>> from that environment, interleaving commands to each of them, and
>> checking results from each of them as the test plan progresses.  Any
>> code snippets or URLs to help me understand that are welcome.  (It
>> seems clear enough with DBI, but I'm trying to avoid that per the
>> above.)
>
> Then I don't see much of a point in using Perl.  You might as well fire
> up a few psqls from a shell script.

I don't see how that would work, but I might have misunderstood what 
we're reaching for here.  What I think would be most useful would be to 
interleave statements between transactions, not just randomly fire psql 
sessions and hope for race conditions.


Regards,
Marko Tiikkaja

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-06T22:24:06Z

Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> wrote:
> On 2010-01-07 00:08 +0200, Peter Eisentraut wrote:

>> Then I don't see much of a point in using Perl.  You might as
>> well fire up a few psqls from a shell script.
> 
> I don't see how that would work, but I might have misunderstood
> what we're reaching for here.  What I think would be most useful
> would be to interleave statements between transactions, not just
> randomly fire psql sessions and hope for race conditions.

Yeah, I want to test specific interleavings of statements on
concurrent connections.  There may *also* be some tests which throw
a lot at the server concurrently in a more random fashion, but it is
important to be able to have some very controlled tests where we
don't count on randomly creating the desired conflicts.

It would be valuable to be able to include some of these tests with
controlled and predicatable statement interleavings in the "make
check" tests.

-Kevin

Re: Testing with concurrent sessions

Alvaro Herrera <alvherre@commandprompt.com> — 2010-01-06T23:04:32Z

Marko Tiikkaja wrote:
> On 2010-01-07 00:08 +0200, Peter Eisentraut wrote:

> >Then I don't see much of a point in using Perl.  You might as well fire
> >up a few psqls from a shell script.
> 
> I don't see how that would work, but I might have misunderstood what
> we're reaching for here.  What I think would be most useful would be
> to interleave statements between transactions, not just randomly
> fire psql sessions and hope for race conditions.

Open a few psql with -f pointing to a pipe, and from the shell write
into the pipe?  I don't think it's straightforward, but it should be
possible.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-06T23:11:18Z

Alvaro Herrera <alvherre@commandprompt.com> wrote:
 
> Open a few psql with -f pointing to a pipe, and from the shell
> write into the pipe?  I don't think it's straightforward, but it
> should be possible.
 
I'll play with it and see what I can do.
 
Thanks,
 
-Kevin

Re: Testing with concurrent sessions

Robert Haas <robertmhaas@gmail.com> — 2010-01-07T01:15:13Z

On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
> "David E. Wheeler" <david@kineticode.com> wrote:
>
>> Last I heard, Andrew was willing to require Test::More for
>> testing, so that a Perl script could handle multiple psql
>> connections (perhaps forked) and output test results based on
>> them. But he wasn't as interested in requiring DBI and DBD::Pg,
>> neither of which are in the Perl core and are more of a PITA to
>> install (not huge, but the barrier might as well stay low).
>
> OK, I've gotten familiar with Perl as a programming language and
> tinkered with Test::More.  What's not clear to me yet is what would
> be considered good technique for launching several psql sessions
> from that environment, interleaving commands to each of them, and
> checking results from each of them as the test plan progresses.  Any
> code snippets or URLs to help me understand that are welcome.  (It
> seems clear enough with DBI, but I'm trying to avoid that per the
> above.)

Doing this without DBI is going to be ten times harder than doing it
with DBI.  Are we really sure that's not a viable option?

...Robert

Re: Testing with concurrent sessions

Andrew Dunstan <andrew@dunslane.net> — 2010-01-07T01:40:28Z


Robert Haas wrote:
> On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner
> <Kevin.Grittner@wicourts.gov> wrote:
>   
>> "David E. Wheeler" <david@kineticode.com> wrote:
>>
>>     
>>> Last I heard, Andrew was willing to require Test::More for
>>> testing, so that a Perl script could handle multiple psql
>>> connections (perhaps forked) and output test results based on
>>> them. But he wasn't as interested in requiring DBI and DBD::Pg,
>>> neither of which are in the Perl core and are more of a PITA to
>>> install (not huge, but the barrier might as well stay low).
>>>       
>> OK, I've gotten familiar with Perl as a programming language and
>> tinkered with Test::More.  What's not clear to me yet is what would
>> be considered good technique for launching several psql sessions
>> from that environment, interleaving commands to each of them, and
>> checking results from each of them as the test plan progresses.  Any
>> code snippets or URLs to help me understand that are welcome.  (It
>> seems clear enough with DBI, but I'm trying to avoid that per the
>> above.)
>>     
>
> Doing this without DBI is going to be ten times harder than doing it
> with DBI.  Are we really sure that's not a viable option?
>
>
>   

In the buildfarm? Yes, I think so. The philosophy of the buildfarm is 
that it should do what you would do yourself by hand.

And adding DBI as a requirement for running a buildfarm member would be 
a significant extra barrier to entry, ISTM. (I am very fond of DBI, and 
use it frequently, BTW)

I'm persuadable on most things, but this one would take a bit of doing.

A parallel psql seems to me a better way to go. We talked about that a 
while ago, but I don't recall what happened to it.

cheers

andrew

Re: Testing with concurrent sessions

Robert Haas <robertmhaas@gmail.com> — 2010-01-07T01:49:38Z

On Wed, Jan 6, 2010 at 8:40 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
>> Doing this without DBI is going to be ten times harder than doing it
>> with DBI.  Are we really sure that's not a viable option?
>
> In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that
> it should do what you would do yourself by hand.

It just seems crazy to me to try to test anything without proper
language bindings.  Opening a psql session and parsing the results
seems extraordinarily painful.  I wonder if it would make sense write
a small wrapper program that uses libpq and dumps out the results in a
format that is easy for Perl to parse.

Another idea would be to make a set of Perl libpq bindings that is
simpler than DBD::Pg and don't go through DBI.  If we put those in the
main source tree (perhaps as a contrib module) they would be available
wherever we need them.

> A parallel psql seems to me a better way to go. We talked about that a while
> ago, but I don't recall what happened to it.

That seems like a dead-end to me.  It's hard for me to imagine it's
ever going to be more than a toy.

...Robert

Re: Testing with concurrent sessions

Tom Lane <tgl@sss.pgh.pa.us> — 2010-01-07T02:26:29Z

Robert Haas <robertmhaas@gmail.com> writes:
> It just seems crazy to me to try to test anything without proper
> language bindings.  Opening a psql session and parsing the results
> seems extraordinarily painful.  I wonder if it would make sense write
> a small wrapper program that uses libpq and dumps out the results in a
> format that is easy for Perl to parse.

> Another idea would be to make a set of Perl libpq bindings that is
> simpler than DBD::Pg and don't go through DBI.  If we put those in the
> main source tree (perhaps as a contrib module) they would be available
> wherever we need them.

We have not yet fully accepted the notion that you must have Perl to
build (and, in fact, I am right now trying to verify that you don't).
I don't think that requiring Perl to test is going to fly.

>> A parallel psql seems to me a better way to go. We talked about that a while
>> ago, but I don't recall what happened to it.

> That seems like a dead-end to me.  It's hard for me to imagine it's
> ever going to be more than a toy.

Well, the argument there is that it might be useful for actual use,
not only testing.

			regards, tom lane

Re: Testing with concurrent sessions

Robert Haas <robertmhaas@gmail.com> — 2010-01-07T02:49:38Z

On Wed, Jan 6, 2010 at 9:26 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> It just seems crazy to me to try to test anything without proper
>> language bindings.  Opening a psql session and parsing the results
>> seems extraordinarily painful.  I wonder if it would make sense write
>> a small wrapper program that uses libpq and dumps out the results in a
>> format that is easy for Perl to parse.
>
>> Another idea would be to make a set of Perl libpq bindings that is
>> simpler than DBD::Pg and don't go through DBI.  If we put those in the
>> main source tree (perhaps as a contrib module) they would be available
>> wherever we need them.
>
> We have not yet fully accepted the notion that you must have Perl to
> build (and, in fact, I am right now trying to verify that you don't).
> I don't think that requiring Perl to test is going to fly.

I suppose that depends on the context.  I'm not exactly sure what
Kevin's goal is here.  For basic regression tests, yeah, we'd probably
like to keep that Perl-free.  For more complex testing, I think using
Perl makes sense.  Or to put the shoe on the other foot, if we DON'T
allow the use of Perl for more complex testing, then we're probably
not going to have any more complex tests.  If we use a hypothetical
concurrent psql implementation to run the tests, how will we analyze
the results?  It's no secret that the current regression tests are
fairly limited, in part because the only thing we can do with them is
diff the output against one or two "known good" results.

...Robert

Re: Testing with concurrent sessions

Greg Smith <greg@2ndquadrant.com> — 2010-01-07T05:29:22Z

Robert Haas wrote:
> It just seems crazy to me to try to test anything without proper
> language bindings.  Opening a psql session and parsing the results
> seems extraordinarily painful.

I've written a Python based program that spawns a captive psql and talks 
to it--twice for different people--that ultimately uses the same sort of 
open3() spawning David mentioned is available via IPC::Open3.  You can 
throw together a prototype that works well enough for some purposes in a 
couple of hours.  I don't know that it would ever reach really robust 
though.

The main problem with that whole approach is that you have to be 
extremely careful in how you deal with the situation where the captive 
program is spewing an unknown amount of information back at you.  How do 
you know when it's done?  Easy for the child and its master to deadlock 
if you're not careful.  In the past I worked around that issue by just 
waiting for the process to end and then returning everything it had 
written until that time.  I don't know that this would be flexible 
enough for what's needed for concurrent testing, where people are 
probably going to want more of a "send a command, get some lines back 
again" approach that keeps the session open.

If I thought a captive psql would work well in this context I'd have 
written a prototype already.  I'm not sure if it's actually possible to 
do this well enough to meet expectations.  Parsing psql output is 
completely viable for trivial purposes though, and if the requirements 
were constrained enough it might work well enough for simple concurrent 
testing.  While both concurrent psql and the libpq shim you suggested 
would take more work, I feel a bit more confident those would result in 
something that really worked as expected on every platform when finished. 

-- 
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

Re: Testing with concurrent sessions

Peter Eisentraut <peter_e@gmx.net> — 2010-01-07T05:30:29Z

On ons, 2010-01-06 at 14:10 -0800, David E. Wheeler wrote:
> On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote:
> 
> > Then I don't see much of a point in using Perl.  You might as well fire
> > up a few psqls from a shell script
> 
> If you're more comfortable with shell, then yes. Although then it won't run on Windows, will it?

Well, I'm not saying that this is necessarily the better alternative.
But you might have to compromise somewhere.  Otherwise this project will
take two years to complete and will be an unmaintainable mess (see
pg_regress).

Re: Testing with concurrent sessions

Peter Eisentraut <peter_e@gmx.net> — 2010-01-07T05:38:48Z

On ons, 2010-01-06 at 20:49 -0500, Robert Haas wrote:
> Another idea would be to make a set of Perl libpq bindings that is
> simpler than DBD::Pg and don't go through DBI.  If we put those in the
> main source tree (perhaps as a contrib module) they would be available
> wherever we need them. 

http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/interfaces/perl5/?hideattic=0

Re: Testing with concurrent sessions

Craig Ringer <craig@postnewspapers.com.au> — 2010-01-07T09:50:38Z

On 7/01/2010 9:15 AM, Robert Haas wrote:
> On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner
> <Kevin.Grittner@wicourts.gov>  wrote:
>> "David E. Wheeler"<david@kineticode.com>  wrote:
>>
>>> Last I heard, Andrew was willing to require Test::More for
>>> testing, so that a Perl script could handle multiple psql
>>> connections (perhaps forked) and output test results based on
>>> them. But he wasn't as interested in requiring DBI and DBD::Pg,
>>> neither of which are in the Perl core and are more of a PITA to
>>> install (not huge, but the barrier might as well stay low).
>>
>> OK, I've gotten familiar with Perl as a programming language and
>> tinkered with Test::More.  What's not clear to me yet is what would
>> be considered good technique for launching several psql sessions
>> from that environment, interleaving commands to each of them, and
>> checking results from each of them as the test plan progresses.  Any
>> code snippets or URLs to help me understand that are welcome.  (It
>> seems clear enough with DBI, but I'm trying to avoid that per the
>> above.)
>
> Doing this without DBI is going to be ten times harder than doing it
> with DBI.  Are we really sure that's not a viable option?

At this point, I'm personally wondering if it's worth putting together a 
simple (ish) C program that reads a file describing command 
interleavings on n connections. It fires up one thread per connection 
required, then begins queuing commands up for the threads to execute in 
per-thread fifo queues. The input file may contain synchronization 
points where two or more explicitly specified threads (or just all 
threads) must finish all their queued work before they may be given more.

Yes, it requires wrangling low-level threading ( pthreads, or the 
practically equivalent for simple purposes but differently spelled win32 
threading ) so it's not going to be beautiful. But it'd permit a 
declarative form for tests and a single, probably fairly maintainable, 
test runner.

I reluctantly suspect that XML would be a good way to describe the tests 
- first a block declaring your connections and their conn strings, then 
a sequence of statements (each of which is associated with a named 
connection) and synchronization points. Though, come to think of it, a 
custom plaintext format would be pretty trivial too.

CONN conn1: dbname=regress, user=regress
CONN conn2: dbname=regress, user=regress
STMT conn1: SELECT blah blah;
STMT conn2: UPDATE blah blah;
SYNC conn1, conn2

etc. Or alternately one-file-per-connection (which would be handy if one 
connection has *lots* of commands and others only occasional ones) - the 
only trouble there being how to conveniently specify synchronization points.

Anyway: If Java were acceptable I'd put one together now - but somehow I 
don't think requiring Java would be popular if Perl is an issue ;-) My 
C/pthreads is more than a little bit rusty (ie: practially nonexistent) 
and mostly confined to exception-controlled C++ code with RAII for lock 
management. If C++ is OK, I can write and post a tool for evaluation, 
but if it must be plain C ... well, I'll avoid scarring you with the 
sight of what I'd produce.

I suspect that a custom tool for this job could actually be fairly 
simple. A lot simpler than a proper, clean and usable parallel psql, anyway.

--
Craig Ringer

Re: Testing with concurrent sessions

Greg Sabino Mullane <greg@turnstep.com> — 2010-01-07T15:17:15Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


>> Doing this without DBI is going to be ten times harder than doing it
>> with DBI.  Are we really sure that's not a viable option?

> In the buildfarm? Yes, I think so. The philosophy of the buildfarm is
> that it should do what you would do yourself by hand.
>
> And adding DBI as a requirement for running a buildfarm member would be
> a significant extra barrier to entry, ISTM. (I am very fond of DBI, and
> use it frequently, BTW)

What about something less than a requirement then? If you have it great,
you can run these extra tests. If you don't have it, no harm, no foul.

We could even bundle DBI and DBD::Pg to ensure that the minimum versions
are there. All the prerequisites should be in place for 99% of the machines:
a C compiler and Perl are the biggies, and I can't see any buildfarm members
running without those. :)

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 201001071014
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAktF+ucACgkQvJuQZxSWSsjYOgCglyLIyGCr60og+iQSnyRgkowd
+lYAnRDjPe/XxC7gb9OBPdpZlqU0wncK
=kPIR
-----END PGP SIGNATURE-----

Re: Testing with concurrent sessions

Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> — 2010-01-07T16:13:59Z

On 2010-01-07 11:50 +0200, Craig Ringer wrote:
> On 7/01/2010 9:15 AM, Robert Haas wrote:
>> Doing this without DBI is going to be ten times harder than doing it
>> with DBI.  Are we really sure that's not a viable option?
>
> At this point, I'm personally wondering if it's worth putting together a
> simple (ish) C program that reads a file describing command
> interleavings on n connections. It fires up one thread per connection
> required, then begins queuing commands up for the threads to execute in
> per-thread fifo queues. The input file may contain synchronization
> points where two or more explicitly specified threads (or just all
> threads) must finish all their queued work before they may be given more.

> CONN conn1: dbname=regress, user=regress
> CONN conn2: dbname=regress, user=regress
> STMT conn1: SELECT blah blah;
> STMT conn2: UPDATE blah blah;
> SYNC conn1, conn2
>
> etc. Or alternately one-file-per-connection (which would be handy if one
> connection has *lots* of commands and others only occasional ones) - the
> only trouble there being how to conveniently specify synchronization points.

I had a similar syntax in mind, but instead of using threads, just 
execute the file in order using asynchronous connections.


Regards,
Marko Tiikkaja

Re: Testing with concurrent sessions

Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> — 2010-01-07T16:27:15Z

On 2010-01-07 18:13 +0200, Marko Tiikkaja wrote:
> I had a similar syntax in mind, but instead of using threads, just
> execute the file in order using asynchronous connections.

I completely failed to make the point here which was to somehow mark 
which statements will (or, should) block.  So here we go:

A=> BEGIN;
B=> BEGIN;
A=> UPDATE foo ..;
&B=> UPDATE foo ..; -- this will block
A=> COMMIT;
B=> SELECT * FROM foo;
B=> COMMIT;

[expected output here]


Regards,
Marko Tiikkaja

Re: Testing with concurrent sessions

David E. Wheeler <david@kineticode.com> — 2010-01-07T16:32:02Z

On Jan 6, 2010, at 6:26 PM, Tom Lane wrote:

> We have not yet fully accepted the notion that you must have Perl to
> build (and, in fact, I am right now trying to verify that you don't).
> I don't think that requiring Perl to test is going to fly.

I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it.

Best,

David

Re: Testing with concurrent sessions

Tom Lane <tgl@sss.pgh.pa.us> — 2010-01-07T16:38:12Z

"Greg Sabino Mullane" <greg@turnstep.com> writes:
> We could even bundle DBI and DBD::Pg to ensure that the minimum versions
> are there.

As a packager, my reaction to that is "over my dead body".  We have
enough trouble keeping our own software up to date, and pretty much
every external component that we've started to bundle has been a
disaster from a maintenance standpoint.  (Examples: the zic database
is constant work and yet almost never up to date; the snowball stemmer
never gets updated.)

			regards, tom lane

Re: Testing with concurrent sessions

Andrew Dunstan <andrew@dunslane.net> — 2010-01-07T16:46:53Z

David E. Wheeler wrote:
> On Jan 6, 2010, at 6:26 PM, Tom Lane wrote:
>
>   
>> We have not yet fully accepted the notion that you must have Perl to
>> build (and, in fact, I am right now trying to verify that you don't).
>> I don't think that requiring Perl to test is going to fly.
>>     
>
> I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it.
>
>
>   

Unless I am mistaken, Perl is required in any case to build from CVS, 
although not from a tarball.

DBI/DBD::pg is quite another issue, however. I have been deliberately 
very conservative about what modules to require for the buildfarm, and 
we have similarly (and I think wisely) been conservative about what 
modules to require for Perl programs in the build process.

Using DBI/DBD::Pg would raise another issue - what version of libpq 
would it be using? Not the one in the build being tested, that's for 
sure. If you really want to use Perl then either a Pure Perl DBI driver 
(which Greg has talked about) or a thin veneer over libpq such as we 
used to have in contrib seems a safer way to go.

cheers

Re: Testing with concurrent sessions

Robert Haas <robertmhaas@gmail.com> — 2010-01-07T16:57:43Z

On Thu, Jan 7, 2010 at 11:46 AM, Andrew Dunstan <andrew@dunslane.net> wrote:
> Using DBI/DBD::Pg would raise another issue - what version of libpq would it
> be using? Not the one in the build being tested, that's for sure. If you
> really want to use Perl then either a Pure Perl DBI driver (which Greg has
> talked about) or a thin veneer over libpq such as we used to have in contrib
> seems a safer way to go.

I completely agree.  As between those two options, count me as +1 for
the thin veneer.

...Robert

Re: Testing with concurrent sessions

David Fetter <david@fetter.org> — 2010-01-07T16:58:30Z

On Wed, Jan 06, 2010 at 08:40:28PM -0500, Andrew Dunstan wrote:
> A parallel psql seems to me a better way to go. We talked about that
> a while ago, but I don't recall what happened to it.

Greg Stark had a patch a couple of years ago.  Dunno what happened to
it since then.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Re: Testing with concurrent sessions

Tom Lane <tgl@sss.pgh.pa.us> — 2010-01-07T17:08:33Z

Andrew Dunstan <andrew@dunslane.net> writes:
> Unless I am mistaken, Perl is required in any case to build from CVS, 
> although not from a tarball.

Right, but to my mind "building from a tarball" needs to include the
ability to run the regression tests on what you built.  So injecting
Perl into that is moving the goalposts on build requirements.

			regards, tom lane

Re: Testing with concurrent sessions

David E. Wheeler <david@kineticode.com> — 2010-01-07T17:10:59Z

On Jan 7, 2010, at 9:08 AM, Tom Lane wrote:

> Right, but to my mind "building from a tarball" needs to include the
> ability to run the regression tests on what you built.  So injecting
> Perl into that is moving the goalposts on build requirements.

In that case, there's nothing for it except concurrent psql. Or else some sort of shell environment that's available on all platforms. do we require bash on Windows? Oh, wait, the Windows build requires Perl…

Best,

David

Re: Testing with concurrent sessions

Tom Lane <tgl@sss.pgh.pa.us> — 2010-01-07T17:19:50Z

"David E. Wheeler" <david@kineticode.com> writes:
> On Jan 7, 2010, at 9:08 AM, Tom Lane wrote:
>> Right, but to my mind "building from a tarball" needs to include the
>> ability to run the regression tests on what you built.  So injecting
>> Perl into that is moving the goalposts on build requirements.

> In that case, there's nothing for it except concurrent psql.

Unless we are prepared to define concurrency testing as something
separate from the basic regression tests.  Which is kind of annoying but
perhaps less so than the alternatives.  It certainly seems to me to
be the kind of thing you wouldn't need to test in order to have
confidence in a local build.

			regards, tom lane

Re: Testing with concurrent sessions

David E. Wheeler <david@kineticode.com> — 2010-01-07T17:22:45Z

On Jan 7, 2010, at 9:19 AM, Tom Lane wrote:

>> In that case, there's nothing for it except concurrent psql.
> 
> Unless we are prepared to define concurrency testing as something
> separate from the basic regression tests.  Which is kind of annoying but
> perhaps less so than the alternatives.  It certainly seems to me to
> be the kind of thing you wouldn't need to test in order to have
> confidence in a local build.

I was rather assuming that was what we were talking about here, since we have in the past discussed testing things like dump and restore, which would require something like Perl to handle multiple processes, and wouldn't work very well for a regular release.

I think if we have the ability to add tests that are not distributed, it gives us a lot more freedom and opportunity to test things that are not currently well-tested.

Best,

David

Re: Testing with concurrent sessions

Greg Sabino Mullane <greg@turnstep.com> — 2010-01-07T17:34:52Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


> Unless we are prepared to define concurrency testing as something
> separate from the basic regression tests.  Which is kind of annoying but
> perhaps less so than the alternatives.  It certainly seems to me to
> be the kind of thing you wouldn't need to test in order to have
> confidence in a local build.

I thought we were leaning towards something separate.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 201001071234
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAktGGxYACgkQvJuQZxSWSsgG0gCfZ4eTpXd/97p3zSJpLqGhKMh6
8nMAoJ2lQUaCWNVeSPDU8fq7VnkO0s4C
=xBOo
-----END PGP SIGNATURE-----

Re: Testing with concurrent sessions

Greg Sabino Mullane <greg@turnstep.com> — 2010-01-07T17:39:55Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


> Using DBI/DBD::Pg would raise another issue - what version of libpq
> would it be using? Not the one in the build being tested, that's for
> sure.

Er...why not? That's what psql uses. As for those advocating using a
custom C program written using libpq - that's basically what
DBI/DBD::Pg ends up being! Only with a shiny Perl outside and years
of real-world testing and usage.

> If you really want to use Perl then either a Pure Perl DBI driver
> (which Greg has talked about) or a thin veneer over libpq such as we
> used to have in contrib seems a safer way to go.

I'm still *very* interested in making a libpq-less pure perl driver,
if anyone feels like funding it, let me know! :)

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 201001071236
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAktGHF0ACgkQvJuQZxSWSsiFmACfUJbRDUJGvDTJNjgj/dyQKVCA
tZwAn2fiXKNWbWzYXobrHZjeE8aSSiVv
=sGzK
-----END PGP SIGNATURE-----

Re: Testing with concurrent sessions

Greg Sabino Mullane <greg@turnstep.com> — 2010-01-07T18:16:08Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


>> We could even bundle DBI and DBD::Pg to ensure that the minimum versions
>> are there.

> As a packager, my reaction to that is "over my dead body".  We have
> enough trouble keeping our own software up to date, and pretty much
> every external component that we've started to bundle has been a
> disaster from a maintenance standpoint.  (Examples: the zic database
> is constant work and yet almost never up to date; the snowball stemmer
> never gets updated.)

As a counterargument, I'll point out that this won't be as critical
as zic, especially if we're talking about an additional/optional
set of tests. Also, Tim Bunce and I are right here, so the maintenance
should not be that bad (and I'd hazard that a lot more people in
the community know Perl/DBI than zic or stemmers).

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 201001071315
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAktGJNQACgkQvJuQZxSWSshoHwCg9urTTT19m55soiIjuYKKuB5L
PjYAoJbDAe7j4xsxsSFfVEkYDFpTjKE9
=48oW
-----END PGP SIGNATURE-----

Re: Testing with concurrent sessions

A.M. <agentm@themactionfaction.com> — 2010-01-07T22:41:41Z

On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote:
> I'm still *very* interested in making a libpq-less pure perl driver,
> if anyone feels like funding it, let me know! :)

You mean this one:

http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm

?

Cheers,
M

Re: Testing with concurrent sessions

Andrew Dunstan <andrew@dunslane.net> — 2010-01-07T22:56:24Z


A.M. wrote:
> On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote:
>   
>> I'm still *very* interested in making a libpq-less pure perl driver,
>> if anyone feels like funding it, let me know! :)
>>     
>
> You mean this one:
>
> http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm
>
> ?
>
>
>   

It has a list of limitations as long as your arm.

cheers

andrew

Re: Testing with concurrent sessions

Craig Ringer <craig@postnewspapers.com.au> — 2010-01-10T01:40:49Z

On 8/01/2010 1:39 AM, Greg Sabino Mullane wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
>
>> Using DBI/DBD::Pg would raise another issue - what version of libpq
>> would it be using? Not the one in the build being tested, that's for
>> sure.
>
> Er...why not? That's what psql uses.

Because you'd have to build DBD::Pg against the new libpq, as you do 
psql. That means you need DBD::Pg sources and the build environment for 
Perl (headers etc) not just a working Perl runtime. Big difference.

There's no guarantee the user's machine has the same major version of 
libpq and thus no guarantee that DBD::Pq can be redirected to use your 
custom libpq by LD_LIBRARY_PATH. It might also override the library 
search path with rpath linking. Building your own would be pretty much 
unavoidable unless you're prepared to either require the user to provide 
a matching version of DBD::Pg or have the tests running with whatever 
random version happens to be lying around.

Using whatever DBD::Pg version happens to be present on the machine 
would be a bit of a nightmare for reproducibility of test results, and 
would be really unattractive for use in the standard tests. "make check 
fails on my some-random-distro" would become painfully common on the 
GENERAL list...

Is bundling a Perl module in the source tree and building it as part of 
the Pg build a reasonable choice? Personally, I don't think so.

Additionally, a dedicated testing tool like some folks have been talking 
about would be really handy for users who want to test their schema. 
I've had to write my own (in Java, or I'd be offering it) for this 
purpose, as psql is completely unsuitable for concurrent-run testing and 
I needed to show that my locking was safe and deadlock-free in some of 
the more complex stored procs I have.

--
Craig Ringer

Re: Testing with concurrent sessions

Greg Sabino Mullane <greg@turnstep.com> — 2010-01-11T04:17:42Z

-----BEGIN PGP SIGNED MESSAGE-----                                          
Hash: RIPEMD160                                                             


>> Using DBI/DBD::Pg would raise another issue - what version of libpq
>> would it be using? Not the one in the build being tested, that's for
>> sure.                                                               
>                                                                      
> Er...why not? That's what psql uses.                                 

> Because you'd have to build DBD::Pg against the new libpq, as you do 
> psql. That means you need DBD::Pg sources and the build environment for 
> Perl (headers etc) not just a working Perl runtime. Big difference.     

Yes, but that is what I was envisioning. As you point out, that's the
only sane way to make sure we have a good version of DBD::Pg with
which to test. As a side effect, it put libpq through some extra
paces as well. :)

> Is bundling a Perl module in the source tree and building it as part of
> the Pg build a reasonable choice? Personally, I don't think so.

*shrug* It's different, but it's the best solution to the problem at
hand. It wouldn't be built as part of Pg, only as part of the tests.

> Additionally, a dedicated testing tool like some folks have been talking
> about would be really handy for users who want to test their schema.
> I've had to write my own (in Java, or I'd be offering it) for this
> purpose, as psql is completely unsuitable for concurrent-run testing and
> I needed to show that my locking was safe and deadlock-free in some of
> the more complex stored procs I have.

Sure, but it's the difference between waiting for someone to write something
(and then dealing with the invevitable bugs, tweaks, and enhancements), or
using a solid, known quantity (DBI + Test::More).

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 201001102316
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAktKplIACgkQvJuQZxSWSsj7TQCeMOXWS+uLIZE9QbeBWPxYv/rg
HhEAn0QZUzE2/8uyg5Oi+K8qL/oTeDSO
=R8Az
-----END PGP SIGNATURE-----

Re: Testing with concurrent sessions

Martijn van Oosterhout <kleptog@svana.org> — 2010-01-11T08:46:21Z

On Mon, Jan 11, 2010 at 04:17:42AM -0000, Greg Sabino Mullane wrote:
> > Because you'd have to build DBD::Pg against the new libpq, as you do 
> > psql. That means you need DBD::Pg sources and the build environment for 
> > Perl (headers etc) not just a working Perl runtime. Big difference.     
> 
> Yes, but that is what I was envisioning. As you point out, that's the
> only sane way to make sure we have a good version of DBD::Pg with
> which to test. As a side effect, it put libpq through some extra
> paces as well. :)

Is there a reason why you're suggesting using DBI? There is also the Pg
perl module which works as well and is one tenth of the size. It also
doesn't have external dependancies. It's just a plain wrapper around
libpq, which for the purposes of testing may be better.

http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm

Have a nice day,
-- 
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while 
> boarding. Thank you for flying nlogn airlines.

Re: Testing with concurrent sessions

Greg Sabino Mullane <greg@turnstep.com> — 2010-01-11T17:42:08Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


> Is there a reason why you're suggesting using DBI? There is also the Pg
> perl module which works as well and is one tenth of the size. It also
> doesn't have external dependancies. It's just a plain wrapper around
> libpq, which for the purposes of testing may be better.
>
> http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm

Works as well? Did you take a look at that link? The last update was
early 2000, which should give you an indication of just how dead
it is.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 201001111241
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8

-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAktLYtwACgkQvJuQZxSWSsjNJgCg1IZBNIZsUMZ2V83/NjgJnrGO
3NEAoK/w0byO45zmni/i9lnhliD4UpkU
=ndmb
-----END PGP SIGNATURE-----

Re: Testing with concurrent sessions

Martijn van Oosterhout <kleptog@svana.org> — 2010-01-11T21:30:07Z

On Mon, Jan 11, 2010 at 05:42:08PM -0000, Greg Sabino Mullane wrote:
> > Is there a reason why you're suggesting using DBI? There is also the Pg
> > perl module which works as well and is one tenth of the size. It also
> > doesn't have external dependancies. It's just a plain wrapper around
> > libpq, which for the purposes of testing may be better.
> >
> > http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm
> 
> Works as well? Did you take a look at that link? The last update was
> early 2000, which should give you an indication of just how dead
> it is.

Dead or not, it still works, even against 8.4. I have many programs
that use it. It's simply a wrapper around the libpq interface and as
long as the libpq interface remains stable (which we go to great pains
to do), so will this module.

Given the talk of importing some perl module into the postgresql tree
it just seemed more logical to me to take something that was close to
libpq and had no external dependancies than taking a module with an
external dependancy (namely DBI).

Have a nice day,
-- 
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while 
> boarding. Thank you for flying nlogn airlines.

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-14T20:08:19Z

Hi,

Markus Wanner wrote:
> Sorry, if that didn't get clear. I'm trying to put together something I 
> can release real soon now (tm). I'll keep you informed.

Okay, here we go: dtester version 0.0.

This emerged out of Postgres-R, where I don't just need to test multiple 
client connections, but multiple postmasters interacting with each 
other. None the less, it may be suitable for other needs as well, 
especially testing with concurrent sessions.

I've decided to release this as a separate project named dtester, as 
proposed by Michael Tan (thanks for your inspiration).

It's certainly missing lots of things, mainly documentation. However, 
I've attached a patch which integrates nicely into the Postgres 
Makefiles, so you just need to say: make dcheck.

That very same patch includes a test case with three concurrent 
transactions with circular dependencies, where the current SERIALIZABLE 
isolation level fails to provide serializability.

Installing dtester itself is as simple as 'python setup.py install' in 
the extracted archive's directory.

Go try it, read the code and simply ask, if you get stuck. I'll try to 
come up with some more documentation and such...

Regards

Markus Wanner

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-14T20:23:01Z

Markus Wanner <markus@bluegap.ch> wrote:
 
> Okay, here we go: dtester version 0.0.
 
C'mon, you could have tried to inspire a *bit* more confidence by
calling it version 0.1 or something!  ;-)
 
> It's certainly missing lots of things, mainly documentation.
> However, I've attached a patch which integrates nicely into the
> Postgres Makefiles, so you just need to say: make dcheck.
 
That sounds very cool.
 
> That very same patch includes a test case with three concurrent 
> transactions with circular dependencies, where the current
> SERIALIZABLE isolation level fails to provide serializability.
 
Fantastic!  I'll expand that a bit....
 
> Installing dtester itself is as simple as 'python setup.py
> install' in the extracted archive's directory.
> 
> Go try it, read the code and simply ask, if you get stuck. I'll
> try to come up with some more documentation and such...
 
I'm reading through it all now.  Expect feedback soon!
 
And THANK YOU VERY MUCH!
 
-Kevin

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-15T05:53:05Z

Hi,

Kevin Grittner wrote:
> C'mon, you could have tried to inspire a *bit* more confidence by
> calling it version 0.1 or something!  ;-)

LOL

As David used to say: JFDI

 > I found that I just needed to ask for python-twisted.

Oh, sorry, yes, requirements: python, twisted.

I must admit that I haven't ever tested on python 2.6 before. I'll try 
that (especially as it's the staircase to 3.0, IIUC).

Two more things: the concurrent update test (in the patch part) is 
complete, while the second one is just a skeleton, ATM. (Just does a 
concurrent COMMIT without actually doing anything).

Second: at the very end of pg_dtester.py, you find the line:
  reporter = StreamReporter()

Try a CursesReporter() instead, it gives much nicer output!

Regards

Markus Wanner

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T15:07:18Z

Markus Wanner <markus@bluegap.ch> wrote:

> I must admit that I haven't ever tested on python 2.6 before. I'll
> try that (especially as it's the staircase to 3.0, IIUC).

I don't use python much, so I can't comment on that.  I do see that
my system has these two versions on it, with a symlink that makes
2.6 the default.

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2

Python 3.0.1+ (r301:69556, Apr 15 2009, 15:59:22)
[GCC 4.3.3] on linux2

I haven't quite gotten it to work yet; I'll start over with 3.0 and
see how it goes.  I'll also attach the results of the 2.6 attempt.

> Try a CursesReporter() instead, it gives much nicer output!

Thanks, I'll try it.

A few other issues in testing so far:

(1)  I see that a 'make dcheck' does a 'make install'.  That's not
right.  For one thing I usually install in a location where I need
to sudo to install; but more importantly, I want to do all checks
*before* I install.  It's easy enough to work around that for now,
but I don't think it's acceptable long-term.

(2)  After a 'make dcheck' failure, the cluster created for the
testing is left running.

(3)  If the install could check dependencies, report problems, and
refuse to install without required packages, that would be less
confusing for python novices (like me).

Perhaps some of these problems will go away with python 3.0, but I
figured I should pass the info along.

Thanks again for this.  It should help me a lot.

-Kevin

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T15:09:31Z
```
I wrote:
 
> I'll also attach the results of the 2.6 attempt.
 
Let's try that again.
 
-Kevin
```

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-15T15:22:04Z

Hi,

Quoting "Kevin Grittner" <Kevin.Grittner@wicourts.gov>:
> I haven't quite gotten it to work yet; I'll start over with 3.0 and
> see how it goes.

Let's stick to 2.x versions, first...

> I'll also attach the results of the 2.6 attempt.

Thanks, that looks already pretty promising. ;-)

> A few other issues in testing so far:
>
> (1)  I see that a 'make dcheck' does a 'make install'.  That's not
> right.  For one thing I usually install in a location where I need
> to sudo to install; but more importantly, I want to do all checks
> *before* I install.  It's easy enough to work around that for now,
> but I don't think it's acceptable long-term.

It does: "temp_install: creating temporary installation" means it's  
running make install in the background.

> (2)  After a 'make dcheck' failure, the cluster created for the
> testing is left running.

That counts as a bug. I also get that from time to time (and with  
Postgres-R testing on 3+ instances, it's even more annoying).

Note that the error just before that is, that a psql process it starts  
cannot connect to its postmaster ("startup of test test-conn-0A  
failed, skipping.") Please check the log  
(src/test/regress/dtester.log) for why that failed in the first place.  
Can you connect manually to the database (that's still running after a  
make dcheck)?

> (3)  If the install could check dependencies, report problems, and
> refuse to install without required packages, that would be less
> confusing for python novices (like me).

I'm not exactly a distutils hacker... Anybody else got any clue here?

> Perhaps some of these problems will go away with python 3.0, but I
> figured I should pass the info along.

I'd rather suspect that more of them will arise.

Regards

Markus

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T15:34:16Z

"Markus Wanner" <markus@bluegap.ch> wrote:
> Quoting "Kevin Grittner" <Kevin.Grittner@wicourts.gov>:
>> I haven't quite gotten it to work yet; I'll start over with 3.0
>> and see how it goes.
>
> Let's stick to 2.x versions, first...
 
OK
 
> It does: "temp_install: creating temporary installation" means
> it's running make install in the background.
 
OK, sorry for misreading that.
 
>> (2)  After a 'make dcheck' failure, the cluster created for the
>> testing is left running.
>
> That counts as a bug. I also get that from time to time (and with
> Postgres-R testing on 3+ instances, it's even more annoying).
>
> Note that the error just before that is, that a psql process it
> starts cannot connect to its postmaster ("startup of test
> test-conn-0A failed, skipping.") Please check the log
> (src/test/regress/dtester.log) for why that failed in the first
> place.
 
Not sure what's relevant there.  Entire file tarball attached.
 
> Can you connect manually to the database (that's still running
> after a make dcheck)?
 
Yes I can.  Any queries you'd like me to run in there?
 
-Kevin

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T15:47:28Z

Markus Wanner <markus@bluegap.ch> wrote:

> Second: at the very end of pg_dtester.py, you find the line:
>   reporter = StreamReporter()
> 
> Try a CursesReporter() instead, it gives much nicer output!

When I try to do that, Kate complains (I'm even copying their typo):

You are trying to save a python file as non ASCII, without
specifiying a correct source encoding line for encoding "utf-8"

It offers these options:

Insert: # -*- coding: utf-8 -*-
Save Nevertheless
Cancel

Should that coding line be in there?

-Kevin

Re: Testing with concurrent sessions

Greg Sabino Mullane <greg@turnstep.com> — 2010-01-15T16:33:16Z

-----BEGIN PGP SIGNED MESSAGE-----                             
Hash: RIPEMD160


> Dead or not, it still works, even against 8.4. I have many programs
> that use it. It's simply a wrapper around the libpq interface and as
> long as the libpq interface remains stable (which we go to great pains
> to do), so will this module.

Well, I stand corrected. Good to know.

> Given the talk of importing some perl module into the postgresql tree
> it just seemed more logical to me to take something that was close to
> libpq and had no external dependancies than taking a module with an
> external dependancy (namely DBI).

Yes, I could see that. Actually, I just came across another one
by Hiroyuki OYAMA and Aaron Crane. This was last updated January 10, 2010! :

http://search.cpan.org/~arc/DBD-PgPP-0.08/

Still requires DBI of course, but no Perl library or compiling required
as DBD::Pg does. So we've not got three valid options. :)

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 201001151129
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAktQmI0ACgkQvJuQZxSWSsgNugCgjwkT9QwGpvhcIXCNYhRcTwSW
JZcAnjvrsjwpO/QvJ1LzU+cUZ4UqajxV
=bu4q
-----END PGP SIGNATURE-----

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-15T18:20:35Z

Hi,

Kevin Grittner wrote:
> Not sure what's relevant there.  Entire file tarball attached.

Due to reasons mentioned in this thread as well, I've decided to use 
psql to connect to the database. dtester is parsing its output and 
checks that against expectations. Hawever, that has its own pitfalls, so 
in the end I'm almost about to change back to using libpq or 
implementing the bare minimum protocol (that might have its own merits 
within the twisted world, if implemented in the required async fashion).

Strangely, your log has escape codes in it, which I'm assuming makes the 
  parsing choke. Is that something special to your installation? My psql 
never colors its outputs...

However, the quickest way forward probably is to filter out escape 
sequences. Turning off tty is not really an option, because dtester 
doesn't have a chance to capture all necessary events, in that mode.

> Yes I can.  Any queries you'd like me to run in there?

It looks like psql can connect, too. It's just the parsing of outputs 
which fails.

Regards

Markus

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-15T18:23:25Z

Hi,

Kevin Grittner wrote:
> You are trying to save a python file as non ASCII, without
> specifiying a correct source encoding line for encoding "utf-8"

I wasn't aware I had non-ascii characters in there. Inserting an 
encoding line seems fine. I'll fix that for the upcoming version 0.1.

Regards

Markus

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T18:33:27Z

Markus Wanner <markus@bluegap.ch> wrote:

> I wasn't aware I had non-ascii characters in there. Inserting an 
> encoding line seems fine. I'll fix that for the upcoming version
> 0.1.

Yeah, I couldn't find any, either.  I just tried creating a minimal
python file in Kate, and it gave me that even though I *know* it was
all ASCII characters right off my keyboard.  I guess Kate is being
overly picky.  On the other hand, if it silences an annoying message
sometimes, maybe that's reason enough to add it.

-Kevin

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T18:36:10Z

Markus Wanner <markus@bluegap.ch> wrote:

> Strangely, your log has escape codes in it, which I'm assuming
> makes the parsing choke. Is that something special to your
> installation? My psql never colors its outputs...

I haven't configured anything like that intentionally.  I don't
*see* any colors when I use psql.  Can you think of anywhere I
should check something which might be causing this?

-Kevin

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-15T18:51:36Z

Hi,

Kevin Grittner wrote:
> I haven't configured anything like that intentionally.  I don't
> *see* any colors when I use psql.  Can you think of anywhere I
> should check something which might be causing this?

No idea ATM.

However, just to make sure that has absolutely nothing to do with the 
curses reporter I've written: is that dtester.log you just sent the log 
from a run with the StreamReporter or the CursesReporter? (Should not 
have any influence for the log, but you never know).

Please recheck with the StreamReporter and try to grep the lines 
starting with "[psql0]", "[psql1]" and "[psql2]". Dtester simply logs 
all and any output of all 3rd party processes started.

Alternatively, you may want to filter out all lines that start with 
"[postmaster0]", that might already reduce what we can consider noise in 
this case.

Regards

Markus Wanner

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T18:59:06Z

Markus Wanner <markus@bluegap.ch> wrote:

> Strangely, your log has escape codes in it, which I'm assuming
> makes the parsing choke. Is that something special to your
> installation?

My pager is "less"; could that cause it?  Could the twisted
environment look like one where the pager should kick in?

-Kevin

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-15T19:09:32Z

Hi,

Kevin Grittner wrote:
> My pager is "less"; could that cause it?  Could the twisted
> environment look like one where the pager should kick in?

Yes, that could be it. At least it fails here, too, if I set PAGER=less. 
Try:

   PAGER=more make dcheck

So, the solution probably lies in adjusting the environment, before 
starting psql. (Maybe even dropping all existing environment variables 
for better control of the situation). Will add that for dtester 0.1.

(Also note that I plan to move most of what's currently in the patch to 
the dtester package itself. However, that requires it to be (even more) 
generic.)

Thank you for testing the tester ;-)

Regards

Markus

Re: Testing with concurrent sessions

Andrew Dunstan <andrew@dunslane.net> — 2010-01-15T19:18:14Z


Markus Wanner wrote:
> Hi,
>
> Kevin Grittner wrote:
>> My pager is "less"; could that cause it?  Could the twisted
>> environment look like one where the pager should kick in?
>
> Yes, that could be it. At least it fails here, too, if I set 
> PAGER=less. Try:
>
>   PAGER=more make dcheck
>

Surely for automated use you want the psql pager off altogether. "psql 
--pset pager=off" or some such invocation should do it.

cheers

andrew

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T20:32:56Z

Markus Wanner <markus@bluegap.ch> wrote:

> So, the solution probably lies in adjusting the environment,
> before starting psql. (Maybe even dropping all existing
> environment variables for better control of the situation). Will
> add that for dtester 0.1.

Based on Andrew's suggestion, I changed line 276 to:

            args=['psql', '-A', '--pset=pager=off',

I now get 5 of 6 tests succeeded (83.3%), processed in 18.5 seconds.

I'm not clear on what you want to see from the run or whether it
might be better sent off-list.

Also, in looking closer at how you have the tests defined, it
doesn't look to me like you're carefully interleaving specific
sequences of statements on specific connections so much as opening
multiple connections and then for each statement saying "run this on
all connections."  That's certainly a valid test to include, but I
need the more controlled format, too.  It does appear that that's
pretty straightforward to code; you just haven't chosen to do so in
the particular tests here, correct?

-Kevin

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T20:53:44Z

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> wrote:

> Also, in looking closer at how you have the tests defined, it
> doesn't look to me like you're carefully interleaving specific
> sequences of statements on specific connections so much as opening
> multiple connections and then for each statement saying "run this
> on all connections."

I take it back; you've got both.

I do want to expand the tests quite a bit -- do I work them all into
this same file, or how would I proceed?  I think I'll need about 20
more tests, but I don't want to get in the way of your work on the
framework which runs them.

-Kevin

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T21:09:52Z

Markus Wanner <markus@bluegap.ch> wrote:

> Please recheck with the StreamReporter and try to grep the lines 
> starting with "[psql0]", "[psql1]" and "[psql2]". Dtester simply
> logs all and any output of all 3rd party processes started.

For me, all psql output seems to be [psql0]; no [psql1] or [psql2].

Bug?

-Kevin

Re: Testing with concurrent sessions

Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-01-15T22:39:12Z

Markus Wanner <markus@bluegap.ch> wrote:

> Go try it, read the code and simply ask, if you get stuck. I'll
> try to come up with some more documentation and such...

I'm a little unclear about the differences between "uses",
"depends", and "onlyAfter".  Here's what they *sound* like they
mean, to me; although I don't think the code isn't entirely
consistent with this interpretation.

"uses" means that the referenced task has complimentary setUp and
tearDown methods, and the dependent task may only run after a
successful invocation of the referenced task's setUp method, and the
referenced task will wait for completion of all dependent tasks
before invoking tearDown.

"depends" means that the tearDown method of the referenced task
doesn't undo the work of its setUp, at least for purposes of the
dependent task.  The dependent task can only start after successful
completion of the referenced class's work (*just* setUp, or all the
way to tearDown?), but the referenced task doesn't need to wait for
the dependent task.

"onlyAfter" means that the dependent task must wait for completion
of the referenced task, but doesn't care whether or not the
referenced class completed successfully.

How close am I?

-Kevin

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-16T06:23:34Z

Hi,

Kevin Grittner wrote:
> I'm a little unclear about the differences between "uses",
> "depends", and "onlyAfter".  Here's what they *sound* like they
> mean, to me; although I don't think the code isn't entirely
> consistent with this interpretation.

Wow, you are way ahead of me. I intended to write some documentation 
about that, but...

I differentiate tests and test suites. Tests mainly have a run method, 
while test suites have setUp and tearDown ones.

> "uses" means that the referenced task has complimentary setUp and
> tearDown methods, and the dependent task may only run after a
> successful invocation of the referenced task's setUp method, and the
> referenced task will wait for completion of all dependent tasks
> before invoking tearDown.

Absolutely correct (may I just copy that para for documentation)? ;-)

Two additional things: tests and test suites may have requirements (in 
the form of interfaces). The used test suites are passed to the 
dependent task and it may call the referenced tasks's methods, for 
example to get the database directory or to run a certain SQL command.

Second, if the referenced task fails, any running dependent task is 
getting aborted as well. That might be obvious, though.

> "depends" means that the tearDown method of the referenced task
> doesn't undo the work of its setUp, at least for purposes of the
> dependent task.  The dependent task can only start after successful
> completion of the referenced class's work (*just* setUp, or all the
> way to tearDown?), but the referenced task doesn't need to wait for
> the dependent task.

Hm.. no, not quite. The fact that not all suites clean up after them has 
nothing to do with how they are referenced ("uses" or "depends"). So 
far, it's entirely up to the test suite. I dislike that, but it works. 
(I've been thinking about some separate resource allocation handling and 
what not, but..)

The only difference between "depends" and "uses" is the requirements 
fulfilling. "uses" does that, while "depends" only adds the timing and 
functional dependencies, but doesn't pass the referenced task as an 
argument to the dependent task.

> "onlyAfter" means that the dependent task must wait for completion
> of the referenced task, but doesn't care whether or not the
> referenced class completed successfully.

That's how I think it *should* be. ATM "onlyAfter" requires successful 
completion of the dependent task.

I'd like to change that to support "onlyAfter", "onlyAfterSuccessOf" and 
"onlyAfterFailureOf". Plus "onlyBefore" for convenience.

This is all work in progress and I'm open to suggestions and requests.

Thank you for thinking through all of this. I'm sure you understand now, 
why it's not a version 0.1, yet :-)

Regards

Markus

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-16T06:51:00Z

Hi,

Kevin Grittner wrote:
> Based on Andrew's suggestion, I changed line 276 to:
>  
>             args=['psql', '-A', '--pset=pager=off',

That looks like a correct fix for psql, yes. Thanks for pointing that 
out Andrew.

Other processes might be confused by (or at least act differently with) 
a PAGER env variable, so that still needs to be cleared in general.

> I now get 5 of 6 tests succeeded (83.3%), processed in 18.5 seconds.

That's perfect. The one test that fails is expected to fail (another 
thing dtester doesn't support, yet). The serialization code you write 
should finally make that test pass ;-)

 > I do want to expand the tests quite a bit -- do I work them all into
 > this same file, or how would I proceed?  I think I'll need about 20
 > more tests, but I don't want to get in the way of your work on the
 > framework which runs them.

Well, first of all, another piece of the missing manual: there are 
BaseTest and SyncTest classes. Those based on BaseTest runs within the 
event loop of the twisted framework, thus need to be written in the very 
same asynchronous fashion. Mostly calling async methods that return a 
Deferred object, on which you may addCallback() or addErrback(). See the 
fine twisted documentation, especially the part about "Low-Level 
Networking and Event Loop" here:

http://twistedmatrix.com/documents/current/core/howto/index.html

The SyncTest is based on BaseTest, but a new thread is created to run 
its run method, passing back its results to the main event loop when 
done. That allows you to call blocking methods without having to care 
about blocking the entire event loop.

However, it makes interacting between the two models a bit complicated. 
To call an async function from a SyncTest, you need to call the syncCall 
method. The separate thread then waits for some callback in the main 
event loop.

Both have their own set of caveats, IMO.

I'm not sure about how to organize the tests and ongoing development of 
the framework. I've already broken the Postgres-R tests with dtester-0.0.

Maybe we put up a git branch with the dtester patches included? So 
whenever I want to change the framework, I can check if and how it 
affects your tests.

Regards

Markus

Re: Testing with concurrent sessions

Jan Urbański <wulczer@wulczer.org> — 2010-01-16T11:11:37Z

Markus Wanner wrote:
>> I do want to expand the tests quite a bit -- do I work them all into
>> this same file, or how would I proceed?  I think I'll need about 20
>> more tests, but I don't want to get in the way of your work on the
>> framework which runs them.
> 
> Well, first of all, another piece of the missing manual: there are
> BaseTest and SyncTest classes. Those based on BaseTest runs within the
> event loop of the twisted framework, thus need to be written in the very
> same asynchronous fashion. Mostly calling async methods that return a
> Deferred object, on which you may addCallback() or addErrback(). See the
> fine twisted documentation, especially the part about "Low-Level
> Networking and Event Loop" here:
> 
> http://twistedmatrix.com/documents/current/core/howto/index.html

> I'm not sure about how to organize the tests and ongoing development of
> the framework. I've already broken the Postgres-R tests with dtester-0.0.

Hi,

sorry to butt in to the conversation, but I have spent some time
wrapping/refining the concepts in dtester, and the results are here:

http://git.wulczer.org/?p=twisted-psql.git;a=summary

It reqires Twisted and has been tested on Python 2.5 (should work on
2.6, no idea about 3.0). The program you use to run it - trial - should
come with your distro's Twisted packages. The tests don't start a server
or anything, so you need to have a PG instance running. To try it:

git clone git://wulczer.org/twisted-psql.git
cd twisted-psql # this is important, or Python won't find the modules
$EDITOR config.py # set the path to psql and connection details for PG
trial test.test_serialization_error
trial test.test_true_serialization

Both tests should pass, the latter being marked as an expectedFailure.
You can then look at test/*.py to see my (puny) attempt at having some
abstraction layer over the asynchronocity of the tests.

I borrowed the idea of wrapping a psql in a Twisted protocol and added a
Deferred interface around it, which made it possible to run tests with
trial: the Twisted unit testing framework.

As a developer of a failry large Python system based on Twisted, that
sports hundreds of trial-based tests, I very strongly recommend trial
for asynchronous unit testing. It handles lots of boring details, is
well maintained and Twisted itself is simply designed to do asynchronous
programming. As an added bonus, the runnning and reporting
infrastructure is already there, you just write the tests.

My code is very rough and lacks good error reporting, for instance
failed tests will probably result in a "test hung" and the need to
Ctrl+C, but that can be easily improved. A thing that would help
tremendously would be a real Twisted protocol that talks to PG on the
protocol level, not by parsing psql output (which is very clumsy and
error prone IMHO).

I found one such project:
http://www.jamwt.com/pgasync/
but it had some issues with committing (all my test programs were
exiting before PG got the final COMMIT, which resulted in the
impossibility to do anything) and it does too much things that Python PG
drivers like to do (like declaring a CURSOR for each query, bleah). A
good implementation would hugely improve the quality and robustness of
any such testsuite.

Cheers,
Jan
-- 
Jan Urbanski
GPG key ID: E583D7D2

ouden estin

Re: Testing with concurrent sessions

Markus Wanner <markus@bluegap.ch> — 2010-01-19T21:34:40Z

Hi,

Jan Urbański wrote:
> sorry to butt in to the conversation, but I have spent some time
> wrapping/refining the concepts in dtester, and the results are here:
> 
> http://git.wulczer.org/?p=twisted-psql.git;a=summary

That seems to cover the concurrent psql part of dtester. But I don't see
how that's wrapping or refining dtester.

> I borrowed the idea of wrapping a psql in a Twisted protocol and added a
> Deferred interface around it, which made it possible to run tests with
> trial: the Twisted unit testing framework.

dtester works pretty much the same way, except that it doesn't use
trial. But the Deferred interface is about the same.

> As a developer of a failry large Python system based on Twisted, that
> sports hundreds of trial-based tests, I very strongly recommend trial
> for asynchronous unit testing. It handles lots of boring details, is
> well maintained and Twisted itself is simply designed to do asynchronous
> programming. As an added bonus, the runnning and reporting
> infrastructure is already there, you just write the tests.

I'm using trial for two other projects as well and I've started with it
for Postgres. IMO trial is very good for unit tests, but fails horribly
for anything that involves complex setups and watch-guarding of multiple
processes. That's where dtester shines.

> My code is very rough and lacks good error reporting, for instance
> failed tests will probably result in a "test hung" and the need to
> Ctrl+C, but that can be easily improved.

Don't underestimate that. There are lots of sources of errors. Not
having a sentinel over the postmaster itself, unit tests from trial
simply fail to notice errors there. (And coding for Postgres, these are
the ones you are interested in). Now combine all of that and error
handling and (useful) reporting is *the* major challenge in automating
testing.

> A thing that would help
> tremendously would be a real Twisted protocol that talks to PG on the
> protocol level, not by parsing psql output (which is very clumsy and
> error prone IMHO).

I agree. (Well, now I do).

> I found one such project:
> http://www.jamwt.com/pgasync/
> but it had some issues with committing

Yeah, I didn't like that one, either.

Regards

Markus