TAP output format in pg_regress

Daniel Gustafsson <daniel@yesql.se>

From: Daniel Gustafsson <daniel@yesql.se>
To: PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>, Andres Freund <andres@anarazel.de>
Date: 2022-02-21T23:08:11Z
Lists: pgsql-hackers

Attachments

Starting a new thread on the TAP patch from the "[RFC] building postgres with
meson" thread at 20220221165228.aqnfg45mceym7njm@alap3.anarazel.de to have
somewhere to discuss this patch.

> On 21 Feb 2022, at 17:52, Andres Freund <andres@anarazel.de> wrote:
> On 2021-10-13 13:54:10 +0200, Daniel Gustafsson wrote:

>> I added a --tap option for TAP output to pg_regress together with Jinbao Chen
>> for giggles and killing some time a while back.
> 
> Sorry for not replying to this earlier. I somehow thought I had, but the
> archives disagree.

No worries, I had forgotten about it myself.

> I think this would be great.

Cool, I'll pick it up again then.  I didn't have time to dig into the patch
tonight but the attached is a rebased version which just cleans up the bitrot
it had accumulated to the point where it at least compiles and seems to run.

>> One thing that came out of this, is that we don't really handle the ignored
>> tests in the way the code thinks it does for normal output, the attached treats
>> ignored tests as SKIP tests.
> 
> I can't really parse the first sentence...

I admittedly don't remember exactly what I meant, but I'm fairly sure it's
wrong.  I think I thought ignored tests were counted incorrectly, but skimming
the patch just now I think it's doing it wrong as it counts ignored as failed
even if they passed.  I'll fix that.

>> 	if (exit_status != 0)
>> 		log_child_failure(exit_status);
>> @@ -2152,6 +2413,7 @@ regression_main(int argc, char *argv[],
>> 		{"config-auth", required_argument, NULL, 24},
>> 		{"max-concurrent-tests", required_argument, NULL, 25},
>> 		{"make-testtablespace-dir", no_argument, NULL, 26},
>> +		{"tap", no_argument, NULL, 27},
>> 		{NULL, 0, NULL, 0}
>> 	};
> 
> I'd make it a --format=(regress|tap) or such.

That makes sense, done in the attached.

--
Daniel Gustafsson		https://vmware.com/

Commits

  1. pg_regress: Emit TAP compliant output

  2. Consider a failed process as a failed test in pg_regress

  3. Avoid reference to nonexistent array element in ExecInitAgg().