Horton

Thread

Commits

  1. oauth_validator: Shorten JSON responses in test logs

  2. pg_dump tests: don't put dumps in stdout

  1. reduce size of logs stored by buildfarm

    Álvaro Herrera <alvherre@kurilemu.de> — 2025-11-25T13:30:33Z

    Hi,

I just noticed that one regress log file for the pg_dump test is about
7MB long because it contains totally unnecessary dump files in the
regress log output.  This is because the tests run pg_dump of schema
pg_catalog without redirecting the output anywhere, which means it goes
to stdout, and then `prove` helpfully puts it in the regress log file.
This is completely pointless.  We can make the log file 1 MB instead by
adding a --file parameter to that command, as 0001 attached.


We also have a couple of tests that leave rather large server logs
behind that are perhaps not very useful.  I would propose to run those
servers with log_statement=off and avoid some of the bloat.  0002 does
that.

Another possible approach, instead of individually tweaking t/*.pl files
to add 'log_statements=none', would be to set it that way in
PostgreSQL::Test::Cluster globally.  Right now we have
log_statements=all there, but maybe that was not a great idea.


I also noticed that an oauth test file contains a couple MBs of a
gigantic string of just 'x'.  I suppose that will compress well (with
gzip at least, the 2 MB file becomes 8 kB).  Still, it's kinda
ridiculous and useless to bloat a 67 kB file to 2 MB that way.  The
problem here is that we print 'sending JSON response' and the verbatim
response, included the 1 MB of "x" as '_pad_'.  Can we supress this?
0003 does that by simply cutting the string short at 10k, which reduces
the size of the log from 2 MB to some 86 kB.  Maybe there are better
ways to deal with this?  Jacob?


With these three patches, we go from having 62 MB bytes in files with
"log" in their names under testrun/ (except those that have "catalog")
to 30 MB.  I think that's not a bad reduction.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
 Are you not unsure you want to delete Firefox?
       [Not unsure]     [Not not unsure]    [Cancel]
                   http://smylers.hates-software.com/2008/01/03/566e45b2.html

  2. Re: reduce size of logs stored by buildfarm

    Andres Freund <andres@anarazel.de> — 2025-11-25T15:03:37Z

    Hi,

On 2025-11-25 14:30:33 +0100, Álvaro Herrera wrote:
> I just noticed that one regress log file for the pg_dump test is about
> 7MB long because it contains totally unnecessary dump files in the
> regress log output.  This is because the tests run pg_dump of schema
> pg_catalog without redirecting the output anywhere, which means it goes
> to stdout, and then `prove` helpfully puts it in the regress log file.
> This is completely pointless.  We can make the log file 1 MB instead by
> adding a --file parameter to that command, as 0001 attached.

Yea, that makes no sense.


> We also have a couple of tests that leave rather large server logs
> behind that are perhaps not very useful.  I would propose to run those
> servers with log_statement=off and avoid some of the bloat.  0002 does
> that.
> 
> Another possible approach, instead of individually tweaking t/*.pl files
> to add 'log_statements=none', would be to set it that way in
> PostgreSQL::Test::Cluster globally.  Right now we have
> log_statements=all there, but maybe that was not a great idea.

I think that be a bad idea - many tests are essentially undebuggable that way
because there's no other way to correlate the regress_log and the server log
than the statement log.

TBH, I have even needed the pg_dump log_statement output on the buildfarm to
debug issues :(

I am against doing this for 027_stream_regress.pl, it's definitely
undebuggable without the statement logs.

Greetings,

Andres Freund

  3. Re: reduce size of logs stored by buildfarm

    Tom Lane <tgl@sss.pgh.pa.us> — 2025-11-25T15:37:07Z

    =?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes:
> With these three patches, we go from having 62 MB bytes in files with
> "log" in their names under testrun/ (except those that have "catalog")
> to 30 MB.  I think that's not a bad reduction.

I wonder how much this overlaps with what Andrew just did
to the BF client [1].

			regards, tom lane

[1] https://www.postgresql.org/message-id/601bb91d-b55d-4fa6-bb57-c2126fb22620%40dunslane.net

  4. Re: reduce size of logs stored by buildfarm

    Andrew Dunstan <andrew@dunslane.net> — 2025-11-25T17:15:20Z

    On 2025-11-25 Tu 10:37 AM, Tom Lane wrote:
> =?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes:
>> With these three patches, we go from having 62 MB bytes in files with
>> "log" in their names under testrun/ (except those that have "catalog")
>> to 30 MB.  I think that's not a bad reduction.
> I wonder how much this overlaps with what Andrew just did
> to the BF client [1].
>
>
> [1] https://www.postgresql.org/message-id/601bb91d-b55d-4fa6-bb57-c2126fb22620%40dunslane.net
>
>

Pretty much none of it, although both Alvaro and I are clearly motivated 
by the same thing, namely disk space pressure on the buildfarm server.

I think his patches 1 and 3 seem like no-brainers.

I did two things in the BF client:

. prevent uploading redundant copies of the same file in meson builds 
(this was a bug)

. inhibit loading postmaster log files from installcheck runs and 
cross-version upgrade runs, but only if there hasn't been a failure.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  5. Re: reduce size of logs stored by buildfarm

    Álvaro Herrera <alvherre@kurilemu.de> — 2025-11-25T18:14:56Z

    On 2025-Nov-25, Andrew Dunstan wrote:

> I think his patches 1 and 3 seem like no-brainers.


On 2025-Nov-25, Andres Freund wrote:

> On 2025-11-25 14:30:33 +0100, Álvaro Herrera wrote:
> > I just noticed that one regress log file for the pg_dump test is about
> > 7MB long because it contains totally unnecessary dump files in the
> > regress log output.  This is because the tests run pg_dump of schema
> > pg_catalog without redirecting the output anywhere, which means it goes
> > to stdout, and then `prove` helpfully puts it in the regress log file.
> > This is completely pointless.  We can make the log file 1 MB instead by
> > adding a --file parameter to that command, as 0001 attached.
> 
> Yea, that makes no sense.

Thanks, pushed 0001.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"Postgres is bloatware by design: it was built to house
 PhD theses." (Joey Hellerstein, SIGMOD annual conference 2002)

  6. Re: reduce size of logs stored by buildfarm

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-11-25T18:39:29Z

    On Tue, Nov 25, 2025 at 5:30 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:
> I also noticed that an oauth test file contains a couple MBs of a
> gigantic string of just 'x'.  I suppose that will compress well (with
> gzip at least, the 2 MB file becomes 8 kB).  Still, it's kinda
> ridiculous and useless to bloat a 67 kB file to 2 MB that way.

Sorry about that.

> 0003 does that by simply cutting the string short at 10k, which reduces
> the size of the log from 2 MB to some 86 kB.  Maybe there are better
> ways to deal with this?  Jacob?

Only thing I don't like about this is that the JSON you need for
debugging might be after the useless padding. Attached is a patch that
does things more surgically, tested against Python 3.6, and I'm
running it through the CI now.

Thanks,
--Jacob

  7. Re: reduce size of logs stored by buildfarm

    Álvaro Herrera <alvherre@kurilemu.de> — 2025-11-25T18:46:03Z

    On 2025-Nov-25, Jacob Champion wrote:

> On Tue, Nov 25, 2025 at 5:30 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:
> > I also noticed that an oauth test file contains a couple MBs of a
> > gigantic string of just 'x'.  I suppose that will compress well (with
> > gzip at least, the 2 MB file becomes 8 kB).  Still, it's kinda
> > ridiculous and useless to bloat a 67 kB file to 2 MB that way.
> 
> Sorry about that.

No need!

> > 0003 does that by simply cutting the string short at 10k, which reduces
> > the size of the log from 2 MB to some 86 kB.  Maybe there are better
> > ways to deal with this?  Jacob?
> 
> Only thing I don't like about this is that the JSON you need for
> debugging might be after the useless padding. Attached is a patch that
> does things more surgically, tested against Python 3.6, and I'm
> running it through the CI now.

Brilliant, thanks.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"You're _really_ hosed if the person doing the hiring doesn't understand
relational systems: you end up with a whole raft of programmers, none of
whom has had a Date with the clue stick."              (Andrew Sullivan)
https://postgr.es/m/20050809113420.GD2768@phlogiston.dyndns.org

  8. Re: reduce size of logs stored by buildfarm

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-12-01T17:01:36Z

    On Tue, Nov 25, 2025 at 10:46 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:
> > Only thing I don't like about this is that the JSON you need for
> > debugging might be after the useless padding. Attached is a patch that
> > does things more surgically, tested against Python 3.6, and I'm
> > running it through the CI now.
>
> Brilliant, thanks.

(This was pushed last week; forgot to follow up here.)

Thanks,
--Jacob