Re: benchmark results comparing versions 15.2 and 16

Hi,

On 2023-05-15 14:20:24 +0900, Michael Paquier wrote:
> On Thu, May 11, 2023 at 01:28:40PM +0900, Michael Paquier wrote:
> > On Tue, May 09, 2023 at 09:48:24AM -0700, Andres Freund wrote:
> >> On 2023-05-08 12:11:17 -0700, Andres Freund wrote:
> >>> I can reproduce a significant regression due to f193883fc of a workload just
> >>> running
> >>> SELECT CURRENT_TIMESTAMP;
> >>> 
> >>> A single session running it on my workstation via pgbench -Mprepared gets
> >>> before:
> >>> tps = 89359.128359 (without initial connection time)
> >>> after:
> >>> tps = 83843.585152 (without initial connection time)
> >>> 
> >>> Obviously this is an extreme workload, but that nevertheless seems too large
> >>> to just accept...
> 
> Extreme is adapted for a worst-case scenario.  Looking at my notes
> from a few months back, that's kind of what I did on my laptop, which
> was the only machine I had at hand back then:
> - Compilation of code with -O2.

I assume without assertions as well?


> I have re-run a bit more pgbench (1 client, prepared query with a
> single SELECT on a SQL keyword, etc.).  And, TBH, I am not seeing as
> much difference as you do (nothing with default pgbench setup, FWIW),
> still that's able to show a bit more difference than the other two
> cases.

> HEAD shows me an average output close to 43900 TPS (3 run of
> 60s each, for instance), while relying on SQLValueFunction shows an
> average of 45000TPS.  That counts for ~2.4% output regression here
> on bigbox based on these numbers.  Not a regression as high as
> mentioned above, still that's visible.

45k seems too low for a modern machine, given that I get > 80k in such a
workload, on a workstation with server CPUs (i.e. many cores, but not that
fast individually).  Hence wondering about assertions being enabled...

I get quite variable performance if I don't pin client / server to the same
core, but even the slow performance is faster than 45k.

Greetings,

Andres Freund