Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

pg_test_timing: Show additional TSC clock source debug info
- 5ba34f6dc838 19 (unreleased) landed
instrumentation: Avoid CPUID 0x15/0x16 for Hypervisor TSC frequency
- 7fc36c5db550 19 (unreleased) landed
pg_test_timing: Also test RDTSC[P] timing, report time source, TSC frequency
- 16fca4825483 19 (unreleased) landed
Allow retrieving x86 TSC frequency/flags from CPUID
- bcb2cf41f964 19 (unreleased) landed
instrumentation: Standardize ticks to nanosecond conversion method
- 0022622c93d9 19 (unreleased) landed
instrumentation: Use Time-Stamp Counter on x86-64 to lower overhead
- 294520c44487 19 (unreleased) landed
Check for __cpuidex and __get_cpuid_count separately
- effaa464afd3 19 (unreleased) landed
pg_test_timing: Reduce per-loop overhead
- 82c0cb4e672d 19 (unreleased) landed
Refactor handling of x86 CPUID instructions
- be6a7494d2e3 19 (unreleased) landed
instrumentation: Drop INSTR_TIME_SET_CURRENT_LAZY macro
- 9d6294c09ed0 19 (unreleased) landed
Rename pg_crc32c_sse42_choose.c for general purpose
- b9278871f991 19 (unreleased) cited
Zero initialize uses of instr_time about to trigger compiler warnings
- 25b2aba0c3a5 16.0 landed
instr_time: Represent time as an int64 on all platforms
- 03023a2664f8 16.0 landed
Add 250c8ee07ed to git-blame-ignore-revs
- ff23b592ad66 16.0 cited

Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2020-06-12T23:28:10Z

Hi,

Currently using EXPLAIN (ANALYZE) without TIMING OFF regularly changes
the resulting timing enough that the times aren't meaningful. E.g.

CREATE TABLE lotsarows(key int not null);
INSERT INTO lotsarows SELECT generate_series(1, 50000000);
VACUUM FREEZE lotsarows;


-- best of three:
SELECT count(*) FROM lotsarows;
Time: 1923.394 ms (00:01.923)

-- best of three:
EXPLAIN (ANALYZE, TIMING OFF) SELECT count(*) FROM lotsarows;
Time: 2319.830 ms (00:02.320)

-- best of three:
EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 4202.649 ms (00:04.203)

That nearly *double* the execution time without TIMING.


Looking at a profile of this shows that we spend a good bit of cycles
"normalizing" timstamps etc. That seems pretty unnecessary, just forced
on us due to struct timespec. So the first attached patch just turns
instr_time to be a 64bit integer, counting nanoseconds.

That helps, a tiny bit:
EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 4179.302 ms (00:04.179)

but obviously doesn't move the needle.


Looking at a profile it's easy to confirm that we spend a lot of time
acquiring time:
-   95.49%     0.00%  postgres     postgres                 [.] agg_retrieve_direct (inlined)
   - agg_retrieve_direct (inlined)
      - 79.27% fetch_input_tuple
         - ExecProcNode (inlined)
            - 75.72% ExecProcNodeInstr
               + 25.22% SeqNext
               - 21.74% InstrStopNode
                  + 17.80% __GI___clock_gettime (inlined)
               - 21.44% InstrStartNode
                  + 19.23% __GI___clock_gettime (inlined)
               + 4.06% ExecScan
      + 13.09% advance_aggregates (inlined)
        1.06% MemoryContextReset

And that's even though linux avoids a syscall (in most cases) etc to
acquire the time. Unless the kernel detects there's a reason not to do
so, linux does this by using 'rdtscp' and multiplying it by kernel
provided factors to turn the cycles into time.

Some of the time is spent doing function calls, dividing into struct
timespec, etc. But most of it just the rdtscp instruction:
 65.30 │1  63:   rdtscp


The reason for that is largely that rdtscp waits until all prior
instructions have finished (but it allows later instructions to already
start). Multiple times for each tuple.


In the second attached prototype patch I've change instr_time to count
in cpu cycles instead of nanoseconds. And then just turned the cycles
into seconds in INSTR_TIME_GET_DOUBLE() (more about that part later).

When using rdtsc that results in *vastly* lower overhead:
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                          QUERY PLAN                                                           │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Aggregate  (cost=846239.20..846239.21 rows=1 width=8) (actual time=2610.235..2610.235 rows=1 loops=1)                         │
│   ->  Seq Scan on lotsarows  (cost=0.00..721239.16 rows=50000016 width=0) (actual time=0.006..1512.886 rows=50000000 loops=1) │
│ Planning Time: 0.028 ms                                                                                                       │
│ Execution Time: 2610.256 ms                                                                                                   │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(4 rows)

Time: 2610.589 ms (00:02.611)

And there's still some smaller improvements that could be made ontop of
that.

As a comparison, here's the time when using rdtscp directly in
instr_time, instead of going through clock_gettime:
Time: 3481.162 ms (00:03.481)

That shows pretty well how big the cost of the added pipeline stalls
are, and how important out-of-order execution is for decent
performance...


In my opinion, for the use in InstrStartNode(), InstrStopNode() etc, we
do *not* want to wait for prior instructions to finish, since that
actually leads to the timing being less accurate, rather than
more. There are other cases where that'd be different, e.g. measuring
how long an entire query takes or such (but there it's probably
irrelevant which to use).


I've above skipped a bit over the details of how to turn the cycles
returned by rdtsc into time:

On x86 CPUs of the last ~12 years rdtsc doesn't return the cycles that
have actually been run, but instead returns the number of 'reference
cycles'. That's important because otherwise things like turbo mode and
lower power modes would lead to completely bogus times.

Thus, knowing the "base frequency" of the CPU allows to turn the
difference between two rdtsc return values into seconds.

In the attached prototype I just determined the number of cycles using
cpuid(0x16). That's only available since Skylake (I think). On older
CPUs we'd have to look at /proc/cpuinfo or
/sys/devices/system/cpu/cpu0/cpufreq/base_frequency.


There's also other issues with using rdtsc directly: On older CPUs, in
particular older multi-socket systems, the tsc will not be synchronized
in detail across cores. There's bits that'd let us check whether tsc is
suitable or not.  The more current issue of that is that things like
virtual machines being migrated can lead to rdtsc suddenly returning a
different value / the frequency differening. But that is supposed to be
solved these days, by having virtualization technologies set frequency
multipliers and offsets which then cause rdtsc[p] to return something
meaningful, even after migration.


The attached patches are really just a prototype. I'm also not really
planning to work on getting this into a "production ready" patchset
anytime soon. I developed it primarily because I found it the overhead
made it too hard to nail down in which part of a query tree performance
changed.  If somebody else wants to continue from here...

I do think it's be a pretty significant improvement if we could reduce
the timing overhead of EXPLAIN ANALYZE by this much. Even if requires a
bunch of low-level code.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Pavel Stehule <pavel.stehule@gmail.com> — 2020-06-13T03:53:01Z

so 13. 6. 2020 v 1:28 odesílatel Andres Freund <andres@anarazel.de> napsal:

> Hi,
>
> Currently using EXPLAIN (ANALYZE) without TIMING OFF regularly changes
> the resulting timing enough that the times aren't meaningful. E.g.
>
> CREATE TABLE lotsarows(key int not null);
> INSERT INTO lotsarows SELECT generate_series(1, 50000000);
> VACUUM FREEZE lotsarows;
>
>
> -- best of three:
> SELECT count(*) FROM lotsarows;
> Time: 1923.394 ms (00:01.923)
>
> -- best of three:
> EXPLAIN (ANALYZE, TIMING OFF) SELECT count(*) FROM lotsarows;
> Time: 2319.830 ms (00:02.320)
>
> -- best of three:
> EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 4202.649 ms (00:04.203)
>
> That nearly *double* the execution time without TIMING.
>
>
> Looking at a profile of this shows that we spend a good bit of cycles
> "normalizing" timstamps etc. That seems pretty unnecessary, just forced
> on us due to struct timespec. So the first attached patch just turns
> instr_time to be a 64bit integer, counting nanoseconds.
>
> That helps, a tiny bit:
> EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 4179.302 ms (00:04.179)
>
> but obviously doesn't move the needle.
>
>
> Looking at a profile it's easy to confirm that we spend a lot of time
> acquiring time:
> -   95.49%     0.00%  postgres     postgres                 [.]
> agg_retrieve_direct (inlined)
>    - agg_retrieve_direct (inlined)
>       - 79.27% fetch_input_tuple
>          - ExecProcNode (inlined)
>             - 75.72% ExecProcNodeInstr
>                + 25.22% SeqNext
>                - 21.74% InstrStopNode
>                   + 17.80% __GI___clock_gettime (inlined)
>                - 21.44% InstrStartNode
>                   + 19.23% __GI___clock_gettime (inlined)
>                + 4.06% ExecScan
>       + 13.09% advance_aggregates (inlined)
>         1.06% MemoryContextReset
>
> And that's even though linux avoids a syscall (in most cases) etc to
> acquire the time. Unless the kernel detects there's a reason not to do
> so, linux does this by using 'rdtscp' and multiplying it by kernel
> provided factors to turn the cycles into time.
>
> Some of the time is spent doing function calls, dividing into struct
> timespec, etc. But most of it just the rdtscp instruction:
>  65.30 │1  63:   rdtscp
>
>
> The reason for that is largely that rdtscp waits until all prior
> instructions have finished (but it allows later instructions to already
> start). Multiple times for each tuple.
>
>
> In the second attached prototype patch I've change instr_time to count
> in cpu cycles instead of nanoseconds. And then just turned the cycles
> into seconds in INSTR_TIME_GET_DOUBLE() (more about that part later).
>
> When using rdtsc that results in *vastly* lower overhead:
>
> ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
> │                                                          QUERY PLAN
>                                                      │
>
> ├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
> │ Aggregate  (cost=846239.20..846239.21 rows=1 width=8) (actual
> time=2610.235..2610.235 rows=1 loops=1)                         │
> │   ->  Seq Scan on lotsarows  (cost=0.00..721239.16 rows=50000016
> width=0) (actual time=0.006..1512.886 rows=50000000 loops=1) │
> │ Planning Time: 0.028 ms
>                                                      │
> │ Execution Time: 2610.256 ms
>                                                      │
>
> └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
> (4 rows)
>
> Time: 2610.589 ms (00:02.611)
>
> And there's still some smaller improvements that could be made ontop of
> that.
>
> As a comparison, here's the time when using rdtscp directly in
> instr_time, instead of going through clock_gettime:
> Time: 3481.162 ms (00:03.481)
>
> That shows pretty well how big the cost of the added pipeline stalls
> are, and how important out-of-order execution is for decent
> performance...
>
>
> In my opinion, for the use in InstrStartNode(), InstrStopNode() etc, we
> do *not* want to wait for prior instructions to finish, since that
> actually leads to the timing being less accurate, rather than
> more. There are other cases where that'd be different, e.g. measuring
> how long an entire query takes or such (but there it's probably
> irrelevant which to use).
>
>
> I've above skipped a bit over the details of how to turn the cycles
> returned by rdtsc into time:
>
> On x86 CPUs of the last ~12 years rdtsc doesn't return the cycles that
> have actually been run, but instead returns the number of 'reference
> cycles'. That's important because otherwise things like turbo mode and
> lower power modes would lead to completely bogus times.
>
> Thus, knowing the "base frequency" of the CPU allows to turn the
> difference between two rdtsc return values into seconds.
>
> In the attached prototype I just determined the number of cycles using
> cpuid(0x16). That's only available since Skylake (I think). On older
> CPUs we'd have to look at /proc/cpuinfo or
> /sys/devices/system/cpu/cpu0/cpufreq/base_frequency.
>
>
> There's also other issues with using rdtsc directly: On older CPUs, in
> particular older multi-socket systems, the tsc will not be synchronized
> in detail across cores. There's bits that'd let us check whether tsc is
> suitable or not.  The more current issue of that is that things like
> virtual machines being migrated can lead to rdtsc suddenly returning a
> different value / the frequency differening. But that is supposed to be
> solved these days, by having virtualization technologies set frequency
> multipliers and offsets which then cause rdtsc[p] to return something
> meaningful, even after migration.
>
>
> The attached patches are really just a prototype. I'm also not really
> planning to work on getting this into a "production ready" patchset
> anytime soon. I developed it primarily because I found it the overhead
> made it too hard to nail down in which part of a query tree performance
> changed.  If somebody else wants to continue from here...
>
> I do think it's be a pretty significant improvement if we could reduce
> the timing overhead of EXPLAIN ANALYZE by this much. Even if requires a
> bunch of low-level code.
>

+1

Pavel


> Greetings,
>
> Andres Freund
>

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Thomas Munro <thomas.munro@gmail.com> — 2021-03-13T00:34:16Z

On Sat, Jun 13, 2020 at 11:28 AM Andres Freund <andres@anarazel.de> wrote:
> [PATCH v1 1/2] WIP: Change instr_time to just store nanoseconds, that's cheaper.

Makes a lot of sense.  If we do this, I'll need to update pgbench,
which just did something similar locally.  If I'd been paying
attention to this thread I might not have committed that piece of the
recent pgbench changes, but it's trivial stuff and I'll be happy to
tidy that up when the time comes.

> [PATCH v1 2/2] WIP: Use cpu reference cycles, via rdtsc, to measure time for instrumentation.

> Some of the time is spent doing function calls, dividing into struct
> timespec, etc. But most of it just the rdtscp instruction:
>  65.30 │1  63:   rdtscp

> The reason for that is largely that rdtscp waits until all prior
> instructions have finished (but it allows later instructions to already
> start). Multiple times for each tuple.

Yeah, after reading a bit about this, I agree that there is no reason
to think that the stalling version makes the answer better in any way.
It might make sense if you use it once at the beginning of a large
computation, but it makes no sense if you sprinkle it around inside
blocks that will run multiple times.  It destroys your
instructions-per-cycle while, turning your fancy super scalar Pentium
into a 486.  It does raise some interesting questions about what
exactly you're measuring, though: I don't know enough to have a good
grip on how far out of order the TSC could be read!

> There's also other issues with using rdtsc directly: On older CPUs, in
> particular older multi-socket systems, the tsc will not be synchronized
> in detail across cores. There's bits that'd let us check whether tsc is
> suitable or not.  The more current issue of that is that things like
> virtual machines being migrated can lead to rdtsc suddenly returning a
> different value / the frequency differening. But that is supposed to be
> solved these days, by having virtualization technologies set frequency
> multipliers and offsets which then cause rdtsc[p] to return something
> meaningful, even after migration.

Googling tells me that Nehalem (2008) introduced "invariant TSC"
(clock rate independent) and also socket synchronisation at the same
time, so systems without it are already pretty long in the tooth.

A quick peek at an AMD manual[1] tells me that a similar change
happened in 15H/Bulldozer/Piledriver/Steamroller/Excavator (2011),
identified with the same CPUID test.

My first reaction is that it seems like TSC would be the least of your
worries if you're measuring a VM that's currently migrating between
hosts, but maybe the idea is just that you have to make sure you don't
assume it can't ever go backwards or something like that...

Google Benchmark has some clues about how to spell this on MSVC, what
some instructions might be to research on ARM, etc.

[1] https://www.amd.com/system/files/TechDocs/47414_15h_sw_opt_guide.pdf
(page 373)
[2] https://github.com/google/benchmark/blob/master/src/cycleclock.h

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2022-07-01T08:23:01Z

On Fri, Jun 12, 2020 at 4:28 PM Andres Freund <andres@anarazel.de> wrote:

> The attached patches are really just a prototype. I'm also not really
> planning to work on getting this into a "production ready" patchset
> anytime soon. I developed it primarily because I found it the overhead
> made it too hard to nail down in which part of a query tree performance
> changed.  If somebody else wants to continue from here...
>
> I do think it's be a pretty significant improvement if we could reduce
> the timing overhead of EXPLAIN ANALYZE by this much. Even if requires a
> bunch of low-level code.
>

Based on an off-list conversation with Andres, I decided to dust off this
old
patch for using rdtsc directly. The significant EXPLAIN ANALYZE performance
improvements (especially when using rdtsc instead of rdtsc*p*) seem to
warrant
giving this a more thorough look.

See attached an updated patch (adding it to the July commitfest), with a few
changes:

- Keep using clock_gettime() as a fallback if we decide to not use rdtsc
- Fallback to /proc/cpuinfo for clock frequency, if cpuid(0x16) doesn't work
- The decision to use rdtsc (or not) is made at runtime, in the new
  INSTR_TIME_INITIALIZE() -- we can't make this decision at compile time
  because this is dependent on the specific CPU in use, amongst other things
- In an abundance of caution, for now I've decided to only enable this if we
  are on Linux/x86, and the current kernel clocksource is TSC (the kernel
has
  quite sophisticated logic around making this decision, see [1])

Note that if we implemented the decision logic ourselves (instead of relying
on the Linux kernel), I'd be most worried about older virtualization
technology. In my understanding getting this right is notably more
complicated
than just checking cpuid, see [2].

Known WIP problems with this patch version:

* There appears to be a timing discrepancy I haven't yet worked out, where
  the \timing data reported by psql doesn't match what EXPLAIN ANALYZE is
  reporting. With Andres' earlier test case, I'm seeing a consistent ~700ms
  higher for \timing than for the EXPLAIN ANALYZE time reported on the
server
  side, only when rdtsc measurement is used -- its likely there is a problem
  somewhere with how we perform the cycles to time conversion
* Possibly related, the floating point handling for the cycles_to_sec
variable
  is problematic in terms of precision (see FIXME, taken over from Andres'
POC)

Open questions from me:

1) Do we need to account for different TSC offsets on different CPUs in SMP
   systems? (the Linux kernel certainly has logic to that extent, but [3]
   suggests this is no longer a problem on Nehalem and newer chips, i.e.
those
   having an invariant TSC)

2) Should we have a setting "--with-tsc" for configure? (instead of always
   enabling it when on Linux/x86 with a TSC clocksource)

3) Are there cases where we actually want to use rdtsc*p*? (i.e. wait for
   current instructions to finish -- the prior discussion seemed to suggest
   we don't want it for node instruction measurements, but possibly we do
want
   this in other cases?)

4) Should we support using the "mrs" instruction on ARM? (which is similar
to
   rdtsc, see [4])

Thanks,
Lukas

[1] https://github.com/torvalds/linux/blob/master/arch/x86/kernel/tsc.c
[2] http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/
[3] https://stackoverflow.com/a/11060619/1652607
[4] https://cpufun.substack.com/p/fun-with-timers-and-cpuid

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2022-07-01T17:26:39Z

Hi,

On 2022-07-01 01:23:01 -0700, Lukas Fittl wrote:
> On Fri, Jun 12, 2020 at 4:28 PM Andres Freund <andres@anarazel.de> wrote:
> 
> > The attached patches are really just a prototype. I'm also not really
> > planning to work on getting this into a "production ready" patchset
> > anytime soon. I developed it primarily because I found it the overhead
> > made it too hard to nail down in which part of a query tree performance
> > changed.  If somebody else wants to continue from here...
> >
> > I do think it's be a pretty significant improvement if we could reduce
> > the timing overhead of EXPLAIN ANALYZE by this much. Even if requires a
> > bunch of low-level code.
> >
> 
> Based on an off-list conversation with Andres, I decided to dust off this
> old
> patch for using rdtsc directly. The significant EXPLAIN ANALYZE performance
> improvements (especially when using rdtsc instead of rdtsc*p*) seem to
> warrant
> giving this a more thorough look.
> 
> See attached an updated patch (adding it to the July commitfest), with a few
> changes:
> 
> - Keep using clock_gettime() as a fallback if we decide to not use rdtsc

Yep.


> - Fallback to /proc/cpuinfo for clock frequency, if cpuid(0x16) doesn't work

I suspect that this might not be needed anymore. Seems like it'd be ok to just
fall back to clock_gettime() in that case.


> - In an abundance of caution, for now I've decided to only enable this if we
>   are on Linux/x86, and the current kernel clocksource is TSC (the kernel
> has
>   quite sophisticated logic around making this decision, see [1])

I think our requirements are a bit lower than the kernel's - we're not
tracking wall clock over an extended period...


> Note that if we implemented the decision logic ourselves (instead of relying
> on the Linux kernel), I'd be most worried about older virtualization
> technology. In my understanding getting this right is notably more
> complicated
> than just checking cpuid, see [2].


> Known WIP problems with this patch version:
> 
> * There appears to be a timing discrepancy I haven't yet worked out, where
>   the \timing data reported by psql doesn't match what EXPLAIN ANALYZE is
>   reporting. With Andres' earlier test case, I'm seeing a consistent ~700ms
>   higher for \timing than for the EXPLAIN ANALYZE time reported on the
> server
>   side, only when rdtsc measurement is used -- its likely there is a problem
>   somewhere with how we perform the cycles to time conversion

Could you explain a bit more what you're seeing? I just tested your patches
and didn't see that here.


> * Possibly related, the floating point handling for the cycles_to_sec
> variable
>   is problematic in terms of precision (see FIXME, taken over from Andres'
> POC)

And probably also performance...


> Open questions from me:
> 
> 1) Do we need to account for different TSC offsets on different CPUs in SMP
>    systems? (the Linux kernel certainly has logic to that extent, but [3]
>    suggests this is no longer a problem on Nehalem and newer chips, i.e.
> those
>    having an invariant TSC)

I don't think we should cater to systems where we need that.


> 2) Should we have a setting "--with-tsc" for configure? (instead of always
>    enabling it when on Linux/x86 with a TSC clocksource)

Probably not worth it.


> 3) Are there cases where we actually want to use rdtsc*p*? (i.e. wait for
>    current instructions to finish -- the prior discussion seemed to suggest
>    we don't want it for node instruction measurements, but possibly we do
> want
>    this in other cases?)

I was wondering about that too... Perhaps we should add a
INSTR_TIME_SET_CURRENT_BARRIER() or such?


> 4) Should we support using the "mrs" instruction on ARM? (which is similar
> to
>    rdtsc, see [4])

I'd leave that for later personally.



>  #define NS_PER_S INT64CONST(1000000000)
>  #define US_PER_S INT64CONST(1000000)
>  #define MS_PER_S INT64CONST(1000)
> @@ -95,17 +104,37 @@ typedef int64 instr_time;
>  
>  #define INSTR_TIME_SET_ZERO(t)	((t) = 0)
>  
> -static inline instr_time pg_clock_gettime_ns(void)
> +extern double cycles_to_sec;
> +
> +bool use_rdtsc;

This should be extern and inside the ifdef below.


> +#if defined(__x86_64__) && defined(__linux__)
> +extern void pg_clock_gettime_initialize_rdtsc(void);
> +#endif
> +
> +static inline instr_time pg_clock_gettime_ref_cycles(void)
>  {
>  	struct timespec tmp;
>  
> +#if defined(__x86_64__) && defined(__linux__)
> +	if (use_rdtsc)
> +		return __rdtsc();
> +#endif
> +
>  	clock_gettime(PG_INSTR_CLOCK, &tmp);
>  
>  	return tmp.tv_sec * NS_PER_S + tmp.tv_nsec;
>  }
>  

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Maciek Sakrejda <m.sakrejda@gmail.com> — 2022-07-15T18:21:51Z

I ran that original test case with and without the patch. Here are the
numbers I'm seeing:

master (best of three):

postgres=# SELECT count(*) FROM lotsarows;
Time: 582.423 ms

postgres=# EXPLAIN (ANALYZE, TIMING OFF) SELECT count(*) FROM lotsarows;
Time: 616.102 ms

postgres=# EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 1068.700 ms (00:01.069)

patched (best of three):

postgres=# SELECT count(*) FROM lotsarows;
Time: 550.822 ms

postgres=# EXPLAIN (ANALYZE, TIMING OFF) SELECT count(*) FROM lotsarows;
Time: 612.572 ms

postgres=# EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 690.875 ms

On Fri, Jul 1, 2022 at 10:26 AM Andres Freund <andres@anarazel.de> wrote:
> On 2022-07-01 01:23:01 -0700, Lukas Fittl wrote:
>...
> > Known WIP problems with this patch version:
> >
> > * There appears to be a timing discrepancy I haven't yet worked out, where
> >   the \timing data reported by psql doesn't match what EXPLAIN ANALYZE is
> >   reporting. With Andres' earlier test case, I'm seeing a consistent ~700ms
> >   higher for \timing than for the EXPLAIN ANALYZE time reported on the
> > server
> >   side, only when rdtsc measurement is used -- its likely there is a problem
> >   somewhere with how we perform the cycles to time conversion
>
> Could you explain a bit more what you're seeing? I just tested your patches
> and didn't see that here.

I did not see this either, but I did see that the execution time
reported by \timing is (for this test case) consistently 0.5-1ms
*lower* than the Execution Time reported by EXPLAIN. I did not see
that on master. Is that expected?

Thanks,
Maciek

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Ibrar Ahmed <ibrar.ahmad@gmail.com> — 2022-09-06T06:32:18Z

On Fri, Jul 15, 2022 at 11:22 PM Maciek Sakrejda <m.sakrejda@gmail.com>
wrote:

> I ran that original test case with and without the patch. Here are the
> numbers I'm seeing:
>
> master (best of three):
>
> postgres=# SELECT count(*) FROM lotsarows;
> Time: 582.423 ms
>
> postgres=# EXPLAIN (ANALYZE, TIMING OFF) SELECT count(*) FROM lotsarows;
> Time: 616.102 ms
>
> postgres=# EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 1068.700 ms (00:01.069)
>
> patched (best of three):
>
> postgres=# SELECT count(*) FROM lotsarows;
> Time: 550.822 ms
>
> postgres=# EXPLAIN (ANALYZE, TIMING OFF) SELECT count(*) FROM lotsarows;
> Time: 612.572 ms
>
> postgres=# EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 690.875 ms
>
> On Fri, Jul 1, 2022 at 10:26 AM Andres Freund <andres@anarazel.de> wrote:
> > On 2022-07-01 01:23:01 -0700, Lukas Fittl wrote:
> >...
> > > Known WIP problems with this patch version:
> > >
> > > * There appears to be a timing discrepancy I haven't yet worked out,
> where
> > >   the \timing data reported by psql doesn't match what EXPLAIN ANALYZE
> is
> > >   reporting. With Andres' earlier test case, I'm seeing a consistent
> ~700ms
> > >   higher for \timing than for the EXPLAIN ANALYZE time reported on the
> > > server
> > >   side, only when rdtsc measurement is used -- its likely there is a
> problem
> > >   somewhere with how we perform the cycles to time conversion
> >
> > Could you explain a bit more what you're seeing? I just tested your
> patches
> > and didn't see that here.
>
> I did not see this either, but I did see that the execution time
> reported by \timing is (for this test case) consistently 0.5-1ms
> *lower* than the Execution Time reported by EXPLAIN. I did not see
> that on master. Is that expected?
>
> Thanks,
> Maciek
>
>
> The patch requires a rebase; please rebase the patch with the latest code.

Hunk #5 succeeded at 147 with fuzz 2 (offset -3 lines).
Hunk #6 FAILED at 170.
Hunk #7 succeeded at 165 (offset -69 lines).
2 out of 7 hunks FAILED -- saving rejects to file
src/include/portability/instr_time.h.rej
patching file src/tools/msvc/Mkvcbuild.pm



-- 
Ibrar Ahmed

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Michael Paquier <michael@paquier.xyz> — 2022-10-12T08:33:42Z

On Tue, Sep 06, 2022 at 11:32:18AM +0500, Ibrar Ahmed wrote:
> Hunk #5 succeeded at 147 with fuzz 2 (offset -3 lines).
> Hunk #6 FAILED at 170.
> Hunk #7 succeeded at 165 (offset -69 lines).
> 2 out of 7 hunks FAILED -- saving rejects to file
> src/include/portability/instr_time.h.rej
> patching file src/tools/msvc/Mkvcbuild.pm

No rebased version has been sent since this update, so this patch has
been marked as RwF.
--
Michael

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2022-11-19T20:05:28Z

I think it would be great to get this patch committed. Beyond the 
reasons already mentioned, the significant overhead also tends to skew 
the reported runtimes in ways that makes it difficult to compare them. 
For example, if two nodes are executed equally often but one needs twice 
the time to process the rows: in such a case EXPLAIN ANALYZE should 
report timings that are 2x apart. However, currently, the high overhead 
of clock_gettime() tends to skew the relative runtimes.

On 10/12/22 10:33, Michael Paquier wrote:
> No rebased version has been sent since this update, so this patch has
> been marked as RwF.

I've rebased the patch set on latest master and fixed a few compiler 
warnings. Beyond that some findings and thoughts:

You're only using RDTSC if the clock source is 'tsc'. Great idea to not 
bother caring about a lot of hairy TSC details. Looking at the kernel 
code this seems to imply that the TSC is frequency invariant. I don't 
think though that this implies that Linux is not running under a 
hypervisor; which is good because I assume PostgreSQL is used a lot in 
VMs. However, when running under a hypervisor (at least with VMWare) 
CPUID leaf 0x16 is not available. In my tests __get_cpuid() indicated 
success but the returned values were garbage. Instead of using leaf 
0x16, we should then use the hypervisor interface to obtain the TSC 
frequency. Checking if a hypervisor is active can be done via:

bool IsHypervisorActive()
{
     uint32 cpuinfo[4] = {0};
     int res = __get_cpuid(0x1, &cpuinfo[0], &cpuinfo[1], &cpuinfo[2], 
&cpuinfo[3]);
     return res > 0 && (cpuinfo[2] & (1 << 30));
}

Obtaining the TSC frequency via the hypervisor interface can be done 
with the following code. See https://lwn.net/Articles/301888/ for more 
details.

// Under hypervisors (tested with VMWare) leaf 0x16 is not available, 
even though __get_cpuid() succeeds.
// Hence, if running under a hypervisor, use the hypervisor interface to 
obtain TSC frequency.
uint32 cpuinfo[4] = {0};
if (IsHypervisorActive() && __get_cpuid(0x40000001, &cpuinfo[0], 
&cpuinfo[1], &cpuinfo[2], &cpuinfo[3]) > 0)
     cycles_to_sec = 1.0 / ((double)cpuinfo[0] * 1000 * 1000);

Given that we anyways switch between RDTSC and clock_gettime() with a 
global variable, what about exposing the clock source as GUC? That way 
the user can switch back to a working clock source in case we miss a 
detail around activating or reading the TSC.

I'm happy to update the patches accordingly.

--
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2022-11-19T20:06:23Z
```
I missed attaching the patches.

-- 
David Geier
(ServiceNow)
```

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-02T13:28:20Z

Hi,

I re-based again on master and applied the following changes:

I removed the fallback for obtaining the TSC frequency from /proc/cpu as 
suggested by Andres. Worst-case we fall back to clock_gettime().

I added code to obtain the TSC frequency via CPUID when under a 
hypervisor. I had to use __cpuid() directly instead of __get_cpuid(), 
because __get_cpuid() returns an error if the leaf is > 0x80000000 
(probably the implementation pre-dates the hypervisor timing leafs). 
Unfortunately, while testing my implementation under VMWare, I found 
that RDTSC runs awfully slow there (like 30x slower). [1] indicates that 
we cannot generally rely on RDTSC being actually fast on VMs. However, 
the same applies to clock_gettime(). It runs as slow as RDTSC on my 
VMWare setup. Hence, using RDTSC is not at disadvantage. I'm not 
entirely sure if there aren't cases where e.g. clock_gettime() is 
actually faster than RDTSC and it would be advantageous to use 
clock_gettime(). We could add a GUC so that the user can decide which 
clock source to use. Any thoughts?

I also somewhat improved the accuracy of the cycles to milli- and 
microseconds conversion functions by having two more multipliers with 
higher precision. For microseconds we could also keep the computation 
integer-only. I'm wondering what to best do for seconds and 
milliseconds. I'm currently leaning towards just keeping it as is, 
because the durations measured and converted are usually long enough 
that precision shouldn't be a problem.

In vacuum_lazy.c we do if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000). I 
changed that to use INSTR_TIME_GET_MILLISECS() instead. Additionally, I 
initialized a few variables of type instr_time which otherwise resulted 
in warnings due to use of potentially uninitialized variables.

I also couldn't reproduce the reported timing discrepancy. For me the 
runtime reported by \timing is just slightly higher than the time 
reported by EXPLAIN ANALYZE, which is expected.

Beyond that:

What about renaming INSTR_TIME_GET_DOUBLE() to INSTR_TIME_GET_SECS() so 
that it's consistent with the _MILLISEC() and _MICROSEC() variants?

The INSTR_TIME_GET_MICROSEC() returns a uint64 while the other variants 
return double. This seems error prone. What about renaming the function 
or also have the function return a double and cast where necessary at 
the call site?

If no one objects I would also re-register this patch in the commit fest.

[1] 
https://vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf 
(page 11 "Virtual TSC")

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2023-01-02T19:50:04Z

Hi David,

Thanks for continuing to work on this patch, and my apologies for silence
on the patch.

Its been hard to make time, and especially so because I typically develop
on an ARM-based macOS system where I can't test this directly - hence my
tests with virtualized EC2 instances, where I ran into the timing oddities.

On Mon, Jan 2, 2023 at 5:28 AM David Geier <geidav.pg@gmail.com> wrote:

> The INSTR_TIME_GET_MICROSEC() returns a uint64 while the other variants
> return double. This seems error prone. What about renaming the function
> or also have the function return a double and cast where necessary at
> the call site?
>

Minor note, but in my understanding using a uint64 (where we can) is faster
for any simple arithmetic we do with the values.

> If no one objects I would also re-register this patch in the commit fest.
>

+1, and feel free to carry this patch forward - I'll try to make an effort
to review my earlier testing issues again, as well as your later
improvements to the patch.

Also, FYI, I just posted an alternate idea for speeding up EXPLAIN ANALYZE
with timing over in [0], using a sampling-based approach to reduce the
timing overhead.

[0]:
https://www.postgresql.org/message-id/CAP53PkxXMk0j-%2B0%3DYwRti2pFR5UB2Gu4v2Oyk8hhZS0DRART6g%40mail.gmail.com

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Maciek Sakrejda <m.sakrejda@gmail.com> — 2023-01-02T20:44:42Z

On Fri, Jul 15, 2022 at 11:21 AM Maciek Sakrejda <m.sakrejda@gmail.com> wrote:
> On Fri, Jul 1, 2022 at 10:26 AM Andres Freund <andres@anarazel.de> wrote:
> > On 2022-07-01 01:23:01 -0700, Lukas Fittl wrote:
> >...
> > > Known WIP problems with this patch version:
> > >
> > > * There appears to be a timing discrepancy I haven't yet worked out, where
> > >   the \timing data reported by psql doesn't match what EXPLAIN ANALYZE is
> > >   reporting. With Andres' earlier test case, I'm seeing a consistent ~700ms
> > >   higher for \timing than for the EXPLAIN ANALYZE time reported on the
> > > server
> > >   side, only when rdtsc measurement is used -- its likely there is a problem
> > >   somewhere with how we perform the cycles to time conversion
> >
> > Could you explain a bit more what you're seeing? I just tested your patches
> > and didn't see that here.
>
> I did not see this either, but I did see that the execution time
> reported by \timing is (for this test case) consistently 0.5-1ms
> *lower* than the Execution Time reported by EXPLAIN. I did not see
> that on master. Is that expected?

For what it's worth, I can no longer reproduce this. In fact, I went
back to master-as-of-around-then and applied Lukas' v2 patches again,
and I still can't reproduce that. I do remember it happening
consistently across several executions, but now \timing consistently
shows 0.5-1ms slower, as expected. This does not explain the different
timing issue Lukas was seeing in his tests, but I think we can assume
what I reported originally here is not an issue.

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-03T08:38:20Z

Hi Lukas,

On 1/2/23 20:50, Lukas Fittl wrote:
> Thanks for continuing to work on this patch, and my apologies for 
> silence on the patch.

It would be great if you could review it.
Please also share your thoughts around exposing the used clock source as 
GUC and renaming INSTR_TIME_GET_DOUBLE() to _SECS().

I rebased again on master because of [1]. Patches attached.

>
> Its been hard to make time, and especially so because I typically 
> develop on an ARM-based macOS system where I can't test this directly 
> - hence my tests with virtualized EC2 instances, where I ran into the 
> timing oddities.
That's good and bad. Bad to do the development and good to test the 
implementation on more virtualized setups; given that I also encountered 
"interesting" behavior on VMWare (see my previous mails).
>
> On Mon, Jan 2, 2023 at 5:28 AM David Geier <geidav.pg@gmail.com> wrote:
>
>     The INSTR_TIME_GET_MICROSEC() returns a uint64 while the other
>     variants
>     return double. This seems error prone. What about renaming the
>     function
>     or also have the function return a double and cast where necessary at
>     the call site?
>
>
> Minor note, but in my understanding using a uint64 (where we can) is 
> faster for any simple arithmetic we do with the values.

That's true. So the argument could be that for seconds and milliseconds 
we want the extra precision while microseconds are precise enough. 
Still, we could also make the seconds and milliseconds conversion code 
integer only and e.g. return two integers with the value before and 
after the comma. FWICS, the functions are nowhere used in performance 
critical code, so it doesn't really make a difference performance-wise.

>
> +1, and feel free to carry this patch forward - I'll try to make an 
> effort to review my earlier testing issues again, as well as your 
> later improvements to the patch.
Moved to the current commit fest. Will you become reviewer?
>
> Also, FYI, I just posted an alternate idea for speeding up EXPLAIN 
> ANALYZE with timing over in [0], using a sampling-based approach to 
> reduce the timing overhead.

Interesting idea. I'll reply with some thoughts on the corresponding thread.

[1] 
https://www.postgresql.org/message-id/flat/CALDaNm3kRBGPhndujr9JcjjbDCG3anhj0vW8b9YtbXrBDMSvvw%40mail.gmail.com

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

vignesh C <vignesh21@gmail.com> — 2023-01-04T10:15:05Z

On Tue, 3 Jan 2023 at 14:08, David Geier <geidav.pg@gmail.com> wrote:
>
> Hi Lukas,
>
> On 1/2/23 20:50, Lukas Fittl wrote:
> > Thanks for continuing to work on this patch, and my apologies for
> > silence on the patch.
>
> It would be great if you could review it.
> Please also share your thoughts around exposing the used clock source as
> GUC and renaming INSTR_TIME_GET_DOUBLE() to _SECS().
>
> I rebased again on master because of [1]. Patches attached.
>
> >
> > Its been hard to make time, and especially so because I typically
> > develop on an ARM-based macOS system where I can't test this directly
> > - hence my tests with virtualized EC2 instances, where I ran into the
> > timing oddities.
> That's good and bad. Bad to do the development and good to test the
> implementation on more virtualized setups; given that I also encountered
> "interesting" behavior on VMWare (see my previous mails).
> >
> > On Mon, Jan 2, 2023 at 5:28 AM David Geier <geidav.pg@gmail.com> wrote:
> >
> >     The INSTR_TIME_GET_MICROSEC() returns a uint64 while the other
> >     variants
> >     return double. This seems error prone. What about renaming the
> >     function
> >     or also have the function return a double and cast where necessary at
> >     the call site?
> >
> >
> > Minor note, but in my understanding using a uint64 (where we can) is
> > faster for any simple arithmetic we do with the values.
>
> That's true. So the argument could be that for seconds and milliseconds
> we want the extra precision while microseconds are precise enough.
> Still, we could also make the seconds and milliseconds conversion code
> integer only and e.g. return two integers with the value before and
> after the comma. FWICS, the functions are nowhere used in performance
> critical code, so it doesn't really make a difference performance-wise.
>
> >
> > +1, and feel free to carry this patch forward - I'll try to make an
> > effort to review my earlier testing issues again, as well as your
> > later improvements to the patch.
> Moved to the current commit fest. Will you become reviewer?
> >
> > Also, FYI, I just posted an alternate idea for speeding up EXPLAIN
> > ANALYZE with timing over in [0], using a sampling-based approach to
> > reduce the timing overhead.
>
> Interesting idea. I'll reply with some thoughts on the corresponding thread.
>
> [1]
> https://www.postgresql.org/message-id/flat/CALDaNm3kRBGPhndujr9JcjjbDCG3anhj0vW8b9YtbXrBDMSvvw%40mail.gmail.com

CFBot shows some compilation errors as in [1], please post an updated
version for the same:
09:08:12.525] /usr/bin/ld:
src/bin/pg_test_timing/pg_test_timing.p/pg_test_timing.c.o: warning:
relocation against `cycles_to_sec' in read-only section `.text'
[09:08:12.525] /usr/bin/ld:
src/bin/pg_test_timing/pg_test_timing.p/pg_test_timing.c.o: in
function `pg_clock_gettime_ref_cycles':
[09:08:12.525] /tmp/cirrus-ci-build/build/../src/include/portability/instr_time.h:119:
undefined reference to `use_rdtsc'
[09:08:12.525] /usr/bin/ld:
src/bin/pg_test_timing/pg_test_timing.p/pg_test_timing.c.o: in
function `test_timing':
[09:08:12.525] /tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:135:
undefined reference to `pg_clock_gettime_initialize_rdtsc'
[09:08:12.525] /usr/bin/ld:
/tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:137:
undefined reference to `cycles_to_us'
[09:08:12.525] /usr/bin/ld:
/tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:146:
undefined reference to `cycles_to_us'
[09:08:12.525] /usr/bin/ld:
/tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:169:
undefined reference to `cycles_to_us'
[09:08:12.525] /usr/bin/ld:
/tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:176:
undefined reference to `cycles_to_sec'
[09:08:12.525] /usr/bin/ld: warning: creating DT_TEXTREL in a PIE
[09:08:12.525] collect2: error: ld returned 1 exit status

[1] - https://cirrus-ci.com/task/5375312565895168

Regards,
Vignesh

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-04T12:02:05Z

Hi,

> CFBot shows some compilation errors as in [1], please post an updated
> version for the same:
> 09:08:12.525] /usr/bin/ld:
> src/bin/pg_test_timing/pg_test_timing.p/pg_test_timing.c.o: warning:
> relocation against `cycles_to_sec' in read-only section `.text'
> [09:08:12.525] /usr/bin/ld:
> src/bin/pg_test_timing/pg_test_timing.p/pg_test_timing.c.o: in
> function `pg_clock_gettime_ref_cycles':
> [09:08:12.525] /tmp/cirrus-ci-build/build/../src/include/portability/instr_time.h:119:
> undefined reference to `use_rdtsc'
> [09:08:12.525] /usr/bin/ld:
> src/bin/pg_test_timing/pg_test_timing.p/pg_test_timing.c.o: in
> function `test_timing':
> [09:08:12.525] /tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:135:
> undefined reference to `pg_clock_gettime_initialize_rdtsc'
> [09:08:12.525] /usr/bin/ld:
> /tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:137:
> undefined reference to `cycles_to_us'
> [09:08:12.525] /usr/bin/ld:
> /tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:146:
> undefined reference to `cycles_to_us'
> [09:08:12.525] /usr/bin/ld:
> /tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:169:
> undefined reference to `cycles_to_us'
> [09:08:12.525] /usr/bin/ld:
> /tmp/cirrus-ci-build/build/../src/bin/pg_test_timing/pg_test_timing.c:176:
> undefined reference to `cycles_to_sec'
> [09:08:12.525] /usr/bin/ld: warning: creating DT_TEXTREL in a PIE
> [09:08:12.525] collect2: error: ld returned 1 exit status
>
> [1] - https://cirrus-ci.com/task/5375312565895168
>
> Regards,
> Vignesh

I fixed the compilation error on CFBot.
I missed adding instr_time.c to the Meson makefile.
New patch set attached.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-13T19:55:47Z

Hi,

On 2023-01-04 13:02:05 +0100, David Geier wrote:
> From be18633d4735f680c7910fcb4e8ac90c4eada131 Mon Sep 17 00:00:00 2001
> From: David Geier <geidav.pg@gmail.com>
> Date: Thu, 17 Nov 2022 10:22:01 +0100
> Subject: [PATCH 1/3] Change instr_time to just store nanoseconds, that's
>  cheaper.

Does anybody see a reason to not move forward with this aspect? We do a fair
amount of INSTR_TIME_ACCUM_DIFF() etc, and that gets a good bit cheaper by
just using nanoseconds. We'd also save memory in BufferUsage (144-122 bytes),
Instrumentation (16 bytes saved in Instrumentation itself, 32 via
BufferUsage).

While the range of instr_time storing nanoseconds wouldn't be good enough for
a generic timestamp facility (hence using microsecs for Timestamp), the range
seems plenty for its use of measuring runtime:

(2 ** 63) - 1) / ((10 ** 9) * 60 * 60 * 24 * 365) = ~292 years

Of course, when using CLOCK_REALTIME, this is relative to 1970-01-01, so just
239 years.

It could theoretically be a different story, if we stored instr_time's on
disk. But we don't, they're ephemeral.

This doesn't buy a whole lot of performance - the bottlenck is the actual
timestamp computation. But in a query with not much else going on, it's
visible and reproducible. It's, unsurprisingly, a lot easier to see when using
BUFFERS.

For both timespec and nanosecond, I measured three server starts, and for each
started server three executions of
pgbench -n -Mprepared -c1 -P5 -T15 -f <(echo "EXPLAIN (ANALYZE, BUFFERS) SELECT generate_series(1, 10000000) OFFSET 10000000;")

the best result is:
timespec: 1073.431
nanosec:  957.532
a ~10% difference

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-13T20:25:16Z

Andres Freund <andres@anarazel.de> writes:
> On 2023-01-04 13:02:05 +0100, David Geier wrote:
>> Subject: [PATCH 1/3] Change instr_time to just store nanoseconds, that's
>> cheaper.

> Does anybody see a reason to not move forward with this aspect? We do a fair
> amount of INSTR_TIME_ACCUM_DIFF() etc, and that gets a good bit cheaper by
> just using nanoseconds.

Cheaper, and perhaps more accurate too?  Don't recall if we have any code
paths where the input timestamps are likely to be better-than-microsecond,
but surely that's coming someday.

I'm unsure that we want to deal with rdtsc's vagaries in general, but
no objection to changing instr_time.

			regards, tom lane

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-13T20:59:33Z

Hi,

On 2023-01-13 15:25:16 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Does anybody see a reason to not move forward with this aspect? We do a fair
> > amount of INSTR_TIME_ACCUM_DIFF() etc, and that gets a good bit cheaper by
> > just using nanoseconds.
>
> Cheaper, and perhaps more accurate too?  Don't recall if we have any code
> paths where the input timestamps are likely to be better-than-microsecond,
> but surely that's coming someday.

instr_time on !WIN32 use struct timespec, so we already should have nanosecond
precision available. IOW, we could add a INSTR_TIME_GET_NANOSEC today. Or am I
misunderstanding what you mean?

> I'm unsure that we want to deal with rdtsc's vagaries in general, but
> no objection to changing instr_time.

Cool.

Looking at the instr_time.h part of the change, I think it should go further,
and basically do the same thing in the WIN32 path. The only part that needs to
be be win32 specific is INSTR_TIME_SET_CURRENT(). That'd reduce duplication a
good bit.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

vignesh C <vignesh21@gmail.com> — 2023-01-14T06:58:44Z

On Wed, 4 Jan 2023 at 17:32, David Geier <geidav.pg@gmail.com> wrote:
>
> I fixed the compilation error on CFBot.
> I missed adding instr_time.c to the Meson makefile.
> New patch set attached.

The patch does not apply on top of HEAD as in [1], please post a rebased patch:
=== Applying patches on top of PostgreSQL commit ID
ff23b592ad6621563d3128b26860bcb41daf9542 ===
=== applying patch
./0002-Use-CPU-reference-cycles-via-RDTSC-to-measure-time-v6.patch
....
patching file src/tools/msvc/Mkvcbuild.pm
Hunk #1 FAILED at 135.
1 out of 1 hunk FAILED -- saving rejects to file src/tools/msvc/Mkvcbuild.pm.rej

[1] - http://cfbot.cputube.org/patch_41_3751.log

Regards,
Vignesh

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-16T02:36:39Z

Hi,

On 2023-01-13 11:55:47 -0800, Andres Freund wrote:
> Does anybody see a reason to not move forward with this aspect? We do a fair
> amount of INSTR_TIME_ACCUM_DIFF() etc, and that gets a good bit cheaper by
> just using nanoseconds. We'd also save memory in BufferUsage (144-122 bytes),
> Instrumentation (16 bytes saved in Instrumentation itself, 32 via
> BufferUsage).

This actually under-counted the benefits, because we have two BufferUsage and
two WalUsage in Instrumentation.

Before:
        /* size: 448, cachelines: 7, members: 20 */
        /* sum members: 445, holes: 1, sum holes: 3 */
After
        /* size: 368, cachelines: 6, members: 20 */
        /* sum members: 365, holes: 1, sum holes: 3 */

The difference in the number of instructions in InstrStopNode is astounding:
1016 instructions with timespec, 96 instructions with nanoseconds. Some of
that is the simpler data structure, some because the compiler now can
auto-vectorize the four INSTR_TIME_ACCUM_DIFF in BufferUsageAccumDiff into
one.

We probably should convert Instrumentation->firsttuple to a instr_time now as
well, no point in having the code for conversion to double in the hot routine,
that can easily happen in explain. But that's for a later patch.

I suggested downthread that we should convert the win32 implementation to be
more similar to the unix-nanoseconds representation. A blind conversion looks
good, and lets us share a number of macros.

I wonder if we should deprecate INSTR_TIME_IS_ZERO()/INSTR_TIME_SET_ZERO() and
allow 0 to be used instead. Not needing INSTR_TIME_SET_ZERO() allows variable
definitions to initialize the value, which does avoid some unnecessarily
awkward code.  Alternatively we could introduce INSTR_TIME_ZERO() for that
purpose?

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-16T17:37:23Z

Hi,

On 2023-01-02 14:28:20 +0100, David Geier wrote:
> I also somewhat improved the accuracy of the cycles to milli- and
> microseconds conversion functions by having two more multipliers with higher
> precision. For microseconds we could also keep the computation integer-only.
> I'm wondering what to best do for seconds and milliseconds. I'm currently
> leaning towards just keeping it as is, because the durations measured and
> converted are usually long enough that precision shouldn't be a problem.

I'm doubtful this is worth the complexity it incurs. By the time we convert
out of the instr_time format, the times shouldn't be small enough that the
accuracy is affected much.

Looking around, most of the existing uses of INSTR_TIME_GET_MICROSEC()
actually accumulate themselves, and should instead keep things in the
instr_time format and convert later. We'd win more accuracy / speed that way.

I don't think the introduction of pg_time_usec_t was a great idea, but oh
well.

> Additionally, I initialized a few variables of type instr_time which
> otherwise resulted in warnings due to use of potentially uninitialized
> variables.

Unless we decide, as I suggested downthread, that we deprecate
INSTR_TIME_SET_ZERO(), that's unfortunately not the right fix. I've a similar
patch that adds all the necesarry INSTR_TIME_SET_ZERO() calls.

> What about renaming INSTR_TIME_GET_DOUBLE() to INSTR_TIME_GET_SECS() so that
> it's consistent with the _MILLISEC() and _MICROSEC() variants?

> The INSTR_TIME_GET_MICROSEC() returns a uint64 while the other variants
> return double. This seems error prone. What about renaming the function or
> also have the function return a double and cast where necessary at the call
> site?

I think those should be a separate discussion / patch.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Tomas Vondra <tomas.vondra@enterprisedb.com> — 2023-01-16T20:34:42Z

Hi,

there's minor bitrot in the Mkvcbuild.pm change, making cfbot unhappy.

As for the patch, I don't have much comments. I'm wondering if it'd be
useful to indicate which timing source was actually used for EXPLAIN
ANALYZE, say something like:

 Planning time: 0.197 ms
 Execution time: 0.225 ms
 Timing source: clock_gettime (or tsc)

There has been a proposal to expose this as a GUC (or perhaps as explain
option), to allow users to pick what timing source to use. I wouldn't go
that far - AFAICS is this is meant to be universally better when
available. But knowing which source was used seems useful.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Pavel Stehule <pavel.stehule@gmail.com> — 2023-01-16T20:39:04Z

po 16. 1. 2023 v 21:34 odesílatel Tomas Vondra <
tomas.vondra@enterprisedb.com> napsal:

> Hi,
>
> there's minor bitrot in the Mkvcbuild.pm change, making cfbot unhappy.
>
> As for the patch, I don't have much comments. I'm wondering if it'd be
> useful to indicate which timing source was actually used for EXPLAIN
> ANALYZE, say something like:
>
>  Planning time: 0.197 ms
>  Execution time: 0.225 ms
>  Timing source: clock_gettime (or tsc)
>
> There has been a proposal to expose this as a GUC (or perhaps as explain
> option), to allow users to pick what timing source to use. I wouldn't go
> that far - AFAICS is this is meant to be universally better when
> available. But knowing which source was used seems useful.
>

+1

Pavel


>
> regards
>
> --
> Tomas Vondra
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
>
>

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Robert Haas <robertmhaas@gmail.com> — 2023-01-17T13:46:12Z

On Fri, Jan 13, 2023 at 2:56 PM Andres Freund <andres@anarazel.de> wrote:
> Does anybody see a reason to not move forward with this aspect? We do a fair
> amount of INSTR_TIME_ACCUM_DIFF() etc, and that gets a good bit cheaper by
> just using nanoseconds. We'd also save memory in BufferUsage (144-122 bytes),
> Instrumentation (16 bytes saved in Instrumentation itself, 32 via
> BufferUsage).

I read through 0001 and it seems basically fine to me. Comments:

1. pg_clock_gettime_ns() doesn't follow pgindent conventions.

2. I'm not entirely sure that the new .?S_PER_.?S macros are
worthwhile but maybe they are, and in any case I don't care very much.

3. I've always found 'struct timespec' to be pretty annoying
notationally, so I like the fact that this patch would reduce use of
it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-17T16:47:58Z

Hi,

On 2023-01-17 08:46:12 -0500, Robert Haas wrote:
> On Fri, Jan 13, 2023 at 2:56 PM Andres Freund <andres@anarazel.de> wrote:
> > Does anybody see a reason to not move forward with this aspect? We do a fair
> > amount of INSTR_TIME_ACCUM_DIFF() etc, and that gets a good bit cheaper by
> > just using nanoseconds. We'd also save memory in BufferUsage (144-122 bytes),
> > Instrumentation (16 bytes saved in Instrumentation itself, 32 via
> > BufferUsage).

Here's an updated version of the move to representing instr_time as
nanoseconds. It's now split into a few patches:

0001) Add INSTR_TIME_SET_ZERO() calls where otherwise 0002 causes gcc to
      warn

      Alternatively we can decide to deprecate INSTR_TIME_SET_ZERO() and
      just allow to assign 0.

0002) Convert instr_time to uint64

      This is the cleaned up version of the prior patch. The main change is
      that it deduplicated a lot of the code between the architectures.

0003) Add INSTR_TIME_SET_SECOND()

      This is used in 0004. Just allows setting an instr_time to a time in
      seconds, allowing for a cheaper loop exit condition in 0004.

0004) report nanoseconds in pg_test_timing

I also couldn't help and hacked a bit on the rdtsc pieces. I did figure out
how to do the cycles->nanosecond conversion with integer shift and multiply in
the common case, which does show a noticable speedup. But that's for another
day.

I fought a bit with myself about whether to send those patches in this thread,
because it'll take over the CF entry. But decided that it's ok, given that
David's patches should be rebased over these anyway?

> I read through 0001 and it seems basically fine to me. Comments:
>
> 1. pg_clock_gettime_ns() doesn't follow pgindent conventions.

Fixed.

> 2. I'm not entirely sure that the new .?S_PER_.?S macros are
> worthwhile but maybe they are, and in any case I don't care very much.

There's now fewer. But those I'd like to keep. I just end up counting digits
manually way too many times.

> 3. I've always found 'struct timespec' to be pretty annoying
> notationally, so I like the fact that this patch would reduce use of
> it.

Same.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-17T17:26:57Z

Andres Freund <andres@anarazel.de> writes:
> Here's an updated version of the move to representing instr_time as
> nanoseconds. It's now split into a few patches:

I took a quick look through this.

> 0001) Add INSTR_TIME_SET_ZERO() calls where otherwise 0002 causes gcc to
>       warn
>       Alternatively we can decide to deprecate INSTR_TIME_SET_ZERO() and
>       just allow to assign 0.

I think it's probably wise to keep the macro.  If we ever rethink this
again, we'll be glad we kept it.  Similarly, IS_ZERO is a good idea
even if it would work with just compare-to-zero.  I'm almost tempted
to suggest you define instr_time as a struct with a uint64 field,
just to help keep us honest about that.

> 0003) Add INSTR_TIME_SET_SECOND()
>       This is used in 0004. Just allows setting an instr_time to a time in
>       seconds, allowing for a cheaper loop exit condition in 0004.

Code and comments are inconsistent about whether it's SET_SECOND or
SET_SECONDS.  I think I prefer the latter, but don't care that much.

> 0004) report nanoseconds in pg_test_timing

Didn't examine 0004 in any detail, but the others look good to go
other than these nits.

			regards, tom lane

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-17T18:50:53Z

Hi,

On 2023-01-17 12:26:57 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Here's an updated version of the move to representing instr_time as
> > nanoseconds. It's now split into a few patches:
> 
> I took a quick look through this.

Thanks!


> > 0001) Add INSTR_TIME_SET_ZERO() calls where otherwise 0002 causes gcc to
> >       warn
> >       Alternatively we can decide to deprecate INSTR_TIME_SET_ZERO() and
> >       just allow to assign 0.
> 
> I think it's probably wise to keep the macro.  If we ever rethink this
> again, we'll be glad we kept it.  Similarly, IS_ZERO is a good idea
> even if it would work with just compare-to-zero.

Perhaps an INSTR_TIME_ZERO() that could be assigned in variable definitions
could give us the best of both worlds?


> I'm almost tempted to suggest you define instr_time as a struct with a
> uint64 field, just to help keep us honest about that.

I can see that making sense. Unless somebody pipes up with opposition to that
plan soon, I'll see how it goes.


> > 0003) Add INSTR_TIME_SET_SECOND()
> >       This is used in 0004. Just allows setting an instr_time to a time in
> >       seconds, allowing for a cheaper loop exit condition in 0004.
> 
> Code and comments are inconsistent about whether it's SET_SECOND or
> SET_SECONDS.  I think I prefer the latter, but don't care that much.

That's probably because I couldn't decide... So I'll go with your preference.


> > 0004) report nanoseconds in pg_test_timing
> 
> Didn't examine 0004 in any detail, but the others look good to go
> other than these nits.

Thanks for looking!

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-18T12:52:05Z

On 1/16/23 21:39, Pavel Stehule wrote:
>
> po 16. 1. 2023 v 21:34 odesílatel Tomas Vondra 
> <tomas.vondra@enterprisedb.com> napsal:
>
>     Hi,
>
>     there's minor bitrot in the Mkvcbuild.pm change, making cfbot unhappy.
>
>     As for the patch, I don't have much comments. I'm wondering if it'd be
>     useful to indicate which timing source was actually used for EXPLAIN
>     ANALYZE, say something like:
>
>      Planning time: 0.197 ms
>      Execution time: 0.225 ms
>      Timing source: clock_gettime (or tsc)
>
>     There has been a proposal to expose this as a GUC (or perhaps as
>     explain
>     option), to allow users to pick what timing source to use. I
>     wouldn't go
>     that far - AFAICS is this is meant to be universally better when
>     available. But knowing which source was used seems useful.
>
>
> +1

Thanks for looking at the patch.

I'll fix the merge conflict.

I like the idea of exposing the timing source in the EXPLAIN ANALYZE output.
It's a good tradeoff between inspectability and effort, given that RDTSC 
should always be better to use.
If there are no objections I go this way.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-18T13:02:48Z

On 1/16/23 18:37, Andres Freund wrote:
> Hi,
>
> On 2023-01-02 14:28:20 +0100, David Geier wrote:
>
> I'm doubtful this is worth the complexity it incurs. By the time we convert
> out of the instr_time format, the times shouldn't be small enough that the
> accuracy is affected much.
I don't feel strong about it and you have a point that we most likely 
only convert ones we've accumulated a fair amount of cycles.
> Looking around, most of the existing uses of INSTR_TIME_GET_MICROSEC()
> actually accumulate themselves, and should instead keep things in the
> instr_time format and convert later. We'd win more accuracy / speed that way.
>
> I don't think the introduction of pg_time_usec_t was a great idea, but oh
> well.
Fully agreed. Why not replacing pg_time_usec_t with instr_time in a 
separate patch? I haven't looked carefully enough if all occurrences 
could sanely replaced but at least the ones that accumulate time seem 
good starting points.
>> Additionally, I initialized a few variables of type instr_time which
>> otherwise resulted in warnings due to use of potentially uninitialized
>> variables.
> Unless we decide, as I suggested downthread, that we deprecate
> INSTR_TIME_SET_ZERO(), that's unfortunately not the right fix. I've a similar
> patch that adds all the necesarry INSTR_TIME_SET_ZERO() calls.
I don't feel strong about it, but like Tom tend towards keeping the 
initialization macro.
Thanks that you have improved on the first patch and fixed these issues 
in a better way.
>> What about renaming INSTR_TIME_GET_DOUBLE() to INSTR_TIME_GET_SECS() so that
>> it's consistent with the _MILLISEC() and _MICROSEC() variants?
>> The INSTR_TIME_GET_MICROSEC() returns a uint64 while the other variants
>> return double. This seems error prone. What about renaming the function or
>> also have the function return a double and cast where necessary at the call
>> site?
> I think those should be a separate discussion / patch.

OK. I'll propose follow-on patches ones we're done with the ones at hand.

I'll then rebase the RDTSC patches on your patch set.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-18T13:05:35Z

Hi,

@Andres: will you take care of these changes and provide me with an 
updated patch set so I can rebase the RDTSC changes?
Otherwise, I can also apply Tom suggestions to your patch set and send 
out the complete patch set.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-19T10:47:49Z

Hi Andres,

> I also couldn't help and hacked a bit on the rdtsc pieces. I did figure out
> how to do the cycles->nanosecond conversion with integer shift and multiply in
> the common case, which does show a noticable speedup. But that's for another
> day.
I also have code for that here. I decided against integrating it because 
we don't convert frequently enough to make it matter. Or am I missing 
something?
> I fought a bit with myself about whether to send those patches in this thread,
> because it'll take over the CF entry. But decided that it's ok, given that
> David's patches should be rebased over these anyway?
That's alright.
Though, I would hope we attempt to bring your patch set as well as the 
RDTSC patch set in.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-20T06:43:00Z

On 1/18/23 13:52, David Geier wrote:
> On 1/16/23 21:39, Pavel Stehule wrote:
>>
>> po 16. 1. 2023 v 21:34 odesílatel Tomas Vondra 
>> <tomas.vondra@enterprisedb.com> napsal:
>>
>>     Hi,
>>
>>     there's minor bitrot in the Mkvcbuild.pm change, making cfbot 
>> unhappy.
>>
>>     As for the patch, I don't have much comments. I'm wondering if 
>> it'd be
>>     useful to indicate which timing source was actually used for EXPLAIN
>>     ANALYZE, say something like:
>>
>>      Planning time: 0.197 ms
>>      Execution time: 0.225 ms
>>      Timing source: clock_gettime (or tsc)
>>
>> +1
>
> I like the idea of exposing the timing source in the EXPLAIN ANALYZE 
> output.
> It's a good tradeoff between inspectability and effort, given that 
> RDTSC should always be better to use.
> If there are no objections I go this way.
Thinking about this a little more made me realize that this will cause 
different pg_regress output depending on the platform. So if we go this 
route we would at least need an option for EXPLAIN ANALYZE to disable 
it. Or rather have it disabled by default and allow for enabling it. 
Thoughts?

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Tomas Vondra <tomas.vondra@enterprisedb.com> — 2023-01-20T11:13:09Z


On 1/20/23 07:43, David Geier wrote:
> On 1/18/23 13:52, David Geier wrote:
>> On 1/16/23 21:39, Pavel Stehule wrote:
>>>
>>> po 16. 1. 2023 v 21:34 odesílatel Tomas Vondra
>>> <tomas.vondra@enterprisedb.com> napsal:
>>>
>>>     Hi,
>>>
>>>     there's minor bitrot in the Mkvcbuild.pm change, making cfbot
>>> unhappy.
>>>
>>>     As for the patch, I don't have much comments. I'm wondering if
>>> it'd be
>>>     useful to indicate which timing source was actually used for EXPLAIN
>>>     ANALYZE, say something like:
>>>
>>>      Planning time: 0.197 ms
>>>      Execution time: 0.225 ms
>>>      Timing source: clock_gettime (or tsc)
>>>
>>> +1
>>
>> I like the idea of exposing the timing source in the EXPLAIN ANALYZE
>> output.
>> It's a good tradeoff between inspectability and effort, given that
>> RDTSC should always be better to use.
>> If there are no objections I go this way.
> Thinking about this a little more made me realize that this will cause
> different pg_regress output depending on the platform. So if we go this
> route we would at least need an option for EXPLAIN ANALYZE to disable
> it. Or rather have it disabled by default and allow for enabling it.
> Thoughts?
> 

What about only showing it for VERBOSE mode? I don't think there are
very many tests doing EXPLAIN (ANALYZE, VERBOSE) - a quick grep found
one such place in partition_prune.sql.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T00:40:32Z

Hi,

On 2023-01-17 10:50:53 -0800, Andres Freund wrote:
> On 2023-01-17 12:26:57 -0500, Tom Lane wrote:
> > > 0001) Add INSTR_TIME_SET_ZERO() calls where otherwise 0002 causes gcc to
> > >       warn
> > >       Alternatively we can decide to deprecate INSTR_TIME_SET_ZERO() and
> > >       just allow to assign 0.
> >
> > I think it's probably wise to keep the macro.  If we ever rethink this
> > again, we'll be glad we kept it.  Similarly, IS_ZERO is a good idea
> > even if it would work with just compare-to-zero.
>
> Perhaps an INSTR_TIME_ZERO() that could be assigned in variable definitions
> could give us the best of both worlds?

I tried that in the attached 0005. I found that it reads better if I also add
INSTR_TIME_CURRENT(). If we decide to go for this, I'd roll it into 0001
instead, but I wanted to get agreement on it first.

Comments?

> > I'm almost tempted to suggest you define instr_time as a struct with a
> > uint64 field, just to help keep us honest about that.
>
> I can see that making sense. Unless somebody pipes up with opposition to that
> plan soon, I'll see how it goes.

Done in the attached. I think it looks good. Actually found a type confusion
buglet in 0004, so the type safety benefit is noticable.

It does require a new INSTR_TIME_IS_LT() for the loop exit condition in 0004,
but that seems fine.

Besides cosmetic stuff I also added back the cast to double in window's
INSTR_TIME_GET_NANOSEC() - I think there's an overflow danger without it.

We should make this faster by pre-computing
  (double) NS_PER_S / GetTimerFrequency()
once, as that'd avoid doing the the slow division on every conversion. But
that's an old issue and thus better tackled separately.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-21T03:27:07Z

Andres Freund <andres@anarazel.de> writes:
>> Perhaps an INSTR_TIME_ZERO() that could be assigned in variable definitions
>> could give us the best of both worlds?

> I tried that in the attached 0005. I found that it reads better if I also add
> INSTR_TIME_CURRENT(). If we decide to go for this, I'd roll it into 0001
> instead, but I wanted to get agreement on it first.

-1 from here.  This forecloses the possibility that it's best to use more
than one assignment to initialize the value, and the code doesn't read
any better than it did before.

			regards, tom lane

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T03:54:56Z

Hi,

On 2023-01-20 22:27:07 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> >> Perhaps an INSTR_TIME_ZERO() that could be assigned in variable definitions
> >> could give us the best of both worlds?
> 
> > I tried that in the attached 0005. I found that it reads better if I also add
> > INSTR_TIME_CURRENT(). If we decide to go for this, I'd roll it into 0001
> > instead, but I wanted to get agreement on it first.
> 
> -1 from here.  This forecloses the possibility that it's best to use more
> than one assignment to initialize the value, and the code doesn't read
> any better than it did before.

I think it does read a bit better, but it's a pretty small improvement. So
I'll leave this aspect be for now.

Thanks for checking.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T04:12:00Z

Hi,

On 2023-01-19 11:47:49 +0100, David Geier wrote:
> > I also couldn't help and hacked a bit on the rdtsc pieces. I did figure out
> > how to do the cycles->nanosecond conversion with integer shift and multiply in
> > the common case, which does show a noticable speedup. But that's for another
> > day.
> I also have code for that here. I decided against integrating it because we
> don't convert frequently enough to make it matter. Or am I missing
> something?

We do currently do the conversion quite frequently.  Admittedly I was
partially motivated by trying to get the per-loop overhead in pg_test_timing
down ;)

But I think it's a real issue. Places where we do, but shouldn't, convert:

- ExecReScan() - quite painful, we can end up with a lot of those
- InstrStopNode() - adds a good bit of overhead to simple
- PendingWalStats.wal_write_time - this is particularly bad because it happens
  within very contended code
- calls to pgstat_count_buffer_read_time(), pgstat_count_buffer_write_time() -
  they can be very frequent
- pgbench.c, as we already discussed
- pg_stat_statements.c
- ...

These all will get a bit slower when moving to a "variable" frequency.

What was your approach for avoiding the costly operation?  I ended up with a
integer multiplication + shift approximation for the floating point
multiplication (which in turn uses the inverse of the division by the
frequency). To allow for sufficient precision while also avoiding overflows, I
had to make that branch conditional, with a slow path for large numbers of
nanoseconds.

> > I fought a bit with myself about whether to send those patches in this thread,
> > because it'll take over the CF entry. But decided that it's ok, given that
> > David's patches should be rebased over these anyway?
> That's alright.
> Though, I would hope we attempt to bring your patch set as well as the RDTSC
> patch set in.

I think it'd be great - but I'm not sure we're there yet, reliability and
code-complexity wise.

I think it might be worth makign the rdts aspect somewhat
measurable. E.g. allowing pg_test_timing to use both at the same time, and
have it compare elapsed time with both sources of counters.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T04:14:39Z

Hi,

On 2023-01-20 07:43:00 +0100, David Geier wrote:
> On 1/18/23 13:52, David Geier wrote:
> > On 1/16/23 21:39, Pavel Stehule wrote:
> > > 
> > > po 16. 1. 2023 v 21:34 odesílatel Tomas Vondra
> > > <tomas.vondra@enterprisedb.com> napsal:
> > > 
> > >     Hi,
> > > 
> > >     there's minor bitrot in the Mkvcbuild.pm change, making cfbot
> > > unhappy.
> > > 
> > >     As for the patch, I don't have much comments. I'm wondering if
> > > it'd be
> > >     useful to indicate which timing source was actually used for EXPLAIN
> > >     ANALYZE, say something like:
> > > 
> > >      Planning time: 0.197 ms
> > >      Execution time: 0.225 ms
> > >      Timing source: clock_gettime (or tsc)
> > > 
> > > +1
> > 
> > I like the idea of exposing the timing source in the EXPLAIN ANALYZE
> > output.
> > It's a good tradeoff between inspectability and effort, given that RDTSC
> > should always be better to use.
> > If there are no objections I go this way.
> Thinking about this a little more made me realize that this will cause
> different pg_regress output depending on the platform. So if we go this
> route we would at least need an option for EXPLAIN ANALYZE to disable it. Or
> rather have it disabled by default and allow for enabling it. Thoughts?

The elapsed time is already inherently unstable, so we shouldn't have any test
output showing the time.

But I doubt showing it in every explain is a good idea - we use instr_time in
plenty of other places. Why show it in explain, but not in all those other
places?

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T04:16:13Z

Hi,

On 2023-01-18 14:05:35 +0100, David Geier wrote:
> @Andres: will you take care of these changes and provide me with an updated
> patch set so I can rebase the RDTSC changes?
> Otherwise, I can also apply Tom suggestions to your patch set and send out
> the complete patch set.

I'm planning to push most of my changes soon, had hoped to get to it a bit
sooner, but ...

If you have time to look at the pg_test_timing part, it'd be
appreciated. That's a it larger, and nobody looked at it yet. So I'm a bit
hesitant to push it.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T04:29:10Z

Hi,

On 2023-01-18 14:02:48 +0100, David Geier wrote:
> On 1/16/23 18:37, Andres Freund wrote:
> > I'm doubtful this is worth the complexity it incurs. By the time we convert
> > out of the instr_time format, the times shouldn't be small enough that the
> > accuracy is affected much.
>
> I don't feel strong about it and you have a point that we most likely only
> convert ones we've accumulated a fair amount of cycles.

I think we can avoid the issue another way. The inaccuracy comes from the
cycles_to_sec ending up very small, right? Right now your patch has (and
probably my old version similarly had):

cycles_to_sec = 1.0 / (tsc_freq * 1000);

I think it's better if we have one multiplier to convert cycles to nanoseconds
- that'll be a double comparatively close to 1. We can use that to implement
INSTR_TIME_GET_NANOSECONDS(). The conversion to microseconds then is just a
division by 1000 (which most compilers convert into a multiplication/shift
combo), and the conversions to milliseconds and seconds will be similar.

Because we'll never "wrongly" go into the "huge number" or "very small number"
ranges, that should provide sufficient precision? We'll of course still end up
with a very small number when converting a few nanoseconds to seconds, but
that's ok because it's the precision being asked for, instead of loosing
precision in some intermediate representation.

> > Looking around, most of the existing uses of INSTR_TIME_GET_MICROSEC()
> > actually accumulate themselves, and should instead keep things in the
> > instr_time format and convert later. We'd win more accuracy / speed that way.
> > 
> > I don't think the introduction of pg_time_usec_t was a great idea, but oh
> > well.
> Fully agreed. Why not replacing pg_time_usec_t with instr_time in a separate
> patch?

pgbench used to use instr_time, but it was replaced by somebody thinking the
API is too cumbersome. Which I can't quite deny, even though I think the
specific change isn't great.

But yes, this should definitely be a separate patch.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Justin Pryzby <pryzby@telsasoft.com> — 2023-01-21T04:50:37Z

On Fri, Jan 20, 2023 at 04:40:32PM -0800, Andres Freund wrote:
> From 5a458d4584961dedd3f80a07d8faea66e57c5d94 Mon Sep 17 00:00:00 2001
> From: Andres Freund <andres@anarazel.de>
> Date: Mon, 16 Jan 2023 11:19:11 -0800
> Subject: [PATCH v8 4/5] wip: report nanoseconds in pg_test_timing

>    <para>
> -   The i7-860 system measured runs the count query in 9.8 ms while
> -   the <command>EXPLAIN ANALYZE</command> version takes 16.6 ms, each
> -   processing just over 100,000 rows.  That 6.8 ms difference means the timing
> -   overhead per row is 68 ns, about twice what pg_test_timing estimated it
> -   would be.  Even that relatively small amount of overhead is making the fully
> -   timed count statement take almost 70% longer.  On more substantial queries,
> -   the timing overhead would be less problematic.
> +   The i9-9880H system measured shows an execution time of 4.116 ms for the
> +   <literal>TIMING OFF</literal> query, and 6.965 ms for the
> +   <literal>TIMING ON</literal>, each processing 100,000 rows.
> +
> +   That 2.849 ms difference means the timing overhead per row is 28 ns.  As
> +   <literal>TIMING ON</literal> measures timestamps twice per row returned by
> +   an executor node, the overhead is very close to what pg_test_timing
> +   estimated it would be.
> +
> +   more than what pg_test_timing estimated it would be.  Even that relatively
> +   small amount of overhead is making the fully timed count statement take
> +   about 60% longer.  On more substantial queries, the timing overhead would
> +   be less problematic.

I guess you intend to merge these two paragraphs ?

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T05:14:14Z

On 2023-01-20 22:50:37 -0600, Justin Pryzby wrote:
> On Fri, Jan 20, 2023 at 04:40:32PM -0800, Andres Freund wrote:
> > From 5a458d4584961dedd3f80a07d8faea66e57c5d94 Mon Sep 17 00:00:00 2001
> > From: Andres Freund <andres@anarazel.de>
> > Date: Mon, 16 Jan 2023 11:19:11 -0800
> > Subject: [PATCH v8 4/5] wip: report nanoseconds in pg_test_timing
> 
> >    <para>
> > -   The i7-860 system measured runs the count query in 9.8 ms while
> > -   the <command>EXPLAIN ANALYZE</command> version takes 16.6 ms, each
> > -   processing just over 100,000 rows.  That 6.8 ms difference means the timing
> > -   overhead per row is 68 ns, about twice what pg_test_timing estimated it
> > -   would be.  Even that relatively small amount of overhead is making the fully
> > -   timed count statement take almost 70% longer.  On more substantial queries,
> > -   the timing overhead would be less problematic.
> > +   The i9-9880H system measured shows an execution time of 4.116 ms for the
> > +   <literal>TIMING OFF</literal> query, and 6.965 ms for the
> > +   <literal>TIMING ON</literal>, each processing 100,000 rows.
> > +
> > +   That 2.849 ms difference means the timing overhead per row is 28 ns.  As
> > +   <literal>TIMING ON</literal> measures timestamps twice per row returned by
> > +   an executor node, the overhead is very close to what pg_test_timing
> > +   estimated it would be.
> > +
> > +   more than what pg_test_timing estimated it would be.  Even that relatively
> > +   small amount of overhead is making the fully timed count statement take
> > +   about 60% longer.  On more substantial queries, the timing overhead would
> > +   be less problematic.
> 
> I guess you intend to merge these two paragraphs ?

Oops. I was intending to drop the last paragraph.

Looking at the docs again I noticed that I needed to rephrase the 'acpi_pm'
section further, as I'd left the "a small multiple of what's measured directly
by this utility" language in there.

Do the changes otherwise make sense?

The "small multiple" stuff was just due to a) comparing "raw statement" with
explain analyze b) not accounting for two timestamps being taken per row.

I think it makes sense to remove the "jiffies" section - the output shown is
way outdated. And I don't think the jiffies time counter is one something
still sees in the wild, outside of bringing up a new cpu architecture or such.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T05:31:57Z

Hi,

On 2023-01-20 20:16:13 -0800, Andres Freund wrote:
> On 2023-01-18 14:05:35 +0100, David Geier wrote:
> > @Andres: will you take care of these changes and provide me with an updated
> > patch set so I can rebase the RDTSC changes?
> > Otherwise, I can also apply Tom suggestions to your patch set and send out
> > the complete patch set.
> 
> I'm planning to push most of my changes soon, had hoped to get to it a bit
> sooner, but ...

I pushed the int64-ification commits.

> If you have time to look at the pg_test_timing part, it'd be
> appreciated. That's a it larger, and nobody looked at it yet. So I'm a bit
> hesitant to push it.

I haven't yet pushed the pg_test_timing (nor it's small prerequisite)
patch.

Thanks to Justin I've polished the pg_test_timing docs some.

I've attached those two patches. Feel free to include them in your series if
you want, then the CF entry (and thus cfbot) makes sense again...

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-21T19:03:03Z

Hi,

On 2023-01-20 21:31:57 -0800, Andres Freund wrote:
> On 2023-01-20 20:16:13 -0800, Andres Freund wrote:
> > I'm planning to push most of my changes soon, had hoped to get to it a bit
> > sooner, but ...
>
> I pushed the int64-ification commits.

There's an odd compilation failure on AIX.

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&dt=2023-01-21%2007%3A01%3A42

/opt/IBM/xlc/16.1.0/bin/xlc_r -D_LARGE_FILES=1 -DRANDOMIZE_ALLOCATED_MEMORY -qnoansialias -g -O2 -qmaxmem=33554432 -qsuppress=1500-010:1506-995 -qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264 -qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse -qinfo=nounset  -qvisibility=hidden -I. -I. -I/opt/freeware/include/python3.5m -I../../../src/include -I/home/nm/sw/nopath/icu58.3-64/include   -I/home/nm/sw/nopath/libxml2-64/include/libxml2  -I/home/nm/sw/nopath/uuid-64/include -I/home/nm/sw/nopath/openldap-64/include -I/home/nm/sw/nopath/icu58.3-64/include -I/home/nm/sw/nopath/libxml2-64/include  -c -o plpy_cursorobject.o plpy_cursorobject.c
"../../../src/include/portability/instr_time.h", line 116.9: 1506-304 (I) No function prototype given for "clock_gettime".
"../../../src/include/portability/instr_time.h", line 116.23: 1506-045 (S) Undeclared identifier CLOCK_REALTIME.
<builtin>: recipe for target 'plpy_cursorobject.o' failed

but files including instr_time.h *do* build successfully, e.g. instrument.c:

/opt/IBM/xlc/16.1.0/bin/xlc_r -D_LARGE_FILES=1 -DRANDOMIZE_ALLOCATED_MEMORY -qnoansialias -g -O2 -qmaxmem=33554432 -qsuppress=1500-010:1506-995 -qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264 -qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse -qinfo=nounset -I../../../src/include -I/home/nm/sw/nopath/icu58.3-64/include   -I/home/nm/sw/nopath/libxml2-64/include/libxml2  -I/home/nm/sw/nopath/uuid-64/include -I/home/nm/sw/nopath/openldap-64/include -I/home/nm/sw/nopath/icu58.3-64/include -I/home/nm/sw/nopath/libxml2-64/include  -c -o instrument.o instrument.c

Before the change the clock_gettime() call was in a macro and thus could be
referenced even without a prior declaration, as long as places using
INSTR_TIME_SET_CURRENT() had all the necessary includes and defines.

Argh:

There's nice bit in plpython.h:

/*
 * Include order should be: postgres.h, other postgres headers, plpython.h,
 * other plpython headers.  (In practice, other plpython headers will also
 * include this file, so that they can compile standalone.)
 */
#ifndef POSTGRES_H
#error postgres.h must be included before plpython.h
#endif

/*
 * Undefine some things that get (re)defined in the Python headers. They aren't
 * used by the PL/Python code, and all PostgreSQL headers should be included
 * earlier, so this should be pretty safe.
 */
#undef _POSIX_C_SOURCE
#undef _XOPEN_SOURCE

the relevant stuff in time.h is indeed guarded by
#if _XOPEN_SOURCE>=500

I don't think the plpython actually code follows the rule about including all
postgres headers earlier.

plpy_typeio.h:

#include "access/htup.h"
#include "fmgr.h"
#include "plpython.h"
#include "utils/typcache.h"

plpy_curserobject.c:

#include "access/xact.h"
#include "catalog/pg_type.h"
#include "mb/pg_wchar.h"
#include "plpy_cursorobject.h"
#include "plpy_elog.h"
#include "plpy_main.h"
#include "plpy_planobject.h"
#include "plpy_procedure.h"
#include "plpy_resultobject.h"
#include "plpy_spi.h"
#include "plpython.h"
#include "utils/memutils.h"

It strikes me as a uh, not good idea to undefine _POSIX_C_SOURCE,
_XOPEN_SOURCE.

The include order aspect was perhaps feasible when there just was plpython.c,
but with the split into many different C files and many headers, it seems hard
to maintain. There's a lot of violations afaics.

The undefines were added in a11cf433413, the split in 147c2482542.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-23T17:23:17Z

Hi,

On 1/21/23 05:14, Andres Freund wrote:
> The elapsed time is already inherently unstable, so we shouldn't have any test
> output showing the time.
>
> But I doubt showing it in every explain is a good idea - we use instr_time in
> plenty of other places. Why show it in explain, but not in all those other
> places?

Yeah. I thought it would only be an issue if we showed it 
unconditionally in EXPLAIN ANALYZE. If we only show it with TIMING ON, 
we're likely fine with pretty much all regression tests.

But given the different opinions, I'll leave it out in the new patch set 
for the moment being.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-23T17:41:58Z

Hi,

On 2023-01-23 18:23:17 +0100, David Geier wrote:
> On 1/21/23 05:14, Andres Freund wrote:
> > The elapsed time is already inherently unstable, so we shouldn't have any test
> > output showing the time.
> > 
> > But I doubt showing it in every explain is a good idea - we use instr_time in
> > plenty of other places. Why show it in explain, but not in all those other
> > places?
> 
> Yeah. I thought it would only be an issue if we showed it unconditionally in
> EXPLAIN ANALYZE. If we only show it with TIMING ON, we're likely fine with
> pretty much all regression tests.

If we add it, it probably shouldn't depend on TIMING, but on
SUMMARY. Regression test queries showing EXPLAIN ANALYZE output all do
something like
  EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)

the SUMMARY OFF gets rid of the "top-level" "Planning Time" and "Execution
Time", whereas the TIMING OFF gets rid of the per-node timing. Those are
separate options because per-node timing is problematic performance-wise
(right now), but whole-query timing rarely is.

> But given the different opinions, I'll leave it out in the new patch set for
> the moment being.

Makes sense.

Another, independent, thing worth thinking about: I think we might want to
expose both rdtsc and rdtscp. For something like
InstrStartNode()/InstrStopNode(), avoiding the "one-way barrier" of rdtscp is
quite important to avoid changing the query performance. But for measuring
whole-query time, we likely want to measure the actual time.

It probably won't matter hugely for the whole query time - the out of order
window of modern CPUs is large, but not *that* large - but I don't think we
can generally assume that.

I'm thinking of something like INSTR_TIME_SET_CURRENT() and
INSTR_TIME_SET_CURRENT_FAST() or _NOBARRIER().

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-23T17:49:37Z

Hi,

On 1/21/23 05:12, Andres Freund wrote:
> We do currently do the conversion quite frequently.  Admittedly I was
> partially motivated by trying to get the per-loop overhead in pg_test_timing
> down ;)
>
> But I think it's a real issue. Places where we do, but shouldn't, convert:
>
> - ExecReScan() - quite painful, we can end up with a lot of those
> - InstrStopNode() - adds a good bit of overhead to simple
InstrStopNode() doesn't convert in the general case but only for the 
first tuple or when async. So it goes somewhat hand in hand with 
ExecReScan().
> - PendingWalStats.wal_write_time - this is particularly bad because it happens
>    within very contended code
> - calls to pgstat_count_buffer_read_time(), pgstat_count_buffer_write_time() -
>    they can be very frequent
> - pgbench.c, as we already discussed
> - pg_stat_statements.c
> - ...
>
> These all will get a bit slower when moving to a "variable" frequency.
I wonder if we will be able to measure any of them easily. But given 
that it's many more places than I had realized and given that the 
optimized code is not too involved, let's give it a try.
> What was your approach for avoiding the costly operation?  I ended up with a
> integer multiplication + shift approximation for the floating point
> multiplication (which in turn uses the inverse of the division by the
> frequency). To allow for sufficient precision while also avoiding overflows, I
> had to make that branch conditional, with a slow path for large numbers of
> nanoseconds.

It seems like we ended up with the same. I do:

sec = ticks / frequency_hz
ns  = ticks / frequency_hz * 1,000,000,000
ns  = ticks * (1,000,000,000 / frequency_hz)
ns  = ticks * (1,000,000 / frequency_khz) <-- now in kilohertz

Now, the constant scaling factor in parentheses is typically a floating 
point number. For example for a frequency of 2.5 GHz it would be 2.5. To 
work around that we can do something like:

ns  = ticks * (1,000,000 * scaler / frequency_khz) / scaler

Where scaler is a power-of-2, big enough to maintain enough precision 
while allowing for a shift to implement the division.

The additional multiplication with scaler makes that the maximum range 
go down, because we must ensure we never overflow. I'm wondering if we 
cannot pick scaler in such a way that remaining range of cycles is large 
enough for our use case and we can therefore live without bothering for 
the overflow case. What would be "enough"? 1 year? 10 years? ...

Otherwise, we indeed need code that cares for the potential overflow. My 
hunch is that it can be done branchless, but it for sure adds dependent 
instructions. Maybe in that case a branch is better that almost 
certainly will never be taken?

I'll include the code in the new patch set which I'll latest submit 
tomorrow.

> I think it'd be great - but I'm not sure we're there yet, reliability and
> code-complexity wise.
Thanks to your commits, the diff of the new patch set will be already 
much smaller and easier to review. What's your biggest concern in terms 
of reliability?
> I think it might be worth makign the rdts aspect somewhat
> measurable. E.g. allowing pg_test_timing to use both at the same time, and
> have it compare elapsed time with both sources of counters.
I haven't yet looked into pg_test_timing. I'll do that while including 
your patches into the new patch set.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-23T17:52:44Z

Hi,

On 1/21/23 06:31, Andres Freund wrote:
> I pushed the int64-ification commits.

Great. I started rebasing.

One thing I was wondering about: why did you chose to use a signed 
instead of an unsigned 64-bit integer for the ticks?
>> If you have time to look at the pg_test_timing part, it'd be
>> appreciated. That's a it larger, and nobody looked at it yet. So I'm a bit
>> hesitant to push it.
> I haven't yet pushed the pg_test_timing (nor it's small prerequisite)
> patch.
>
> I've attached those two patches. Feel free to include them in your series if
> you want, then the CF entry (and thus cfbot) makes sense again...
I'll include them in my new patch set and also have a careful look at them.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-23T20:26:19Z

Hi,

On 2023-01-23 18:49:37 +0100, David Geier wrote:
> On 1/21/23 05:12, Andres Freund wrote:
> > We do currently do the conversion quite frequently.  Admittedly I was
> > partially motivated by trying to get the per-loop overhead in pg_test_timing
> > down ;)
> > 
> > But I think it's a real issue. Places where we do, but shouldn't, convert:
> > 
> > - ExecReScan() - quite painful, we can end up with a lot of those
> > - InstrStopNode() - adds a good bit of overhead to simple
> InstrStopNode() doesn't convert in the general case but only for the first
> tuple or when async. So it goes somewhat hand in hand with ExecReScan().

I think even the first-scan portion is likely noticable for quick queries -
you can quickly end up with 5-10 nodes, even for queries processed in the <
0.1ms range.

Of course it's way worse with rescans / loops.


> > - PendingWalStats.wal_write_time - this is particularly bad because it happens
> >    within very contended code
> > - calls to pgstat_count_buffer_read_time(), pgstat_count_buffer_write_time() -
> >    they can be very frequent
> > - pgbench.c, as we already discussed
> > - pg_stat_statements.c
> > - ...
> > 
> > These all will get a bit slower when moving to a "variable" frequency.

> I wonder if we will be able to measure any of them easily. But given that
> it's many more places than I had realized and given that the optimized code
> is not too involved, let's give it a try.

I think at least some should be converted to just accumulate in an
instr_time...



> > What was your approach for avoiding the costly operation?  I ended up with a
> > integer multiplication + shift approximation for the floating point
> > multiplication (which in turn uses the inverse of the division by the
> > frequency). To allow for sufficient precision while also avoiding overflows, I
> > had to make that branch conditional, with a slow path for large numbers of
> > nanoseconds.
> 
> It seems like we ended up with the same. I do:
> 
> sec = ticks / frequency_hz
> ns  = ticks / frequency_hz * 1,000,000,000
> ns  = ticks * (1,000,000,000 / frequency_hz)
> ns  = ticks * (1,000,000 / frequency_khz) <-- now in kilohertz
> 
> Now, the constant scaling factor in parentheses is typically a floating
> point number. For example for a frequency of 2.5 GHz it would be 2.5. To
> work around that we can do something like:
> 
> ns  = ticks * (1,000,000 * scaler / frequency_khz) / scaler
> 
> Where scaler is a power-of-2, big enough to maintain enough precision while
> allowing for a shift to implement the division.

Yep, at least quite similar.


> The additional multiplication with scaler makes that the maximum range go
> down, because we must ensure we never overflow. I'm wondering if we cannot
> pick scaler in such a way that remaining range of cycles is large enough for
> our use case and we can therefore live without bothering for the overflow
> case. What would be "enough"? 1 year? 10 years? ...

Depending on how low we want to keep the error, I don't think we can:

If I set the allowed deviation to 10**-9, we end up requiring a shift by 29
for common ghz ranges. Clearly 33bits isn't an interesting range.

But even if you accept a higher error - we don't have *that* much range
available. Assuming an uint64, the range is ~584 years. If we want 10 years
range, we end up

  math.log(((2**64)-1) / (10 * 365 * 60 * 60 * 24 * 10**9), 2)
  ~= 5.87

So 5 bits available that we could "use" for multiply/shift. For something like
2.5ghz, that'd be ~2% error, clearly not acceptable.  And even just a year of
range, ends up allowing a failure of 30796s = 8min over a year, still too
high.


But I don't think it's really an issue - normally that branch will never be
taken (at least within the memory of the branch predictor), which on modern
CPUs means it'll just be predicted as not taken. So as long as we tell the
compiler what's the likely branch, it should be fine. At least as long as the
branch compares with a hardcoded number.


> > I think it'd be great - but I'm not sure we're there yet, reliability and
> > code-complexity wise.

> Thanks to your commits, the diff of the new patch set will be already much
> smaller and easier to review. What's your biggest concern in terms of
> reliability?

- the restriction just to linux, that'll make testing harder for some, and
  ends up encoding too much OS dependency
- I think we need both the barrier and non-barrier variant, otherwise I
  suspect we'll end up with inccuracies we don't want
- needs lots more documentation about why certain cpuid registers are used
- cpu microarch dependencies - isn't there, e.g., the case that the scale on
  nehalem has to be different than on later architectures?
- lack of facility to evaluate how well the different time sources work


> > I think it might be worth makign the rdts aspect somewhat
> > measurable. E.g. allowing pg_test_timing to use both at the same time, and
> > have it compare elapsed time with both sources of counters.
> I haven't yet looked into pg_test_timing. I'll do that while including your
> patches into the new patch set.

Cool.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-01-23T20:30:06Z

Hi,

On 2023-01-23 18:52:44 +0100, David Geier wrote:
> One thing I was wondering about: why did you chose to use a signed instead
> of an unsigned 64-bit integer for the ticks?

That's been the case since my first post in the thread :). Mainly, it seems
easier to detect underflow cases during subtraction that way. And the factor
of 2 in range doesn't change a whole lot.


> > > If you have time to look at the pg_test_timing part, it'd be
> > > appreciated. That's a it larger, and nobody looked at it yet. So I'm a bit
> > > hesitant to push it.
> > I haven't yet pushed the pg_test_timing (nor it's small prerequisite)
> > patch.
> > 
> > I've attached those two patches. Feel free to include them in your series if
> > you want, then the CF entry (and thus cfbot) makes sense again...
> I'll include them in my new patch set and also have a careful look at them.

Thanks.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-24T13:30:34Z

Hi,

On 1/23/23 18:41, Andres Freund wrote:
> If we add it, it probably shouldn't depend on TIMING, but on
> SUMMARY. Regression test queries showing EXPLAIN ANALYZE output all do
> something like
>    EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
>
> the SUMMARY OFF gets rid of the "top-level" "Planning Time" and "Execution
> Time", whereas the TIMING OFF gets rid of the per-node timing. Those are
> separate options because per-node timing is problematic performance-wise
> (right now), but whole-query timing rarely is.
Makes sense. I wasn't aware of SUMMARY. Let's keep this option in mind, 
in case we'll revisit exposing the clock source in the future.
> Another, independent, thing worth thinking about: I think we might want to
> expose both rdtsc and rdtscp. For something like
> InstrStartNode()/InstrStopNode(), avoiding the "one-way barrier" of rdtscp is
> quite important to avoid changing the query performance. But for measuring
> whole-query time, we likely want to measure the actual time.
>
> It probably won't matter hugely for the whole query time - the out of order
> window of modern CPUs is large, but not *that* large - but I don't think we
> can generally assume that.

That's what I thought as well. I added INSTR_TIME_SET_CURRENT_FAST() and 
for now call that variant from InstrStartNode(), InstrEndNode() and 
pg_test_timing. To do so in InstrEndNode(), I removed 
INSTR_TIME_SET_CURRENT_LAZY(). Otherwise, two variants of that macro 
would be needed. INSTR_TIME_SET_CURRENT_LAZY() was only used in a single 
place and the code is more readable that way. INSTR_TIME_SET_CURRENT() 
is called from a bunch of places. I still have to go through all of them 
and see which should be changed to call the _FAST() variant.

Attached is v7 of the patch:

- Rebased on latest master (most importantly on top of the int64 
instr_time commits). - Includes two commits from Andres which introduce 
INSTR_TIME_SET_SECONDS(), INSTR_TIME_IS_LT() and WIP to report 
pg_test_timing output in nanoseconds. - Converts ticks to nanoseconds 
only with integer math, while accounting for overflow. - Supports RDTSCP 
via INSTR_TIME_SET_CURRENT() and introduced 
INSTR_TIME_SET_CURRENT_FAST() which uses RDTSC.

I haven't gotten to the following:

- Looking through all calls to INSTR_TIME_SET_CURRENT() and check if 
they should be replaced by INSTR_TIME_SET_CURRENT_FAST(). - Reviewing 
Andres commits. Potentially improving on pg_test_timing's output. - 
Looking at enabling RDTSC on more platforms. Is there a minimum set of 
platforms we would like support for? Windows should be easy. That would 
also allow to unify the code a little more. - Add more documentation and 
do more testing around the calls to CPUID. - Profiling and optimizing 
the code. A quick test showed about 10% improvement over master with 
TIMING ON vs TIMING OFF, when using the test-case from Andres' e-mail 
that started this thread.

I hope I'll find time to work on these points during the next days.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-24T13:35:45Z

Hi
>
> I think at least some should be converted to just accumulate in an
> instr_time...
I think that's for a later patch though?
> Yep, at least quite similar.
OK. I coded it up in the latest version of the patch.
> Depending on how low we want to keep the error, I don't think we can:
>
> If I set the allowed deviation to 10**-9, we end up requiring a shift by 29
> for common ghz ranges. Clearly 33bits isn't an interesting range.
>
> But even if you accept a higher error - we don't have *that* much range
> available. Assuming an uint64, the range is ~584 years. If we want 10 years
> range, we end up
>
>    math.log(((2**64)-1) / (10 * 365 * 60 * 60 * 24 * 10**9), 2)
>    ~= 5.87
>
> So 5 bits available that we could "use" for multiply/shift. For something like
> 2.5ghz, that'd be ~2% error, clearly not acceptable.  And even just a year of
> range, ends up allowing a failure of 30796s = 8min over a year, still too
> high.
Thanks for doing the math. Agreed. The latest patch detects overflow and 
correctly handles it.
> But I don't think it's really an issue - normally that branch will never be
> taken (at least within the memory of the branch predictor), which on modern
> CPUs means it'll just be predicted as not taken. So as long as we tell the
> compiler what's the likely branch, it should be fine. At least as long as the
> branch compares with a hardcoded number.
Yeah. The overflow detection just compares two int64. The "overflow 
threshold" is pre-computed.
> - the restriction just to linux, that'll make testing harder for some, and
>    ends up encoding too much OS dependency
> - I think we need both the barrier and non-barrier variant, otherwise I
>    suspect we'll end up with inccuracies we don't want
> - needs lots more documentation about why certain cpuid registers are used
> - cpu microarch dependencies - isn't there, e.g., the case that the scale on
>    nehalem has to be different than on later architectures?
> - lack of facility to evaluate how well the different time sources work
Makes sense. I carried that list over to my latest e-mail which also 
includes the patch to have some sort of summary of where we are in a 
single place.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-01-26T11:21:13Z

Hi,

On 1/23/23 21:30, Andres Freund wrote:
> That's been the case since my first post in the thread :). Mainly, it seems
> easier to detect underflow cases during subtraction that way. And the factor
> of 2 in range doesn't change a whole lot.
I just realized it the other day :).
>>>> If you have time to look at the pg_test_timing part, it'd be
>>>> appreciated. That's a it larger, and nobody looked at it yet. So I'm a bit
>>>> hesitant to push it.
>>> I haven't yet pushed the pg_test_timing (nor it's small prerequisite)
>>> patch.
>>>
>>> I've attached those two patches. Feel free to include them in your series if
>>> you want, then the CF entry (and thus cfbot) makes sense again...
>> I'll include them in my new patch set and also have a careful look at them.

I reviewed the prerequisite patch which introduces 
INSTR_TIME_SET_SECONDS(), as well as the pg_test_timing patch. Here my 
comments:

- The prerequisite patch looks good me.

- By default, the test query in the pg_test_timing doc runs serially. 
What about adding SET max_parallel_workers_per_gather = 0 to make sure 
it really always does (e.g. on a system with different settings for 
parallel_tuple_cost / parallel_setup_cost)? Otherwise, the numbers will 
be much more flaky.

- Why have you added a case distinction for diff == 0? Have you 
encountered this case? If so, how? Maybe add a comment.

- To further reduce overhead we could call INSTR_TIME_SET_CURRENT() 
multiple times. But then again: why do we actually care about the 
per-loop time? Why not instead sum up diff and divide by the number of 
iterations to exclude all the overhead in the first place?

- In the computation of the per-loop time in nanoseconds you can now use 
INSTR_TIME_GET_NANOSEC() instead of INSTR_TIME_GET_DOUBLE() * NS_PER_S.

The rest looks good to me. The rebased patches are part of the patch set 
I sent out yesterday in reply to another mail in this thread.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2023-02-07T18:12:18Z

Hi,

On 2023-01-24 14:30:34 +0100, David Geier wrote:
> Attached is v7 of the patch:
> 
> - Rebased on latest master (most importantly on top of the int64 instr_time
> commits). - Includes two commits from Andres which introduce
> INSTR_TIME_SET_SECONDS(), INSTR_TIME_IS_LT() and WIP to report
> pg_test_timing output in nanoseconds. - Converts ticks to nanoseconds only
> with integer math, while accounting for overflow. - Supports RDTSCP via
> INSTR_TIME_SET_CURRENT() and introduced INSTR_TIME_SET_CURRENT_FAST() which
> uses RDTSC.
> 
> I haven't gotten to the following:
> 
> - Looking through all calls to INSTR_TIME_SET_CURRENT() and check if they
> should be replaced by INSTR_TIME_SET_CURRENT_FAST(). - Reviewing Andres
> commits. Potentially improving on pg_test_timing's output. - Looking at
> enabling RDTSC on more platforms. Is there a minimum set of platforms we
> would like support for? Windows should be easy. That would also allow to
> unify the code a little more. - Add more documentation and do more testing
> around the calls to CPUID. - Profiling and optimizing the code. A quick test
> showed about 10% improvement over master with TIMING ON vs TIMING OFF, when
> using the test-case from Andres' e-mail that started this thread.
> 
> I hope I'll find time to work on these points during the next days.

This fails to build on several platforms:

https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3751

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-02-14T11:11:01Z

Hi,

On 2/7/23 19:12, Andres Freund wrote:
> This fails to build on several platforms:
>
> https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3751

I think I fixed the compilation errors. It was due to a few variables 
being declared under

#if defined(__x86_64__) && defined(__linux__)

while being used also under non x86 Linux.

I also removed again the code to obtain the TSC frequency under 
hypervisors because the TSC is usually emulated and therefore no faster 
than clock_gettime() anyways. So we now simply fallback to 
clock_gettime() on hypervisors when we cannot obtain the frequency via 
leaf 0x16.

Beyond that I reviewed the first two patches a while ago in [1]. I hope 
we can progress with them to further reduce the size of this patch set.

[1] 
https://www.postgresql.org/message-id/3ac157f7-085d-e071-45fc-b87cd306360c%40gmail.com 

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-02-14T12:48:56Z

Hi!

On 2/14/23 12:11, David Geier wrote:
> Hi,
>
> I think I fixed the compilation errors. It was due to a few variables 
> being declared under
>
> #if defined(__x86_64__) && defined(__linux__)
>
> while being used also under non x86 Linux.
>
> I also removed again the code to obtain the TSC frequency under 
> hypervisors because the TSC is usually emulated and therefore no 
> faster than clock_gettime() anyways. So we now simply fallback to 
> clock_gettime() on hypervisors when we cannot obtain the frequency via 
> leaf 0x16.
>
> Beyond that I reviewed the first two patches a while ago in [1]. I 
> hope we can progress with them to further reduce the size of this 
> patch set.
>
> [1] 
> https://www.postgresql.org/message-id/3ac157f7-085d-e071-45fc-b87cd306360c%40gmail.com 
>
>
It still fails.

I'll get Cirrus-CI working on my own Github fork so I can make sure it 
really compiles on all platforms before I submit a new version.

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2023-02-20T10:36:32Z

Hi!

On 2/14/23 13:48, David Geier wrote:
>
> It still fails.
>
> I'll get Cirrus-CI working on my own Github fork so I can make sure it 
> really compiles on all platforms before I submit a new version.

It took some time until Cirrus CI allowed me to run tests against my new 
GitHub account (there's a 3 days freeze to avoid people from getting 
Cirrus CI nodes to mine bitcoins :-D). Attached now the latest patch 
which passes builds, rebased on latest master.

I also reviewed the first two patches a while ago in [1]. I hope we can 
progress with them to further reduce the size of this patch set.

Beyond that: I could work on support for more OSs (e.g. starting with 
Windows). Is there appetite for that or do we rather want to instead 
start with a smaller patch?

[1] 
https://www.postgresql.org/message-id/3ac157f7-085d-e071-45fc-b87cd306360c%40gmail.com

-- 
David Geier
(ServiceNow)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

vignesh C <vignesh21@gmail.com> — 2024-01-20T03:33:47Z

On Mon, 20 Feb 2023 at 16:06, David Geier <geidav.pg@gmail.com> wrote:
>
> Hi!
>
> On 2/14/23 13:48, David Geier wrote:
> >
> > It still fails.
> >
> > I'll get Cirrus-CI working on my own Github fork so I can make sure it
> > really compiles on all platforms before I submit a new version.
>
> It took some time until Cirrus CI allowed me to run tests against my new
> GitHub account (there's a 3 days freeze to avoid people from getting
> Cirrus CI nodes to mine bitcoins :-D). Attached now the latest patch
> which passes builds, rebased on latest master.
>
> I also reviewed the first two patches a while ago in [1]. I hope we can
> progress with them to further reduce the size of this patch set.
>
> Beyond that: I could work on support for more OSs (e.g. starting with
> Windows). Is there appetite for that or do we rather want to instead
> start with a smaller patch?

Are we planning to continue on this and take it further?
I'm seeing that there has been no activity in this thread for nearly 1
year now, I'm planning to close this in the current commitfest unless
someone is planning to take it forward.

Regards,
Vignesh

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

vignesh C <vignesh21@gmail.com> — 2024-02-01T18:44:05Z

On Sat, 20 Jan 2024 at 09:03, vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 20 Feb 2023 at 16:06, David Geier <geidav.pg@gmail.com> wrote:
> >
> > Hi!
> >
> > On 2/14/23 13:48, David Geier wrote:
> > >
> > > It still fails.
> > >
> > > I'll get Cirrus-CI working on my own Github fork so I can make sure it
> > > really compiles on all platforms before I submit a new version.
> >
> > It took some time until Cirrus CI allowed me to run tests against my new
> > GitHub account (there's a 3 days freeze to avoid people from getting
> > Cirrus CI nodes to mine bitcoins :-D). Attached now the latest patch
> > which passes builds, rebased on latest master.
> >
> > I also reviewed the first two patches a while ago in [1]. I hope we can
> > progress with them to further reduce the size of this patch set.
> >
> > Beyond that: I could work on support for more OSs (e.g. starting with
> > Windows). Is there appetite for that or do we rather want to instead
> > start with a smaller patch?
>
> Are we planning to continue on this and take it further?
> I'm seeing that there has been no activity in this thread for nearly 1
> year now, I'm planning to close this in the current commitfest unless
> someone is planning to take it forward.

Since the author or no one else showed interest in taking it forward
and the patch had no activity for more than 1 year, I have changed the
status to RWF. Feel free to add a new CF entry when someone is
planning to resume work more actively.

Regards,
Vignesh

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2024-06-01T18:52:00Z

Hi,

At some point this patch switched from rdtsc to rdtscp, which imo largely
negates the point of it. What lead to that?

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2025-03-01T07:45:58Z

On Sun, Jun 2, 2024 at 1:08 AM Andres Freund <andres@anarazel.de> wrote:

> At some point this patch switched from rdtsc to rdtscp, which imo largely
> negates the point of it. What lead to that?


From what I can gather, it appears this was an oversight when David first
reapplied the work on the instr_time changes that were committed.

I've come back to this and rebased this, as well as:
- Corrected the use of RDTSCP to RDTSC in pg_get_ticks_fast
- Check 16H register if 15H register does not contain frequency information
(per research, relevant for some CPUs)
- Fixed incorrect reporting in pg_test_timing due to too small histogram
(32 => 64 bits)
- Fixed indentation per pgindent
- Added support for VMs running under KVM/VMware Hypervisors

On that last item, this does indeed make a difference on VMs, contrary to
the code comment in earlier versions (and I've not seen any odd behaviors
again, FWIW):

On a c5.xlarge (Skylake-SP or Cascade Lake) on AWS, with the same test as
done initially in this thread:

SELECT COUNT(*) FROM lotsarows;
Time: 974.423 ms

EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM lotsarows;
Time: 1336.196 ms (00:01.336)

Without patch:
EXPLAIN (ANALYZE) SELECT COUNT(*) FROM lotsarows;
Time: 2165.069 ms (00:02.165)

Per loop time including overhead: 22.15 ns

With patch:
EXPLAIN (ANALYZE, TIMING ON) SELECT COUNT(*) FROM lotsarows;
Time: 1654.289 ms (00:01.654)

Per loop time including overhead: 9.81 ns

I'm registering this again in the current commitfest to help reviews.

Open questions I have:
- Could we rely on checking whether the TSC timesource is invariant (via
CPUID), instead of relying on Linux choosing it as a clocksource?
- For the Hypervisor CPUID checks I had to rely on __cpuidex which is only
available on newer GCC versions (
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95973), how do we best check
for its presence? (compiler version, or rather configure check?) -- note
this is also the reason the patch fails the clang compiler warning check in
CI, despite clang having support in recent versions (
https://reviews.llvm.org/D121653)

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2025-07-17T00:48:00Z

Hi,

On 2025-02-28 23:45:58 -0800, Lukas Fittl wrote:
> From what I can gather, it appears this was an oversight when David first
> reapplied the work on the instr_time changes that were committed.

Heh, glad that that's now fixed.  Unfortunately the patch needs an update,
primarily because of the recent pg_test_timing changes.

Applying just patch 2 results in a performance *regression* in pg_test_timing
on my machine, which is due to always hitting the unlikely() path in
INSTR_TIME_GET_NANOSEC() when INSTR_TIME_GET_NANOSEC() is used for an
"absolute" timestamp, rather than a differential timestamp. Which in turn
means hitting a division instruction every time, which on my slightly older
hardware is slower.  That may be because my workstation has been up for 40
days, but clearly that can't lead us down to the slow-path

> On a c5.xlarge (Skylake-SP or Cascade Lake) on AWS, with the same test as
> done initially in this thread:
> 
> SELECT COUNT(*) FROM lotsarows;
> Time: 974.423 ms
> 
> EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM lotsarows;
> Time: 1336.196 ms (00:01.336)
> 
> Without patch:
> EXPLAIN (ANALYZE) SELECT COUNT(*) FROM lotsarows;
> Time: 2165.069 ms (00:02.165)
> 
> Per loop time including overhead: 22.15 ns
> 
> With patch:
> EXPLAIN (ANALYZE, TIMING ON) SELECT COUNT(*) FROM lotsarows;
> Time: 1654.289 ms (00:01.654)
> 
> Per loop time including overhead: 9.81 ns

I still think this would be a rather awesome improvement.

> Open questions I have:
> - Could we rely on checking whether the TSC timesource is invariant (via
> CPUID), instead of relying on Linux choosing it as a clocksource?

I don't see why not?

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2025-07-27T19:50:54Z

Hi,

See attached v11 (and moved to the PG19-2 commitfest), split into a new set
of patches:

0001 - Improve the __cpuidex check added for a different purpose
in 792752af4eb5 to:

- Fix a typo (configure was incorrectly checking for "__get_cpuidex", vs
meson.build was doing it correctly)
- Adds support for non-MSVC compilers as well (e.g. GCC 11+), where
__cpuidex is defined in cpuid.h, not intrin.h

This change should be independently committable, though we wouldn't use
cpuidex on non-MSVC compilers today in practice, I believe.

0002 - The core patch rebased, which, as before:

- Adds INSTR_TIME_SET_CURRENT_FAST (which uses RDTSC if available) and uses
it for InstrStartNode/InstrStopNode
- Changes INSTR_TIME_SET_CURRENT to directly use RDTSCP if available
(instead of pg_clock_gettime)
- Keeps utilizing pg_clock_gettime for both unless we're on Linux x86 and
the clocksource is set to "tsc" (see note below re: that aspect)

0003 - Changes to pg_test_timing utility:

- Show the used time source (clock_gettime + clock type / RDTSC / RDTSCP)
- Allows checking the latency of the "fast" time source (RDTSC) with the
new "--fast" option, and warn if its not available
- Avoids the INSTR_TIME_GET_NANOSEC slowness that Andres reported by
diffing the ticks first and then calculating nanosecs

Note the other pg_test_timing changes regarding nanoseconds should all have
been addressed by 0b096e379e6f I believe.

On Wed, Jul 16, 2025 at 5:48 PM Andres Freund <andres@anarazel.de> wrote:

> Applying just patch 2 results in a performance *regression* in
> pg_test_timing
> on my machine, which is due to always hitting the unlikely() path in
> INSTR_TIME_GET_NANOSEC() when INSTR_TIME_GET_NANOSEC() is used for an
> "absolute" timestamp, rather than a differential timestamp. Which in turn
> means hitting a division instruction every time, which on my slightly older
> hardware is slower.  That may be because my workstation has been up for 40
> days, but clearly that can't lead us down to the slow-path
>

Assuming you didn't restart your workstation, can you retest with this
patch set?

I believe the pg_test_timing changes should address this problem, by
avoiding calculations with the absolute (very large) ticks value.

> Open questions I have:
> > - Could we rely on checking whether the TSC timesource is invariant (via
> > CPUID), instead of relying on Linux choosing it as a clocksource?
>
> I don't see why not?

Thinking this through again, my worry would be that our detection logic for
whether the TSC is safe to use directly, is much less sophisticated than
that of the Linux Kernel - and the Linux Kernel also allows configuring the
clock source explicitly, if the detection goes wrong.

For example, David had previously brought up the worry that accessing the
TSC directly in a VM can be very slow when the TSC is emulated. The Linux
Kernel indeed has checks for this, e.g. in the context of Xen:
https://github.com/torvalds/linux/blob/b711733e89a3f84c8e1e56e2328f9a0fa5facc7c/arch/x86/xen/time.c#L490

Maybe introducing a GUC for this is the way to go, with an OS-dependent
"auto" setting?

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Michael Paquier <michael@paquier.xyz> — 2025-07-28T06:38:20Z

On Sun, Jul 27, 2025 at 12:50:54PM -0700, Lukas Fittl wrote:
> 0001 - Improve the __cpuidex check added for a different purpose
> in 792752af4eb5 to:
> 
> - Fix a typo (configure was incorrectly checking for "__get_cpuidex", vs
> meson.build was doing it correctly)

It seems to me that this is an independent issue that had better be
backpatched down to where this configure check has been introduced,
no?

> - Adds support for non-MSVC compilers as well (e.g. GCC 11+), where
> __cpuidex is defined in cpuid.h, not intrin.h

This one should be a HEAD-only change.
--
Michael

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Michael Paquier <michael@paquier.xyz> — 2025-07-29T02:30:11Z

On Mon, Jul 28, 2025 at 03:38:20PM +0900, Michael Paquier wrote:
> On Sun, Jul 27, 2025 at 12:50:54PM -0700, Lukas Fittl wrote:
>> - Fix a typo (configure was incorrectly checking for "__get_cpuidex", vs
>> meson.build was doing it correctly)
> 
> It seems to me that this is an independent issue that had better be
> backpatched down to where this configure check has been introduced,
> no?

Please note that updates of ./configure should never be manual, these 
are done as follows:
- Update of ./autoconf.ac
- run of autoreconf -i or equivalent to update ./configure.  (I just
use the former, just my no-brainer to handle things.  Committers are
usually responsible for that, but it may matter to keep the CI happy.)

And I have noticed a second inconsistency with __cpuid(), both
introduced by 3dc2d62d0486 as far as I can see, so we have never
checked for these routines.  This is an independent issue for
something that should be backpatched, so I've spawned a new thread
(don't worry you have author credits):
https://www.postgresql.org/message-id/aIgwNYGVt5aRAqTJ@paquier.xyz
--
Michael

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2025-09-01T10:36:24Z

Hi Lukas!

On 01.03.2025 08:45, Lukas Fittl wrote:
> On Sun, Jun 2, 2024 at 1:08 AM Andres Freund <andres@anarazel.de> wrote:
> 
>> At some point this patch switched from rdtsc to rdtscp, which imo largely
>> negates the point of it. What lead to that?
> 
> 
> From what I can gather, it appears this was an oversight when David first
> reapplied the work on the instr_time changes that were committed.

Yes, that was by accident.

> 
> I've come back to this and rebased this, as well as:

Thanks for moving this forward.

> - Added support for VMs running under KVM/VMware Hypervisors
> 
> On that last item, this does indeed make a difference on VMs, contrary to
> the code comment in earlier versions (and I've not seen any odd behaviors
> again, FWIW):

How can we be sure we've actually covered all hypervisors? How much
coverage do we have in the build farm? Are we good if passes on all
build animals?

> 
> On a c5.xlarge (Skylake-SP or Cascade Lake) on AWS, with the same test as
> done initially in this thread:
> 
> SELECT COUNT(*) FROM lotsarows;
> Time: 974.423 ms
> 
> EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM lotsarows;
> Time: 1336.196 ms (00:01.336)
> 
> Without patch:
> EXPLAIN (ANALYZE) SELECT COUNT(*) FROM lotsarows;
> Time: 2165.069 ms (00:02.165)
> 
> Per loop time including overhead: 22.15 ns
> 
> With patch:
> EXPLAIN (ANALYZE, TIMING ON) SELECT COUNT(*) FROM lotsarows;
> Time: 1654.289 ms (00:01.654)
> 
> Per loop time including overhead: 9.81 ns
> 
> I'm registering this again in the current commitfest to help reviews.
> 
> Open questions I have:
> - Could we rely on checking whether the TSC timesource is invariant (via
> CPUID), instead of relying on Linux choosing it as a clocksource?

Why do you want to do that? Are you concerned that Linux might pick a
different clock source even though invariant TSC is available?

We could code our own check but looking at the Linux kernel code, this
is a bit more involved if we want to do it completely right. They check
e.g. if the TSC is also synchronized across different CPUs, which is not
the case if they're on different chassis (see unsynchronized_tsc() ->
apic_is_clustered_box()).

I think it's safer to start with relying on the kernel. Some research
suggests that the TSC is the preferred clock source if available.

> - For the Hypervisor CPUID checks I had to rely on __cpuidex which is only
> available on newer GCC versions (
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95973), how do we best check
> for its presence? (compiler version, or rather configure check?) -- note
> this is also the reason the patch fails the clang compiler warning check in
> CI, despite clang having support in recent versions (
> https://reviews.llvm.org/D121653)

What about instead using #if !__has_builtin(_cpuidex) ... #endif to
define the built-in ourselves as a function in case it doesn't exist?

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Álvaro Herrera <alvherre@kurilemu.de> — 2025-10-19T18:16:20Z

On 2025-Jul-27, Lukas Fittl wrote:

> See attached v11 (and moved to the PG19-2 commitfest), split into a new set
> of patches:

I rebased (but not reviewed) this patchset now that Michael committed
part of 0001, as seen in another thread.

Quickly looking at 0003, I wonder if adding a separate --fast switch to
pg_test_timing is really what we want.  Why not report both the fast and
legacy measurements in platforms that support both, instead?  If I were
a consultant trying to understand a customer's system, I would have to
ask them to run it twice just in case 'fast' is supported, and I don't
think that's very helpful.  Also, were the doc updates lost somehow, or
were they made irrelevant by other concurrent pg_test_timing
development?

Thanks

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"Ninguna manada de bestias tiene una voz tan horrible como la humana" (Orual)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Robert Haas <robertmhaas@gmail.com> — 2025-10-20T19:59:51Z

On Sun, Oct 19, 2025 at 2:16 PM Álvaro Herrera <alvherre@kurilemu.de> wrote:
> If I were
> a consultant trying to understand a customer's system, I would have to
> ask them to run it twice just in case 'fast' is supported, and I don't
> think that's very helpful.

Big +1 from me.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2025-10-22T13:32:03Z

Hi,

On 2025-09-01 12:36:24 +0200, David Geier wrote:
> > Open questions I have:
> > - Could we rely on checking whether the TSC timesource is invariant (via
> > CPUID), instead of relying on Linux choosing it as a clocksource?
> 
> Why do you want to do that? Are you concerned that Linux might pick a
> different clock source even though invariant TSC is available?

Not sure about Lukas, but I'm slightly concerned about making this a linux
specific mechanism unnecessarily.


> We could code our own check but looking at the Linux kernel code, this
> is a bit more involved if we want to do it completely right. They check
> e.g. if the TSC is also synchronized across different CPUs, which is not
> the case if they're on different chassis (see unsynchronized_tsc() ->
> apic_is_clustered_box()).

I think Linux has higher fidelity requirements than our instrumentation usage
- with linux an inaccurate clock would lead to broken timers, wrong wall clock
etc, whereas for us it's just a skewed instrumentation result.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2025-11-19T07:20:25Z

On 20.10.2025 21:59, Robert Haas wrote:
> On Sun, Oct 19, 2025 at 2:16 PM Álvaro Herrera <alvherre@kurilemu.de> wrote:
>> If I were
>> a consultant trying to understand a customer's system, I would have to
>> ask them to run it twice just in case 'fast' is supported, and I don't
>> think that's very helpful.
> 
> Big +1 from me.
> 

That makes sense. I'm planning to rebase the patch the next days. Then
I'll also take care of that.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2025-11-19T07:25:58Z

On 22.10.2025 15:32, Andres Freund wrote:
> Hi,
> 
> On 2025-09-01 12:36:24 +0200, David Geier wrote:
>>> Open questions I have:
>>> - Could we rely on checking whether the TSC timesource is invariant (via
>>> CPUID), instead of relying on Linux choosing it as a clocksource?
>>
>> Why do you want to do that? Are you concerned that Linux might pick a
>> different clock source even though invariant TSC is available?
> 
> Not sure about Lukas, but I'm slightly concerned about making this a linux
> specific mechanism unnecessarily.
> 

Considering [1], Lukas seems to share my concerns that building or own
has the risk of missing cases.


> 
>> We could code our own check but looking at the Linux kernel code, this
>> is a bit more involved if we want to do it completely right. They check
>> e.g. if the TSC is also synchronized across different CPUs, which is not
>> the case if they're on different chassis (see unsynchronized_tsc() ->
>> apic_is_clustered_box()).
> 
> I think Linux has higher fidelity requirements than our instrumentation usage
> - with linux an inaccurate clock would lead to broken timers, wrong wall clock
> etc, whereas for us it's just a skewed instrumentation result.

That's true. As long as we use the RDTSCP basd code only in places where
it doesn't affect "correctness" it's not the end of the world if they're
skewed.

I'll give it a try to code our own detection mechanism and will share
findings here. Then we can make a call based on the learnings.

--
David Geier

[1]
https://www.postgresql.org/message-id/CAP53Pky-BN0Ui%2BA9no3TsU%3DGoMTFpxYSWYtp_LVaDH%3Dy69BxNg%40mail.gmail.com

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2025-11-19T16:54:20Z

On Tue, Nov 18, 2025 at 11:26 PM David Geier <geidav.pg@gmail.com> wrote:

> On 22.10.2025 15:32, Andres Freund wrote:
> > On 2025-09-01 12:36:24 +0200, David Geier wrote:
> >>> Open questions I have:
> >>> - Could we rely on checking whether the TSC timesource is invariant
> (via
> >>> CPUID), instead of relying on Linux choosing it as a clocksource?
> >>
> >> Why do you want to do that? Are you concerned that Linux might pick a
> >> different clock source even though invariant TSC is available?
> >
> > Not sure about Lukas, but I'm slightly concerned about making this a
> linux
> > specific mechanism unnecessarily.
> >
>
> Considering [1], Lukas seems to share my concerns that building or own
> has the risk of missing cases.
>

I had an off-list discussion with Andres about this at PGConf.EU, and one
idea that was floated is that we could keep the Linux specific mechanism
when on Linux, but not do this check on other platforms, as to not affect
portability.


> >> We could code our own check but looking at the Linux kernel code, this
> >> is a bit more involved if we want to do it completely right. They check
> >> e.g. if the TSC is also synchronized across different CPUs, which is not
> >> the case if they're on different chassis (see unsynchronized_tsc() ->
> >> apic_is_clustered_box()).
> >
> > I think Linux has higher fidelity requirements than our instrumentation
> usage
> > - with linux an inaccurate clock would lead to broken timers, wrong wall
> clock
> > etc, whereas for us it's just a skewed instrumentation result.
>
> That's true. As long as we use the RDTSCP basd code only in places where
> it doesn't affect "correctness" it's not the end of the world if they're
> skewed.
>

I think my general worry here is that we basically give the user no escape
hatch - you might end up with a case where Postgres gives you unusable
EXPLAIN timings and you can't do anything to fix that.

Overall, I'm still thinking a GUC might be the way to go, but I don't think
anyone else was enthusiastic about that idea :)

Thanks for working on an updated patch!

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Robert Haas <robertmhaas@gmail.com> — 2025-11-19T19:36:53Z

On Wed, Nov 19, 2025 at 11:55 AM Lukas Fittl <lukas@fittl.com> wrote:
> Overall, I'm still thinking a GUC might be the way to go, but I don't think anyone else was enthusiastic about that idea :)

Reliable feature auto-detection is the best option, but if that's not
possible, I think the choices are add a GUC or give up on the project
altogether. Using a GUC to deal with platform dependencies is a pretty
reasonable concept -- see, e.g. dynamic_shared_memory_type or
huge_pages or io_method. If we can't autodetect it reliably and we
aren't willing to add a GUC, we're basically saying there's not enough
value here to justify adding a configuration parameter. That's often a
totally reasonable conclusion -- it can easily happen that the
benefits of a platform-specific optimization are too small to make it
worth configuring. But I would have thought that in this case the
benefits might be quite large.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2025-12-03T09:50:15Z

On 19.11.2025 08:20, David Geier wrote:
> 
> On 20.10.2025 21:59, Robert Haas wrote:
>> On Sun, Oct 19, 2025 at 2:16 PM Álvaro Herrera <alvherre@kurilemu.de> wrote:
>>> If I were
>>> a consultant trying to understand a customer's system, I would have to
>>> ask them to run it twice just in case 'fast' is supported, and I don't
>>> think that's very helpful.
>>
>> Big +1 from me.
>>
> 
> That makes sense. I'm planning to rebase the patch the next days. Then
> I'll also take care of that.

The attached patched is rebased on latest master and pg_test_timing now
always tests the normal and the fast timing code. If no fast clock
source is available the fast timing code is skipped.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2025-12-03T11:03:25Z

On 19.11.2025 20:36, Robert Haas wrote:
> On Wed, Nov 19, 2025 at 11:55 AM Lukas Fittl <lukas@fittl.com> wrote:
>> Overall, I'm still thinking a GUC might be the way to go, but I don't think anyone else was enthusiastic about that idea :)
> 
> Reliable feature auto-detection is the best option, but if that's not
> possible, I think the choices are add a GUC or give up on the project
> altogether. Using a GUC to deal with platform dependencies is a pretty
> reasonable concept -- see, e.g. dynamic_shared_memory_type or
> huge_pages or io_method. If we can't autodetect it reliably and we
> aren't willing to add a GUC, we're basically saying there's not enough
> value here to justify adding a configuration parameter. That's often a
> totally reasonable conclusion -- it can easily happen that the
> benefits of a platform-specific optimization are too small to make it
> worth configuring. But I would have thought that in this case the
> benefits might be quite large.

I'm also in favor of adding a GUC. Even if we could 100% reliably detect
if using TSC is giving correct results, it could be that it's slow in
some virtualized environment and hence the user wants to disable it.

I'm wondering how to best do a GUC for something that is potentially
unavailable on the system. In that case the GUC would be superfluous.
Maybe a boolean "enable_try_fast_clocksource" GUC or a "clocksource"
enum GUC which can be "default" and "try_rdtsc", where we only include
the "try_rdtsc" enum value on x86 systems?

Any other ideas?

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Robert Haas <robertmhaas@gmail.com> — 2025-12-03T14:15:00Z

On Wed, Dec 3, 2025 at 6:03 AM David Geier <geidav.pg@gmail.com> wrote:
> I'm wondering how to best do a GUC for something that is potentially
> unavailable on the system. In that case the GUC would be superfluous.
> Maybe a boolean "enable_try_fast_clocksource" GUC or a "clocksource"
> enum GUC which can be "default" and "try_rdtsc", where we only include
> the "try_rdtsc" enum value on x86 systems?

huge_pages=on/off/try is one possible precedent. Perhaps for this
case, we might do something like
clock_source=auto/this/that/the_other_thing might be better. If you
set it to any value other than "auto", the named source must be
available, or startup fails. If you set it to auto, it picks what it
believes to be the best option available, and there is some other
read-only GUC (akin to huge_page_status) that tells you what it
picked.

I'm open to other suggestions as to how this should work, but (1) all
of the existing enable_* GUCs are planner GUCs, and (2) I suspect it's
short-sighted to plan for only "fast" and "not fast".

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2025-12-04T08:07:20Z

On 03.12.2025 10:50, David Geier wrote:
> On 19.11.2025 08:20, David Geier wrote:
>>
>> On 20.10.2025 21:59, Robert Haas wrote:
>>> On Sun, Oct 19, 2025 at 2:16 PM Álvaro Herrera <alvherre@kurilemu.de> wrote:
>>>> If I were
>>>> a consultant trying to understand a customer's system, I would have to
>>>> ask them to run it twice just in case 'fast' is supported, and I don't
>>>> think that's very helpful.
>>>
>>> Big +1 from me.
>>>
>>
>> That makes sense. I'm planning to rebase the patch the next days. Then
>> I'll also take care of that.
> 
> The attached patched is rebased on latest master and pg_test_timing now
> always tests the normal and the fast timing code. If no fast clock
> source is available the fast timing code is skipped.
The last patch I sent was incomplete because I had missed committing my
changes. Attached is now the patch with the changes to pg_test_timing.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Hannu Krosing <hannuk@google.com> — 2025-12-04T21:21:29Z

I have not looked at this patch series yet, but when I played around
with using rdtsc (or actually some gcc/clang construct that compiled
to rctsc on c86 and into time register reads on Arm and risc-v) then
any extra step around it had noticeable overhead. I am not sure
putting some if or function call around rdtsc call is a good idea.

On Wed, Dec 3, 2025 at 3:15 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Wed, Dec 3, 2025 at 6:03 AM David Geier <geidav.pg@gmail.com> wrote:
> > I'm wondering how to best do a GUC for something that is potentially
> > unavailable on the system. In that case the GUC would be superfluous.
> > Maybe a boolean "enable_try_fast_clocksource" GUC or a "clocksource"
> > enum GUC which can be "default" and "try_rdtsc", where we only include
> > the "try_rdtsc" enum value on x86 systems?
>
> huge_pages=on/off/try is one possible precedent. Perhaps for this
> case, we might do something like
> clock_source=auto/this/that/the_other_thing might be better. If you
> set it to any value other than "auto", the named source must be
> available, or startup fails. If you set it to auto, it picks what it
> believes to be the best option available, and there is some other
> read-only GUC (akin to huge_page_status) that tells you what it
> picked.
>
> I'm open to other suggestions as to how this should work, but (1) all
> of the existing enable_* GUCs are planner GUCs, and (2) I suspect it's
> short-sighted to plan for only "fast" and "not fast".
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>
>

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2025-12-05T07:46:38Z

On 04.12.2025 22:21, Hannu Krosing wrote:
> I have not looked at this patch series yet, but when I played around
> with using rdtsc (or actually some gcc/clang construct that compiled
> to rctsc on c86 and into time register reads on Arm and risc-v) then
> any extra step around it had noticeable overhead. I am not sure
> putting some if or function call around rdtsc call is a good idea.

We have that already. INSTR_TIME_SET_CURRENT_FAST() is currently
implemented as:

static inline instr_time
pg_get_ticks_fast(void)
{
#if defined(__x86_64__) && defined(__linux__)
	if (has_rdtsc)
	{
		instr_time	now;

		now.ticks = __rdtsc();
		return now;
	}
#endif

	return pg_clock_gettime();
}

Based on Robert's suggestion I wanted to add a "fast_clock_source" enum
GUC which can have the following values "auto", "rdtsc", "try_rdtsc" and
"off". With that, at least no additional checks are needed and
performance will remain as previously benchmarked in this thread.

Beyond that, the condition will always evaluate to the same result, so
there won't be branch mispredictions. Doing completely without any check
is impossible, except if we were to JIT compile InstrStartNode() and
InstrStopNode(). But that's a much bigger project.

I'll still add unlikely() around the if (has_rdtsc).

Any input regarding the proposed GUC is welcome.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2026-01-11T19:26:17Z

> Based on Robert's suggestion I wanted to add a "fast_clock_source" enum
> GUC which can have the following values "auto", "rdtsc", "try_rdtsc" and
> "off". With that, at least no additional checks are needed and
> performance will remain as previously benchmarked in this thread.

The attached patch set is rebased on latest master and contains a commit
which adds a "fast_clock_source" GUC that can be "try", "off" and
"rdtsc" on Linux.

Alternatively, we could call the GUC "clock_source" with "auto",
"clock_gettime" and "rdtsc". Opinions?

I moved the call to INSTR_TIME_INITIALIZE() from InitPostgres() to
PostmasterMain(). In InitPostgres() it kept the database in a recovery
cycle.

> I'll still add unlikely() around the if (has_rdtsc).

Done.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-01-31T20:11:33Z

On Sun, Jan 11, 2026 at 11:26 AM David Geier <geidav.pg@gmail.com> wrote:
>
> > Based on Robert's suggestion I wanted to add a "fast_clock_source" enum
> > GUC which can have the following values "auto", "rdtsc", "try_rdtsc" and
> > "off". With that, at least no additional checks are needed and
> > performance will remain as previously benchmarked in this thread.
>
> The attached patch set is rebased on latest master and contains a commit
> which adds a "fast_clock_source" GUC that can be "try", "off" and
> "rdtsc" on Linux.
>
> Alternatively, we could call the GUC "clock_source" with "auto",
> "clock_gettime" and "rdtsc". Opinions?

No strong opinion on the GUC name ("fast_clock_source" seems fine?),
but I think "try" is a bit confusing if our logic is more than just
checking if the RDTSC(P) instruction is available, so I'd be in favor
of "auto" as the default value.

> I moved the call to INSTR_TIME_INITIALIZE() from InitPostgres() to
> PostmasterMain(). In InitPostgres() it kept the database in a recovery
> cycle.

I think we can actually avoid having anything in PostmasterMain (or
InitPostgres), and instead rely on the GUC assign mechanism.

I've reworked the patch a bit more, see attached v4, with a couple of
noticeable changes:

In regards to the GUC:
- Use the GUC check mechanism to complain if RDTSC clock source is
requested, but its not available
- Use the GUC assign mechanism to set whether we're actually using the
RDTSC clock source
- "auto" now means that we use RDTSC clock source by default if we're
on Linux x86, and the system clocksource is "tsc"
- "rdtsc" now allows using RDTSC on any x86-based Unix-like systems (I
see no reason to restrict the BSDs from using RDTSC when setting it
explicitly)
- Allow changing the clock source GUC at any time, without requiring a
restart (it makes testing much easier, and I don't see a good reason
to require a restart, or even restrict this to superuser?)
- Have pg_test_timing emit whether a fast clock source will be used by
default (or whether one needs to change the GUC)

Additionally:
- If a client program wants to use the fast clock source (like
pg_test_timing does), it first needs to call
pg_initialize_fast_clock_source() -- this replaces the
INSTR_TIME_INITIALIZE calls.
- I've re-introduced a patch (0001) to set HAVE__CPUIDEX on modern
GCC/clang. That's necessary to make this work on VM Hypervisors (per
the patch's commit message)
- I've merged the GUC patch together with the patch that adds the
RDTSC implementation (0002), I don't think that makes sense to review
or commit separately.
- I've unified the RDTSC and RDTSCP handling, so we require both in
order to use TSC as a time source. Because we have the shared
pg_ticks_to_ns() function that gets used on an instr_time regardless
of fast vs "slow" timing, and the variables used in that function are
affected by the RDTSC availability, we must use TSC consistently - I
don't think we can mix RDTSC for fast and pg_clock_gettime() for slow,
as this patch series has done so far.

Open questions for me:
- I'm seeing a CI test failure for "Linux - Debian Trixie - Meson"
(times out), but its not clear if this is a fluke - I'll check if this
recurs on the commitfest patch
- We're doing a lot of work in pg_ticks_to_ns, even when we're not
using RDTSC - and I think that shows in a slightly slower
pg_test_timing measurement compared to master when fast clock source
is off. Can we somehow only do that when we use RDTSC?

Here is a fresh test run with this patch on an AWS c6i.xlarge, i.e.
Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz / "Ice Lake":

CREATE TABLE test (id int);
INSERT INTO test SELECT * FROM generate_series(0, 1000000);

postgres=# SET fast_clock_source = off;
SET
Time: 0.107 ms
postgres=# EXPLAIN ANALYZE SELECT COUNT(*) FROM test;
                                                                QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------
 Finalize Aggregate  (cost=10633.55..10633.56 rows=1 width=8) (actual
time=44.117..44.811 rows=1.00 loops=1)
   Buffers: shared hit=846 read=3579
   ->  Gather  (cost=10633.34..10633.55 rows=2 width=8) (actual
time=44.060..44.804 rows=3.00 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=846 read=3579
         ->  Partial Aggregate  (cost=9633.34..9633.35 rows=1 width=8)
(actual time=42.129..42.130 rows=1.00 loops=3)
               Buffers: shared hit=846 read=3579
               ->  Parallel Seq Scan on test  (cost=0.00..8591.67
rows=416667 width=0) (actual time=0.086..21.595 rows=333333.67
loops=3)
                     Buffers: shared hit=846 read=3579
 Planning Time: 0.043 ms
 Execution Time: 44.836 ms
(12 rows)

Time: 45.076 ms

postgres=# SET fast_clock_source = rdtsc;
SET
Time: 0.123 ms
postgres=# EXPLAIN ANALYZE SELECT COUNT(*) FROM test;
                                                                QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------
 Finalize Aggregate  (cost=10633.55..10633.56 rows=1 width=8) (actual
time=32.943..33.912 rows=1.00 loops=1)
   Buffers: shared hit=1128 read=3297
   ->  Gather  (cost=10633.34..10633.55 rows=2 width=8) (actual
time=32.868..33.906 rows=3.00 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=1128 read=3297
         ->  Partial Aggregate  (cost=9633.34..9633.35 rows=1 width=8)
(actual time=30.705..30.706 rows=1.00 loops=3)
               Buffers: shared hit=1128 read=3297
               ->  Parallel Seq Scan on test  (cost=0.00..8591.67
rows=416667 width=0) (actual time=0.080..15.223 rows=333333.67
loops=3)
                     Buffers: shared hit=1128 read=3297
 Planning Time: 0.042 ms
 Execution Time: 33.935 ms
(12 rows)

Time: 34.180 ms

postgres=# EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM test;
                                                      QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
 Finalize Aggregate  (cost=10633.55..10633.56 rows=1 width=8) (actual
rows=1.00 loops=1)
   Buffers: shared hit=1410 read=3015
   ->  Gather  (cost=10633.34..10633.55 rows=2 width=8) (actual
rows=3.00 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=1410 read=3015
         ->  Partial Aggregate  (cost=9633.34..9633.35 rows=1 width=8)
(actual rows=1.00 loops=3)
               Buffers: shared hit=1410 read=3015
               ->  Parallel Seq Scan on test  (cost=0.00..8591.67
rows=416667 width=0) (actual rows=333333.67 loops=3)
                     Buffers: shared hit=1410 read=3015
 Planning Time: 0.042 ms
 Execution Time: 27.876 ms
(12 rows)

Time: 28.135 ms

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-02-01T03:14:56Z

On Sat, Jan 31, 2026 at 12:11 PM Lukas Fittl <lukas@fittl.com> wrote:
> I've reworked the patch a bit more, see attached v4

And of course, I took the wrong branch when running "git format-patch"
- apologies.

See attached v5.


Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-02-02T09:22:37Z

Hi Lukas,

On Sun, Feb 1, 2026 at 4:16 AM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Sat, Jan 31, 2026 at 12:11 PM Lukas Fittl <lukas@fittl.com> wrote:
> > I've reworked the patch a bit more, see attached v4
>
> And of course, I took the wrong branch when running "git format-patch"
> - apologies.
>
> See attached v5.

> +#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
> +#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d)    /* KVMKVMKVM */
> +
> +static bool
> +get_tsc_frequency_khz()
[..]
> +    /*
> +     * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
> +     * to us in a guest VM
> +     *
> +     * Note that accessing the 0x40000000 leaf for Hypervisor info requires
> +     * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
> +     * function does not work as expected since it contains a check for
> +     * __get_cpuid_max, which has been observed to be lower than the special
> +     * Hypervisor leaf.
> +     */
> +#if defined(HAVE__CPUIDEX)
> +    __cpuidex((int32 *) r, 0x40000000, 0);
> +    if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
> +    {
> +        __cpuidex((int32 *) r, 0x40000010, 0);
> +        if (r[0] > 0)
> +        {
> +            tsc_freq = r[0];
> +            return true;
> +        }
> +    }
> +#endif
> +
> +    return false;
> +}

When trying to understand this code I was thinking how it could be
made smaller or less dependent on such low-level intrinsics, the only
thing that came to my mind was launching systemd-detect-virt(1) via
fork+execve, as after all we do have USE_SYSTEMD (for sd_notify(2) already
consumed in backend/postmaster/postmaster.c) anyway.

Sadly this path for checking VM-types seems like opening can of worms
- they evolved lots of code to cover various other products,
see e.g. in detect_vm() and that thing is not exported.

Another way would be probably inquiring their D-Bus API, something like
below command seems to work:
   busctl get-property org.freedesktop.systemd1
/org/freedesktop/systemd1 org.freedesktop.systemd1.Manager
Virtualization

(that seems to be sd_bus_get_property_string(3)).

It's not that I'm recommending usage of any of those (which is linked
to us most of the time?) or fan of D-Bus (I'm not). I've just thought
it might be less code to use it for autodetection of VM type, but
apparently not (?) See their detect_vm_cpuid() with that vm_table[]
and memcmp() seems to be a more elegant way of writing this.

BTW, -1 to fast_clock_source, +1 to clock_source or maybe
explain_clock_source(?)

Also it would be cool if the patch would provide some way of reporting back
what clock_source was really used in case of FAST_CLOCK_SOURCE_AUTO.
Something like huge_pages_status or some elog(DEBUG).

-J.

[1] - https://github.com/systemd/systemd/blob/e831a44b07ebf48992967e366cfc1bcee2683f3d/src/detect-virt/detect-virt.c#L186
[2] - https://github.com/systemd/systemd/blob/e831a44b07ebf48992967e366cfc1bcee2683f3d/src/basic/virt.c#L450

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-02-02T16:11:32Z

On Mon, Feb 2, 2026 at 1:22 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> > +#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
> > +#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d)    /* KVMKVMKVM */
> ...
>
> When trying to understand this code I was thinking how it could be
> made smaller or less dependent on such low-level intrinsics, the only
> thing that came to my mind was launching systemd-detect-virt(1) via
> fork+execve, as after all we do have USE_SYSTEMD (for sd_notify(2) already
> consumed in backend/postmaster/postmaster.c) anyway.
>
> Sadly this path for checking VM-types seems like opening can of worms
> - they evolved lots of code to cover various other products,
> see e.g. in detect_vm() and that thing is not exported.
>
> Another way would be probably inquiring their D-Bus API, something like
> below command seems to work:
>    busctl get-property org.freedesktop.systemd1
> /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager
> Virtualization
>
> (that seems to be sd_bus_get_property_string(3)).
>
> It's not that I'm recommending usage of any of those (which is linked
> to us most of the time?) or fan of D-Bus (I'm not). I've just thought
> it might be less code to use it for autodetection of VM type, but
> apparently not (?) See their detect_vm_cpuid() with that vm_table[]
> and memcmp() seems to be a more elegant way of writing this.

Thanks for sharing - agreed that the vm_table approach they picked in
systemd's detect_vm_cpuid seems more elegant than the current
comparison mechanism.

I do think its reasonable for us to check this directly, since we're
already working with cpuid information anyway, and we also need the
TSC frequency data in the Hypervisor specific leafs (i.e. its not just
about getting the VM vendor name itself). For example, I don't think
HyperV provides TSC frequency in 0x40000010 - it was something VMware
added initially, that KVM subsequently added.

>
> BTW, -1 to fast_clock_source, +1 to clock_source or maybe
> explain_clock_source(?)

If the combined RDTSC/RDTSCP usage (as in v5) makes sense to people, I
think "clock_source" would probably make most sense - since we'd use
RDTSCP in all "slow" paths, not the system clock source.

On the other hand, if we rework this to only be relevant in the "fast"
code paths (i.e. use RDTSC for fast, but use the system clock source
always for slow) then a more narrow wording seems better.

> Also it would be cool if the patch would provide some way of reporting back
> what clock_source was really used in case of FAST_CLOCK_SOURCE_AUTO.
> Something like huge_pages_status or some elog(DEBUG).

Agreed, I think that would be useful.

Thanks for reviewing!

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-02-03T17:44:21Z

Hi,

On 2026-02-02 10:22:37 +0100, Jakub Wartak wrote:
> When trying to understand this code I was thinking how it could be
> made smaller or less dependent on such low-level intrinsics, the only
> thing that came to my mind was launching systemd-detect-virt(1) via
> fork+execve, as after all we do have USE_SYSTEMD (for sd_notify(2) already
> consumed in backend/postmaster/postmaster.c) anyway.
> 
> Sadly this path for checking VM-types seems like opening can of worms
> - they evolved lots of code to cover various other products,
> see e.g. in detect_vm() and that thing is not exported.
> 
> Another way would be probably inquiring their D-Bus API, something like
> below command seems to work:
>    busctl get-property org.freedesktop.systemd1
> /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager
> Virtualization
> 
> (that seems to be sd_bus_get_property_string(3)).
> 
> It's not that I'm recommending usage of any of those (which is linked
> to us most of the time?) or fan of D-Bus (I'm not). I've just thought
> it might be less code to use it for autodetection of VM type, but
> apparently not (?) See their detect_vm_cpuid() with that vm_table[]
> and memcmp() seems to be a more elegant way of writing this.

Yea, I doubt this is the right path... Particularly because I really think we
ought to eventually use this not just on linux but other OSs as well.


> BTW, -1 to fast_clock_source, +1 to clock_source or maybe
> explain_clock_source(?)

Hm. I think it'd make sense to use eventually use this clock source for other
timing tasks too, not just explain.  E.g. pg_stat_statements could benefit
quite a bit from reducing the timing overhead.

Whereas it'll not make sense for anything that needs wall clock times - which
imo makes a "clock_source" GUC misnamed.  Maybe "clock_source_timing" or such?

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2026-02-04T10:02:42Z

Hi Lukas!

On 31.01.2026 21:11, Lukas Fittl wrote:
> On Sun, Jan 11, 2026 at 11:26 AM David Geier <geidav.pg@gmail.com> wrote:
>>
>>> Based on Robert's suggestion I wanted to add a "fast_clock_source" enum
>>> GUC which can have the following values "auto", "rdtsc", "try_rdtsc" and
>>> "off". With that, at least no additional checks are needed and
>>> performance will remain as previously benchmarked in this thread.
>>
>> The attached patch set is rebased on latest master and contains a commit
>> which adds a "fast_clock_source" GUC that can be "try", "off" and
>> "rdtsc" on Linux.
>>
>> Alternatively, we could call the GUC "clock_source" with "auto",
>> "clock_gettime" and "rdtsc". Opinions?
> 
> No strong opinion on the GUC name ("fast_clock_source" seems fine?),
> but I think "try" is a bit confusing if our logic is more than just
> checking if the RDTSC(P) instruction is available, so I'd be in favor
> of "auto" as the default value.
> 
>> I moved the call to INSTR_TIME_INITIALIZE() from InitPostgres() to
>> PostmasterMain(). In InitPostgres() it kept the database in a recovery
>> cycle.
> 
> I think we can actually avoid having anything in PostmasterMain (or
> InitPostgres), and instead rely on the GUC assign mechanism.
> 

Good idea. Hadn't realized check and assign hooks are always called when
starting up PostgreSQL.

> I've reworked the patch a bit more, see attached v4, with a couple of
> noticeable changes:

Great. Thanks!

> In regards to the GUC:
> - Use the GUC check mechanism to complain if RDTSC clock source is
> requested, but its not available
> - Use the GUC assign mechanism to set whether we're actually using the
> RDTSC clock source

Nice!

> - "auto" now means that we use RDTSC clock source by default if we're
> on Linux x86, and the system clocksource is "tsc"

Not that I care much but I picked "try" because it's consistent with
"try" for the huge_pages GUC. What's your motivation behind "auto"?

We still need to add the new GUC to the documentation. We should mention
that with RDTSC it can be possible to get subpar performance in some
environments (e.g. with emulated TSC).

> - "rdtsc" now allows using RDTSC on any x86-based Unix-like systems (I
> see no reason to restrict the BSDs from using RDTSC when setting it
> explicitly)

Why then not also Windows, or just any x86_64-based operating system? In
reality it's a CPU feature, not an OS feature.

> - Allow changing the clock source GUC at any time, without requiring a
> restart (it makes testing much easier, and I don't see a good reason
> to require a restart, or even restrict this to superuser?)

Not requiring a restart makes sense.

Not sure about allowing any user to set it. I thought system
configuration GUCs we keep restricted to the superuser because the admin
would know best if RDTSC is actually faster or not.

> - Have pg_test_timing emit whether a fast clock source will be used by
> default (or whether one needs to change the GUC)

That's useful.

> Additionally:
> - If a client program wants to use the fast clock source (like
> pg_test_timing does), it first needs to call
> pg_initialize_fast_clock_source() -- this replaces the
> INSTR_TIME_INITIALIZE calls.
> - I've re-introduced a patch (0001) to set HAVE__CPUIDEX on modern
> GCC/clang. That's necessary to make this work on VM Hypervisors (per
> the patch's commit message)

Thanks. I had taken this out accidentally when rebasing.

> - I've merged the GUC patch together with the patch that adds the
> RDTSC implementation (0002), I don't think that makes sense to review
> or commit separately.

Makes sense. Just separated it last time to make it easier to see the
changes.

> - I've unified the RDTSC and RDTSCP handling, so we require both in
> order to use TSC as a time source. Because we have the shared
> pg_ticks_to_ns() function that gets used on an instr_time regardless
> of fast vs "slow" timing, and the variables used in that function are
> affected by the RDTSC availability, we must use TSC consistently - I
> don't think we can mix RDTSC for fast and pg_clock_gettime() for slow,
> as this patch series has done so far.

Makes sense.

> Open questions for me:
> - I'm seeing a CI test failure for "Linux - Debian Trixie - Meson"
> (times out), but its not clear if this is a fluke - I'll check if this
> recurs on the commitfest patch
> - We're doing a lot of work in pg_ticks_to_ns, even when we're not
> using RDTSC - and I think that shows in a slightly slower
> pg_test_timing measurement compared to master when fast clock source
> is off. Can we somehow only do that when we use RDTSC?

The only way I can see to improve on that is by adding another if to
directly return ticks if the default clock is used.

That saves uselessly performing an addition, a multiplication and a
shift. The overflow check shouldn't matter much because it should never
be entered and hence perfectly branch predicted because

INSTR_TIME_TICKS_TO_NANOSEC(max_ticks_no_overflow) == 562949953421311 ==
~6.5 days.

We can test how that compares performance-wise. But I'm leaning towards
keeping it as is.

> Here is a fresh test run with this patch on an AWS c6i.xlarge, i.e.
> Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz / "Ice Lake":
> 
> CREATE TABLE test (id int);
> INSERT INTO test SELECT * FROM generate_series(0, 1000000);
> 
> postgres=# SET fast_clock_source = off;
> SET
> Time: 0.107 ms
> postgres=# EXPLAIN ANALYZE SELECT COUNT(*) FROM test;
>                                                                 QUERY
> PLAN
> ------------------------------------------------------------------------------------------------------------------------------------------
>  Finalize Aggregate  (cost=10633.55..10633.56 rows=1 width=8) (actual
> time=44.117..44.811 rows=1.00 loops=1)
>    Buffers: shared hit=846 read=3579
>    ->  Gather  (cost=10633.34..10633.55 rows=2 width=8) (actual
> time=44.060..44.804 rows=3.00 loops=1)
>          Workers Planned: 2
>          Workers Launched: 2
>          Buffers: shared hit=846 read=3579
>          ->  Partial Aggregate  (cost=9633.34..9633.35 rows=1 width=8)
> (actual time=42.129..42.130 rows=1.00 loops=3)
>                Buffers: shared hit=846 read=3579
>                ->  Parallel Seq Scan on test  (cost=0.00..8591.67
> rows=416667 width=0) (actual time=0.086..21.595 rows=333333.67
> loops=3)
>                      Buffers: shared hit=846 read=3579
>  Planning Time: 0.043 ms
>  Execution Time: 44.836 ms
> (12 rows)
> 
> Time: 45.076 ms
> 
> postgres=# SET fast_clock_source = rdtsc;
> SET
> Time: 0.123 ms
> postgres=# EXPLAIN ANALYZE SELECT COUNT(*) FROM test;
>                                                                 QUERY
> PLAN
> ------------------------------------------------------------------------------------------------------------------------------------------
>  Finalize Aggregate  (cost=10633.55..10633.56 rows=1 width=8) (actual
> time=32.943..33.912 rows=1.00 loops=1)
>    Buffers: shared hit=1128 read=3297
>    ->  Gather  (cost=10633.34..10633.55 rows=2 width=8) (actual
> time=32.868..33.906 rows=3.00 loops=1)
>          Workers Planned: 2
>          Workers Launched: 2
>          Buffers: shared hit=1128 read=3297
>          ->  Partial Aggregate  (cost=9633.34..9633.35 rows=1 width=8)
> (actual time=30.705..30.706 rows=1.00 loops=3)
>                Buffers: shared hit=1128 read=3297
>                ->  Parallel Seq Scan on test  (cost=0.00..8591.67
> rows=416667 width=0) (actual time=0.080..15.223 rows=333333.67
> loops=3)
>                      Buffers: shared hit=1128 read=3297
>  Planning Time: 0.042 ms
>  Execution Time: 33.935 ms
> (12 rows)
> 
> Time: 34.180 ms
> 
> postgres=# EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM test;
>                                                       QUERY PLAN
> -----------------------------------------------------------------------------------------------------------------------
>  Finalize Aggregate  (cost=10633.55..10633.56 rows=1 width=8) (actual
> rows=1.00 loops=1)
>    Buffers: shared hit=1410 read=3015
>    ->  Gather  (cost=10633.34..10633.55 rows=2 width=8) (actual
> rows=3.00 loops=1)
>          Workers Planned: 2
>          Workers Launched: 2
>          Buffers: shared hit=1410 read=3015
>          ->  Partial Aggregate  (cost=9633.34..9633.35 rows=1 width=8)
> (actual rows=1.00 loops=3)
>                Buffers: shared hit=1410 read=3015
>                ->  Parallel Seq Scan on test  (cost=0.00..8591.67
> rows=416667 width=0) (actual rows=333333.67 loops=3)
>                      Buffers: shared hit=1410 read=3015
>  Planning Time: 0.042 ms
>  Execution Time: 27.876 ms
> (12 rows)
> 
> Time: 28.135 ms

That's a nice speedup.

It will probably show even higher with max_parallel_workers_per_gather =
0 because in your example, forking, plan serialization etc. has
significant overhead compared to the total runtime.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2026-02-04T10:06:50Z

> I do think its reasonable for us to check this directly, since we're
> already working with cpuid information anyway, and we also need the
> TSC frequency data in the Hypervisor specific leafs (i.e. its not just
> about getting the VM vendor name itself). For example, I don't think
> HyperV provides TSC frequency in 0x40000010 - it was something VMware
> added initially, that KVM subsequently added.

And then we have a hard dependency on systemd, where at the moment the
user can try to use RDTSC also on systems without.

>> Also it would be cool if the patch would provide some way of reporting back
>> what clock_source was really used in case of FAST_CLOCK_SOURCE_AUTO.
>> Something like huge_pages_status or some elog(DEBUG).
> 
> Agreed, I think that would be useful.

+1

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2026-02-04T10:09:32Z

On 03.02.2026 18:44, Andres Freund wrote:
> Yea, I doubt this is the right path... Particularly because I really think we
> ought to eventually use this not just on linux but other OSs as well.

+1

>> BTW, -1 to fast_clock_source, +1 to clock_source or maybe
>> explain_clock_source(?)
> 
> Hm. I think it'd make sense to use eventually use this clock source for other
> timing tasks too, not just explain.  E.g. pg_stat_statements could benefit
> quite a bit from reducing the timing overhead.
> 
> Whereas it'll not make sense for anything that needs wall clock times - which
> imo makes a "clock_source" GUC misnamed.  Maybe "clock_source_timing" or such?

Makes sense. clock_source_timing works for me, or maybe easier to read
would be timing_clock_source. But doesn't matter much.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-02-04T14:22:54Z

Hi,

On 2026-02-03 12:44:21 -0500, Andres Freund wrote:
> Hm. I think it'd make sense to use eventually use this clock source for other
> timing tasks too, not just explain.  E.g. pg_stat_statements could benefit
> quite a bit from reducing the timing overhead.
> 
> Whereas it'll not make sense for anything that needs wall clock times - which
> imo makes a "clock_source" GUC misnamed.  Maybe "clock_source_timing" or such?

I forgot another important one we really should use the fast timing for:
track_io_timing/track_wal_io_timing. The overhead of IO timing can be
noticeable on busy systems and it's hard to believe that the inaccuracy
matters.  This is probably particularly relevant when measuring IO intensive
workloads where the data resides in the kernel page cache, but doesn't fit
into s_b.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Ants Aasma <ants.aasma@cybertec.at> — 2026-02-05T09:23:15Z

On Wed, 4 Feb 2026 at 16:23, Andres Freund <andres@anarazel.de> wrote:
> I forgot another important one we really should use the fast timing for:
> track_io_timing/track_wal_io_timing. The overhead of IO timing can be
> noticeable on busy systems and it's hard to believe that the inaccuracy
> matters.  This is probably particularly relevant when measuring IO intensive
> workloads where the data resides in the kernel page cache, but doesn't fit
> into s_b.

I have been wanting for those two to be turned on by default with a
"fast enough" clock source. But determining this seemed complicated. I
think if the clock source for them is rdtsc we know that it is
definitely fast enough to be on by default.

--
Ants Aasma

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-02-12T16:05:27Z

On Wed, Feb 4, 2026 at 2:09 AM David Geier <geidav.pg@gmail.com> wrote:
> On 03.02.2026 18:44, Andres Freund wrote:
> > Whereas it'll not make sense for anything that needs wall clock times - which
> > imo makes a "clock_source" GUC misnamed.  Maybe "clock_source_timing" or such?
>
> Makes sense. clock_source_timing works for me, or maybe easier to read
> would be timing_clock_source. But doesn't matter much.

I've gone with "timing_clock_source" for now, that seems best to me so far.

See attached v6, which has the following changes:

Adds a new commit (0002) that:

- Unifies the Windows QueryPerformanceCounter() implementation with
the TSC implementation, and introduces the pg_ticks_to_ns function and
the ticks_per_ns_scaled conversion ratio in this commit.
- I realized that we were unnecessarily calling
QueryPerformanceFrequency on Windows every time we got the ticks, and
this is the same problem we're solving for the TSC frequency.
- This also allows us to independently test if the overhead of
overflow handling of pg_ticks_to_ns is a problem when clock_gettime is
used (since we're always exceeding max_ticks_no_overflow -- see end of
email), and makes the code read much better, I think.
- The one downside is that this means every program that wants to use
INSTR_* macros now has to call pg_initialize_timing() first. In 0003
we can then rely on the GUC mechanism to do this in the backend, as
before.

For the main TSC commit (now 0003):

- Enable the use of TSC on Windows when set explicitly via the GUC,
since I couldn't find a good reason why not - but note I have not done
any manual testing on Windows yet.
- Refactoring to improve readability and better split TSC logic from
general timing clock source logic

In regards to the GUC (part of 0003):

- Renamed to "timing_clock_source", with three settings:
  - "auto" that will use the TSC clock source if we're on Linux and
Linux itself uses the TSC clock source
  - "system" will force use of the system APIs, i.e. clock_gettime()
or QueryPerformanceCounter() -- this was named "off" before, but I
think "system" is more clear
  - "tsc" will force the use of RDTSC/RDTSCP on x86-64, and will fail
if it is not available. Not a possible setting on other architectures.
-- this was named "rdtsc" before, but I think "tsc" is better, since
we use a mix of RDTSC and RDTSCP
- Resolves the ubsan issue in CI, which was caused by a missing
addition to "config_group_names" for the new "Resource / Time" GUC
group. I wonder if we should make this "Resource / Other" instead
though? (it seems unlikely we'll have another GUC for time
specifically?)
- Implements a show hook for the GUC that will show the current value
in parenthesis with auto is selected. Is this sufficient to address
the use case of wanting to know the current clock source?

postgres=# SHOW timing_clock_source;
 timing_clock_source
---------------------
 auto (tsc)

For the pg_test_timing change (0004):
- If available, show both RDTSC and RDTSCP timings (RDTSC indicated as
"Fast"), as well as the system clock source, to help decide whether to
enable TSC.

FWIW, I have not yet investigated expanding the use of fast timing to
other places (e.g. track_io_timing/track_wal_io_timing) as suggested.

Regarding the overhead introduced with pg_ticks_to_ns:

On master (88327092ff0), I'm getting 23.54 ns from pg_test_timing - vs
with 0002 applied, this slows to 25.74 ns. I've tried to see if the
"unlikely(..)" we added in pg_ticks_to_ns is the problem (since in the
clock_gettime() case we'd always be running into that branch due to
the size of the nanoseconds value), but no luck - I think the extra
multiplication/division itself is the problem.

Any ideas how we could do this differently?

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-02-13T00:41:22Z

Hi,

On 2026-02-12 08:05:27 -0800, Lukas Fittl wrote:
> On master (88327092ff0), I'm getting 23.54 ns from pg_test_timing - vs
> with 0002 applied, this slows to 25.74 ns. I've tried to see if the
> "unlikely(..)" we added in pg_ticks_to_ns is the problem (since in the
> clock_gettime() case we'd always be running into that branch due to
> the size of the nanoseconds value), but no luck - I think the extra
> multiplication/division itself is the problem.
> 
> Any ideas how we could do this differently?

The problem looks to be that you're going to take the slowpath when using
clock_gettime(), unless you booted within the last three days, because
pg_test_timing() is doing

		INSTR_TIME_SET_CURRENT(temp);
		cur = INSTR_TIME_GET_NANOSEC(temp);

Which will require reducing time-since-boot in nanoseconds (with
CLOCK_MONOTONIC) via pg_ticks_to_ns() overflow logic, as the way
max_ticks_no_overflow is set in 0001, it'll overflow after a few days of
runtime.

This can largely be addressed by keeping prev and cur in the instr_time
domain and only converting the difference to nanoseconds.

I wonder if pg_test_timing should have a small loop with a fixed count to
determine the timing without all the overhead the existing loop has...

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-02-13T04:11:12Z

On Thu, Feb 12, 2026 at 4:41 PM Andres Freund <andres@anarazel.de> wrote:
> On 2026-02-12 08:05:27 -0800, Lukas Fittl wrote:
> > On master (88327092ff0), I'm getting 23.54 ns from pg_test_timing - vs
> > with 0002 applied, this slows to 25.74 ns. I've tried to see if the
> > "unlikely(..)" we added in pg_ticks_to_ns is the problem (since in the
> > clock_gettime() case we'd always be running into that branch due to
> > the size of the nanoseconds value), but no luck - I think the extra
> > multiplication/division itself is the problem.
> >
> > Any ideas how we could do this differently?
>
> The problem looks to be that you're going to take the slowpath when using
> clock_gettime(), unless you booted within the last three days
>
> This can largely be addressed by keeping prev and cur in the instr_time
> domain and only converting the difference to nanoseconds.

Right - we actually already had that pg_test_timing change in the
patch series in the pg_test_timing patch -- might have even been
authored by you in a prior iteration? -- however, that did not appear
to resolve the regression fully for me. I'll move this to the 0002
patch to ease testing.

> I wonder if pg_test_timing should have a small loop with a fixed count to
> determine the timing without all the overhead the existing loop has...

I agree that using a fixed count in pg_test_timing would be helpful to
measure just the timing gathering itself, vs the translation into
nanoseconds.

As an alternate to fixing this in pg_test_timing alone (in the off
chance another caller does a lot of ticks-to-time conversions), I
spent some time today looking at the assembly today for pg_ticks_to_ns
today, see [0], and specifically tried:

(1) changing the pg_ticks_to_ns logic to have an explicit
"ticks_per_ns_scaled == 0" early check and return at the start, and
setting ticks_per_ns_scaled to 0 when clock_gettime() gets used. This
is similar to what David already suggested in an earlier email.
(2) using uint64 for the ticks_per_ns_scaled/max_ticks_no_overflow
variables - this appears to help GCC generate a bit shift reliably,
instead of an idiv instruction.

That appears to eliminate the regression in my testing. Attached an
updated v7, which also has some slightly improved commit messages.

Additional comparisons with the test case you had back at the start of
this thread, with system clock source on my test VM:

master:

EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 1888.891 ms (best of 3)
pg_test_timing / Average loop time including overhead: 23.53 ns

v6 (0002 + pg_test_timing prev/cur change):

EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 1897.095 ms (best of 3)
pg_test_timing / Average loop time including overhead: 25.52 ns

v7 (0002):

EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 1897.148 ms (best of 3)
Average loop time including overhead: 23.14 ns

And when looking at the TSC time source with the full patch set on the same VM:

v6:

EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 1477.672 ms (best of 3)
pg_test_timing / Average loop time including overhead: 11.79 ns

v7:

EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
Time: 1476.326 ms (best of 3)
pg_test_timing / Average loop time including overhead: 11.78 ns

Thanks,
Lukas

[0]: https://godbolt.org/z/EvK1M66n5

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Hannu Krosing <hannuk@google.com> — 2026-02-13T14:42:36Z

On Fri, Feb 13, 2026 at 5:11 AM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Thu, Feb 12, 2026 at 4:41 PM Andres Freund <andres@anarazel.de> wrote:
> > I wonder if pg_test_timing should have a small loop with a fixed count to
> > determine the timing without all the overhead the existing loop has...
>
> I agree that using a fixed count in pg_test_timing would be helpful to
> measure just the timing gathering itself, vs the translation into
> nanoseconds.

I haven't looked at the code here yet, but when using plain rdtsc on
modern CPUs one sees much more overhead from just the fact that the
code is there than from calling the rdtsc instruction, and the
overhead can vary by orders of magnitude based on how complex the work
is that is timed.

I discovered this when I timed the (then-)new dead tid lookups in the
Vacuum in Pg 17 and saw significantly larger overhead per lookup when
the lookups themselves were slower, i.e. a case where the lookups were
done in random order (inded was on  created on a column filled with
random())

So while just a tight loop of N million rtdsc calls will give you the
lower limit, it is likely not very representative of actual overhead.

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-02-13T16:39:42Z

On Fri, Feb 13, 2026 at 6:42 AM Hannu Krosing <hannuk@google.com> wrote:
> I haven't looked at the code here yet, but when using plain rdtsc on
> modern CPUs one sees much more overhead from just the fact that the
> code is there than from calling the rdtsc instruction, and the
> overhead can vary by orders of magnitude based on how complex the work
> is that is timed.

If I understand you correctly, your comment would refer to this
function in 0003:

static inline instr_time
pg_get_ticks_fast(void)
{
#if defined(__x86_64__)
        if (likely(use_tsc))
        {
                instr_time      now;

                now.ticks = __rdtsc();
                return now;
        }
#endif

        return pg_get_ticks_system(); /* clock_gettime on POSIX */
}

I agree that the code that is here is more complex than just getting
the time via clock_gettime / vDSO - but in practice this does lower
the timing overhead, and I think so far I've not seen this regress
when testing with the system clock on x86-64 instead (i.e. not going
through the likely branch).

If you have a concern here it would be helpful to have a specific
example where you see this behave slower or measure incorrectly.

> I discovered this when I timed the (then-)new dead tid lookups in the
> Vacuum in Pg 17 and saw significantly larger overhead per lookup when
> the lookups themselves were slower, i.e. a case where the lookups were
> done in random order (inded was on  created on a column filled with
> random())

If you can share an example of what you tested here in the past, I'd
also be happy to take a look at it to understand better.

> So while just a tight loop of N million rtdsc calls will give you the
> lower limit, it is likely not very representative of actual overhead.

I agree that a tight loop itself could be scheduled differently on the
CPU than regular code paths, so our tests could be skewed if that's
all we're looking at. But that's why we're doing the combined testing
of a problematic EXPLAIN ANALYZE COUNT(*) and pg_test_timing in this
thread.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andreas Karlsson <andreas@proxel.se> — 2026-02-13T16:47:16Z

On 2/13/26 3:42 PM, Hannu Krosing wrote:
> I haven't looked at the code here yet, but when using plain rdtsc on
> modern CPUs one sees much more overhead from just the fact that the
> code is there than from calling the rdtsc instruction, and the
> overhead can vary by orders of magnitude based on how complex the work
> is that is timed.
> 
> I discovered this when I timed the (then-)new dead tid lookups in the
> Vacuum in Pg 17 and saw significantly larger overhead per lookup when
> the lookups themselves were slower, i.e. a case where the lookups were
> done in random order (inded was on  created on a column filled with
> random())
> 
> So while just a tight loop of N million rtdsc calls will give you the
> lower limit, it is likely not very representative of actual overhead.

Isn't the same issue still there if you call clock_gettime() but that it 
is just less noticeable due to the high cost of clock_gettime()?

Andreas

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2026-02-23T15:24:57Z

Hi Lukas,

Thanks for taking care of incorporating the latest patch feedback.

On 13.02.2026 05:11, Lukas Fittl wrote:
> On Thu, Feb 12, 2026 at 4:41 PM Andres Freund <andres@anarazel.de> wrote:
>> On 2026-02-12 08:05:27 -0800, Lukas Fittl wrote:
> (1) changing the pg_ticks_to_ns logic to have an explicit
> "ticks_per_ns_scaled == 0" early check and return at the start, and
> setting ticks_per_ns_scaled to 0 when clock_gettime() gets used. This
> is similar to what David already suggested in an earlier email.
> (2) using uint64 for the ticks_per_ns_scaled/max_ticks_no_overflow
> variables - this appears to help GCC generate a bit shift reliably,
> instead of an idiv instruction.
> 
> That appears to eliminate the regression in my testing. Attached an
> updated v7, which also has some slightly improved commit messages.
> 
> Additional comparisons with the test case you had back at the start of
> this thread, with system clock source on my test VM:
> 
> master:
> 
> EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 1888.891 ms (best of 3)
> pg_test_timing / Average loop time including overhead: 23.53 ns
> 
> v6 (0002 + pg_test_timing prev/cur change):
> 
> EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 1897.095 ms (best of 3)
> pg_test_timing / Average loop time including overhead: 25.52 ns
> 
> v7 (0002):
> 
> EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 1897.148 ms (best of 3)
> Average loop time including overhead: 23.14 ns

Shouldn't that result be better than master because you optimized the
loop overhead in v7-0002? That's at least what I've measured, see test
results below.

> And when looking at the TSC time source with the full patch set on the same VM:
> 
> v6:
> 
> EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 1477.672 ms (best of 3)
> pg_test_timing / Average loop time including overhead: 11.79 ns
> 
> v7:
> 
> EXPLAIN (ANALYZE, TIMING ON) SELECT count(*) FROM lotsarows;
> Time: 1476.326 ms (best of 3)
> pg_test_timing / Average loop time including overhead: 11.78 ns
> 
> Thanks,
> Lukas
> 
> [0]: https://godbolt.org/z/EvK1M66n5
> 
> --
> Lukas Fittl

The code wasn't compiling properly on Windows because __x86_64__ is not
defined in Visual C++. I've changed the code to use

  #if defined(__x86_64__) || defined(_M_X64)

I've also changed #include <x86intrin.h> to <immintrin.h>.


I've tested v8 of the patch (= v7 plus aforementioned changes) on
Windows. I'm reporting the best of 3 runs.

lotsarows test with parallelism disabled:

master: 2781 ms
v7:     2776 ms (timing_clock_source = 'system')
v7:     2091 ms (timing_clock_source = 'tsc')

pg_test_timing:

master: 27.04 ns
v7:     16.59 ns (QueryxPerformanceCounter)
v7:     13.69 ns (RDTSCP)
v7:      9.42 ns (RDTSC)


v8 of the patch is attached to this mail.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-02-23T22:27:59Z

Hi,

On 2026-02-23 16:24:57 +0100, David Geier wrote:
> The code wasn't compiling properly on Windows because __x86_64__ is not
> defined in Visual C++. I've changed the code to use
>
>   #if defined(__x86_64__) || defined(_M_X64)

Independently of this patchset I wonder if it'd be worth introducing a
PG_ARCH_X64 or such, to avoid this kind of thing.


> I've tested v8 of the patch (= v7 plus aforementioned changes) on
> Windows. I'm reporting the best of 3 runs.
>
> lotsarows test with parallelism disabled:
>
> master: 2781 ms
> v7:     2776 ms (timing_clock_source = 'system')
> v7:     2091 ms (timing_clock_source = 'tsc')

Nice.

> pg_test_timing:
>
> master: 27.04 ns
> v7:     16.59 ns (QueryxPerformanceCounter)
> v7:     13.69 ns (RDTSCP)
> v7:      9.42 ns (RDTSC)

Very nice.


Unfortunately, on linux, applying up to 0002 cause a small regression in
pg_test_timing.

With cpuidle disabled, performance governor, pinned to one core.

pg_test_timing, turboboost disabled:

412f78c66ee	27.70 ns
0002		28.48 ns

pg_test_timing, turboboost enabled:

412f78c66ee	20.41 ns
0002		21.04 ns


However, I tried, but failed, to push an actual EXPLAIN ANALYZE to show that
difference. All the differences I see are well below the run-to-run noise.

Which makes sense - the increase in overhead here probably is visible because
it increases the dependency chain inside the loop, which wouldn't be visible
in a normal explain (and of course, with more patches applied, a lot more is
won).



> From 25b58d2890e65a95ce426a0b80fab41c1c99bd8f Mon Sep 17 00:00:00 2001
> From: Lukas Fittl <lukas@fittl.com>
> Date: Sat, 31 Jan 2026 08:49:46 -0800
> Subject: [PATCH v8 1/4] Check for HAVE__CPUIDEX and HAVE__GET_CPUID_COUNT
>  separately
>
> Previously we would only check for the availability of __cpuidex if
> the related __get_cpuid_count was not available on a platform. But there
> are cases where we want to be able to call __cpuidex as the only viable
> option, specifically, when accessing a high leaf like VM Hypervisor
> information (0x40000000), which __get_cpuid_count does not allow.
>
> This will be used in an future commit to access Hypervisor information
> about the TSC frequency of x86 CPUs, where available.
>
> Note that __cpuidex is defined in cpuid.h for GCC/clang, but in intrin.h
> for MSVC. Because we now set HAVE__CPUIDEX for GCC/clang when available,
> adjust existing code to check for _MSC_VER when including intrin.h.
>
> Author: Lukas Fittl <lukas@fittl.com>
> Reviewed-by:
> Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de



>  # Check for XSAVE intrinsics
> diff --git a/meson.build b/meson.build
> index ebfb85e93e5..312c919eaa4 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -2080,7 +2080,8 @@ elif cc.links('''
>  endif
>
>
> -# Check for __get_cpuid_count() and __cpuidex() in a similar fashion.
> +# Check for __get_cpuid_count() and __cpuidex() separately, since we sometimes
> +# need __cpuidex() even if __get_cpuid_count() is available.
>  if cc.links('''
>      #include <cpuid.h>
>      int main(int arg, char **argv)
> @@ -2091,8 +2092,13 @@ if cc.links('''
>      ''', name: '__get_cpuid_count',
>      args: test_c_args)
>    cdata.set('HAVE__GET_CPUID_COUNT', 1)
> -elif cc.links('''
> +endif
> +if cc.links('''
> +    #ifdef _MSC_VER
>      #include <intrin.h>
> +    #else
> +    #include <cpuid.h>
> +    #endif
>      int main(int arg, char **argv)
>      {
>          unsigned int exx[4] = {0, 0, 0, 0};

FWIW, this seems to trigger a warning locally:

/srv/dev/build/postgres/m-dev-assert/meson-private/tmpw34r2pnc/testfile.c: In function 'main':
/srv/dev/build/postgres/m-dev-assert/meson-private/tmpw34r2pnc/testfile.c:10:19: warning: pointer targets in passing argument 1 of '__cpuidex' differ in signe>
   10 |         __cpuidex(exx, 7, 0);
      |                   ^~~
      |                   |
      |                   unsigned int *
In file included from /srv/dev/build/postgres/m-dev-assert/meson-private/tmpw34r2pnc/testfile.c:5:
/home/andres/build/gcc/master/install/lib/gcc/x86_64-pc-linux-gnu/16/include/cpuid.h:361:16: note: expected 'int *' but argument is of type 'unsigned int *'
  361 | __cpuidex (int __cpuid_info[4], int __leaf, int __subleaf)
      |            ~~~~^~~~~~~~~~~~~~~
-----------
Checking if "__cpuidex" links: YES 




> diff --git a/src/port/pg_crc32c_sse42_choose.c b/src/port/pg_crc32c_sse42_choose.c
> index f586476964f..7a75380b483 100644
> --- a/src/port/pg_crc32c_sse42_choose.c
> +++ b/src/port/pg_crc32c_sse42_choose.c
> @@ -20,11 +20,11 @@
>
>  #include "c.h"
>
> -#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT)
> +#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
>  #include <cpuid.h>
>  #endif

Why would we want to include cpuid.h with msvc if one of the other variables
is defined?


> -#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX)
> +#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
>  #include <intrin.h>
>  #endif

And here, why would we want to include intrin.h if HAVE__CPUID is defined?


Seems like this should just be something like

#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) || defined(HAVE__CPUIDEX)
#if defined(_MSC_VER)
#include <intrin.h>
#else
#include <cpuid.h>
#endif /* defined(_MSC_VER) */
#endif



> From 2392d95626599a1b5562f9216eb8c334db99c932 Mon Sep 17 00:00:00 2001
> From: Lukas Fittl <lukas@fittl.com>
> Date: Fri, 25 Jul 2025 17:57:20 -0700
> Subject: [PATCH v8 2/4] Timing: Streamline ticks to nanosecond conversion
>  across platforms
>
> The timing infrastructure (INSTR_* macros) measures time elapsed using
> clock_gettime() on POSIX systems, which returns the time as nanoseconds,
> and QueryPerformanceCounter() on Windows, which is a specialized timing
> clock source that returns a tick counter that needs to be converted to
> nanoseconds using the result of QueryPerformanceFrequency().
>
> This conversion currently happens ad-hoc on Windows, e.g. when calling
> INSTR_TIME_GET_NANOSEC, which calls QueryPerformanceFrequency() on every
> invocation, despite the frequency being stable after program start,
> incurring unnecessary overhead. It also causes a fractured implementation
> where macros are defined differently between platforms.
>
> To ease code readability, and prepare for a future change that intends
> to use a ticks-to-nanosecond conversion on x86-64 for TSC use, introduce
> a new pg_ticks_to_ns() function that gets called on all platforms.
>
> This function relies on a separately initialized ticks_per_ns_scaled
> value, that represents the conversion ratio. This value is initialized
> from QueryPerformanceFrequency() on Windows, and set to zero on x86-64
> POSIX systems, which results in the ticks being treated as nanoseconds.
> Other architectures always directly return the original ticks.
>
> To support this, pg_initialize_timing() is introduced, and is now
> mandatory for both the backend and any frontend programs to call before
> utilizing INSTR_* macros.

I wonder if it's worth trying to transparently initialize in the overflow
codepath. Probably not, but worth explicitly considering.


> In passing modify pg_test_timing to reduce the per-loop overhead caused
> by repeated divisions in INSTR_TIME_GET_NANOSEC when the ticks variable
> has become very large. Instead diff first and then turn it into nanosecs.

I'd like to see this broken out into a separate change.


> diff --git a/src/bin/pg_test_timing/pg_test_timing.c b/src/bin/pg_test_timing/pg_test_timing.c
> index a5621251afc..9fd630a490a 100644

> @@ -182,9 +184,8 @@ test_timing(unsigned int duration)
>  					bits;
>
>  		prev = cur;
> -		INSTR_TIME_SET_CURRENT(temp);
> -		cur = INSTR_TIME_GET_NANOSEC(temp);
> -		diff = cur - prev;
> +		INSTR_TIME_SET_CURRENT(cur);
> +		diff = INSTR_TIME_DIFF_NANOSEC(cur, prev);
>
>  		/* Did time go backwards? */
>  		if (unlikely(diff < 0))

FWIW, I don't think this needs a special INSTR_TIME macro, it could just use
INSTR_TIME_SUBTRACT() and INSTR_TIME_GET_NANOSEC().



> diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
> index cb4e986092e..c8b233be16c 100644
> --- a/src/bin/pgbench/pgbench.c
> +++ b/src/bin/pgbench/pgbench.c
> @@ -7334,6 +7334,9 @@ main(int argc, char **argv)
>  		initRandomState(&state[i].cs_func_rs);
>  	}
>
> +	/* initialize timing infrastructure (required for INSTR_* calls) */
> +	pg_initialize_timing();
> +
>  	/* opening connection... */
>  	con = doConnect();
>  	if (con == NULL)

FWIW, I also verified that I am am unable to see measure overhead in pgbench
due to the more expensive conversion.  Not surprised, but it did seem like a
possibility, because pgbench unfortunately always converts the gathered time
to microseconds, rather than compute a difference between two timestamps.


> +
> +/*
> + * Stores what the number of ticks needs to be multiplied with to end up
> + * with nanoseconds using integer math.
> + *
> + * On certain platforms (currently Windows) the ticks to nanoseconds conversion
> + * requires floating point math because:
> + *
> + * sec = ticks / frequency_hz
> + * ns  = ticks / frequency_hz * 1,000,000,000
> + * ns  = ticks * (1,000,000,000 / frequency_hz)
> + * ns  = ticks * (1,000,000 / frequency_khz) <-- now in kilohertz
> + *
> + * Here, 'ns' is usually a floating number. For example for a 2.5 GHz CPU
> + * the scaling factor becomes 1,000,000 / 2,500,000 = 1.2.
> + *
> + * To be able to use integer math we work around the lack of precision. We
> + * first scale the integer up and after the multiplication by the number
> + * of ticks in INSTR_TIME_GET_NANOSEC() we divide again by the same value.
> + * We picked the scaler such that it provides enough precision and is a
> + * power-of-two which allows for shifting instead of doing an integer
> + * division. We utilize unsigned integers even though ticks are stored as a
> + * signed value because that encourages compilers to generate better assembly.


> + * On all other platforms we are using clock_gettime(), which uses nanoseconds
> + * as ticks. Hence, we set the multiplier to zero, which causes pg_ticks_to_ns
> + * to return the original value.
> + */
> +uint64		ticks_per_ns_scaled = 0;
> +uint64		max_ticks_no_overflow = 0;
> +
> +static void set_ticks_per_ns(void);
> +
> +void
> +pg_initialize_timing()
> +{
> +	set_ticks_per_ns();
> +}
> +
> +#ifndef WIN32
> +
> +static void
> +set_ticks_per_ns()
> +{
> +	ticks_per_ns_scaled = 0;
> +	max_ticks_no_overflow = 0;
> +}
> +
> +#else							/* WIN32 */
> +
> +/* GetTimerFrequency returns counts per second */
> +static inline double
> +GetTimerFrequency(void)
> +{
> +	LARGE_INTEGER f;
> +
> +	QueryPerformanceFrequency(&f);
> +	return (double) f.QuadPart;
> +}
> +
> +static void
> +set_ticks_per_ns()
> +{
> +	ticks_per_ns_scaled = INT64CONST(1000000000) * TICKS_TO_NS_PRECISION / GetTimerFrequency();


This should probably use NS_PER_S.

I wonder whether we should use an explicit shift here and in pg_ticks_to_ns(),
to avoid having to rely on the compiler to do so for us.


> +static inline int64
> +pg_ticks_to_ns(int64 ticks)
> +{
> +#if defined(__x86_64__) || defined(_M_X64)
> +	int64		ns = 0;
> +
> +	if (ticks_per_ns_scaled == 0)
> +		return ticks;

There should be comment explaining (or referencing another explanation) for
why this exists.


> +	/*
> +	 * Would multiplication overflow? If so perform computation in two parts.
> +	 * Check overflow without actually overflowing via: a * b > max <=> a >
> +	 * max / b
> +	 */
> +	if (unlikely(ticks > (int64) max_ticks_no_overflow))

The "via" comment seems a bit misplaced, given that the transformation is not
really utilized here (but at the point where max_ticks_no_overflow) is
computed.


> +	{
> +		/*
> +		 * Compute how often the maximum number of ticks fits completely into
> +		 * the number of elapsed ticks and convert that number into
> +		 * nanoseconds. Then multiply by the count to arrive at the final
> +		 * value. In a 2nd step we adjust the number of elapsed ticks and
> +		 * convert the remaining ticks.
> +		 */
> +		int64		count = ticks / max_ticks_no_overflow;
> +		int64		max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
> +
> +		ns = max_ns * count;
> +
> +		/*
> +		 * Subtract the ticks that we now already accounted for, so that they
> +		 * don't get counted twice.
> +		 */
> +		ticks -= count * max_ticks_no_overflow;
> +		Assert(ticks >= 0);

I think we could perhaps make the overflow case a good bit cheaper, by
avoiding any divisions with a non-constant factor (assuming I haven't blown
the logic below).  Instead of doing a division we can "transform back" into
the non-scaled representation, I think?

ns = (ticks * ticks_per_ns_scaled) / TICKS_TO_NS_PRECISION

  equals, assuming arbitrary precision

ns = (ticks / TICKS_TO_NS_PRECISION) * ticks_per_ns_scaled

  and not assuming arbitrary precision:

count = ticks // TICKS_TO_NS_PRECISION
rem_ticks = ticks - (count * TICKS_TO_NS_PRECISION)
ns = count * ticks_per_ns_scaled + rem_ticks * ticks_per_ns_scaled // TICKS_TO_NS_PRECISION

None of which afaict would overflow?




> --- a/src/common/instr_time.c
> +++ b/src/common/instr_time.c
> @@ -20,8 +20,8 @@
>   * Stores what the number of ticks needs to be multiplied with to end up
>   * with nanoseconds using integer math.
>   *
> - * On certain platforms (currently Windows) the ticks to nanoseconds conversion
> - * requires floating point math because:
> + * In certain cases (TSC on x86-64, and QueryPerformanceCounter on Windows)
> + * the ticks to nanoseconds conversion requires floating point math because:
>   *
>   * sec = ticks / frequency_hz
>   * ns  = ticks / frequency_hz * 1,000,000,000
> @@ -39,7 +39,7 @@
>   * division. We utilize unsigned integers even though ticks are stored as a
>   * signed value because that encourages compilers to generate better assembly.
>   *
> - * On all other platforms we are using clock_gettime(), which uses nanoseconds
> + * In all other cases we are using clock_gettime(), which uses nanoseconds
>   * as ticks. Hence, we set the multiplier to zero, which causes pg_ticks_to_ns
>   * to return the original value.
>   */
> @@ -48,16 +48,57 @@ uint64		max_ticks_no_overflow = 0;
>
>  static void set_ticks_per_ns(void);
>
> +int			timing_clock_source = TIMING_CLOCK_SOURCE_AUTO;
> +
> +#if defined(__x86_64__) || defined(_M_X64)
> +/* Indicates if TSC instructions (RDTSC and RDTSCP) are usable. */
> +extern bool has_usable_tsc;
> +
> +static void tsc_initialize(void);
> +static bool tsc_use_by_default(void);
> +static void set_ticks_per_ns_for_tsc(void);
> +static bool set_tsc_frequency_khz(void);
> +static bool is_rdtscp_available(void);
> +#endif
> +
>  void
>  pg_initialize_timing()
>  {
> +#if defined(__x86_64__) || defined(_M_X64)
> +	tsc_initialize();
> +#endif
> +
> +	set_ticks_per_ns();
> +}
> +
> +bool
> +pg_set_timing_clock_source(TimingClockSourceType source)
> +{
> +#if defined(__x86_64__) || defined(_M_X64)
> +	switch (source)
> +	{
> +		case TIMING_CLOCK_SOURCE_AUTO:
> +			use_tsc = has_usable_tsc && tsc_use_by_default();
> +			break;
> +		case TIMING_CLOCK_SOURCE_SYSTEM:
> +			use_tsc = false;
> +			break;
> +		case TIMING_CLOCK_SOURCE_TSC:
> +			if (!has_usable_tsc)	/* Tell caller TSC is not usable */
> +				return false;
> +			use_tsc = true;
> +			break;
> +	}
> +#endif
>  	set_ticks_per_ns();
> +	timing_clock_source = source;
> +	return true;
>  }

Perhaps this should ensure that pg_initialize_timing() has already been called?


> +bool
> +check_timing_clock_source(int *newval, void **extra, GucSource source)
> +{
> +#if defined(__x86_64__) || defined(_M_X64)
> +	pg_initialize_timing();
> +
> +	if (*newval == TIMING_CLOCK_SOURCE_TSC && !has_usable_tsc)
> +	{
> +		GUC_check_errdetail("TSC is not supported as fast clock source");
> +		return false;

The GUC name doesn't refer to "fast", so probably this shouldn't either?


> +const char *
> +show_timing_clock_source()
> +{
> +#if defined(__x86_64__) || defined(_M_X64)
> +	TimingClockSourceType effective_source = pg_current_timing_clock_source();
> +
> +	switch (timing_clock_source)
> +	{
> +		case TIMING_CLOCK_SOURCE_AUTO:
> +			if (effective_source == TIMING_CLOCK_SOURCE_TSC)
> +				return "auto (tsc)";
> +			else
> +				return "auto (system)";
> +		case TIMING_CLOCK_SOURCE_SYSTEM:
> +			return "system";
> +		case TIMING_CLOCK_SOURCE_TSC:
> +			return "tsc";
> +	}
> +#else
> +	switch (timing_clock_source)
> +	{
> +		case TIMING_CLOCK_SOURCE_AUTO:
> +			return "auto (system)";
> +		case TIMING_CLOCK_SOURCE_SYSTEM:
> +			return "system";
> +	}
> +#endif

Seems like it'd be nicer if we had one switch with the ifdef-ery inside the
TIMING_CLOCK_SOURCE_AUTO case?  If we add support for tsc based clock sources
on arm as well, this would get a bit unmanageable.



> +static uint32 tsc_frequency_khz = 0;
> +
> +/*
> + * Decide whether we use the RDTSC/RDTSCP instructions at runtime, for Linux/x86-64,
> + * instead of incurring the overhead of a full clock_gettime() call.
> + *
> + * This can't be reliably determined at compile time, since the
> + * availability of an "invariant" TSC (that is not affected by CPU
> + * frequency changes) is dependent on the CPU architecture. Additionally,
> + * there are cases where TSC availability is impacted by virtualization,
> + * where a simple cpuid feature check would not be enough.
> + */
> +static void
> +tsc_initialize(void)
> +{
> +	/*
> +	 * Compute baseline CPU peformance, determines speed at which the TSC
> +	 * advances.
> +	 */
> +	if (!set_tsc_frequency_khz())
> +		return;
> +
> +	has_usable_tsc = is_rdtscp_available();
> +}
> +
> +/*
> + * Decides whether to use TSC clock source if the user did not specify it
> + * one way or the other, and it is available (checked separately).
> + *
> + * Currently only enabled by default on Linux, since Linux already does a
> + * significant amount of work to determine whether TSC is a viable clock
> + * source.
> + */
> +static bool
> +tsc_use_by_default()

Postgres style is still funcname(void).


> +{
> +#if defined(__linux__)
> +	FILE	   *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
> +	char		buf[128];
> +
> +	if (!fp)
> +		return false;
> +
> +	if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
> +	{
> +		fclose(fp);
> +		return true;
> +	}
> +
> +	fclose(fp);
> +#endif
> +
> +	return false;
> +}

I think this will often disable tsc on VMs, due to linux defaulting to
kvm-clock in KVM VMs.

Do we care about that?


If the tsc is not actually viable, is it still listed in
/sys/devices/system/clocksource/clocksource0/available_clocksource
?


> +
> +#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
> +#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d)	/* KVMKVMKVM */
> +
> +static bool
> +set_tsc_frequency_khz()
> +{
> +	uint32		r[4] = {0, 0, 0, 0};
> +
> +#if defined(HAVE__GET_CPUID)
> +	__get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
> +#elif defined(HAVE__CPUID)
> +	__cpuid(r, 0x15);
> +#else
> +#error cpuid instruction not available
> +#endif
> +
> +	if (r[2] > 0)
> +	{
> +		if (r[0] == 0 || r[1] == 0)
> +			return false;
> +
> +		tsc_frequency_khz = r[2] / 1000 * r[1] / r[0];
> +		return true;
> +	}

I think there should be some explanation about what this is testing.
Including perhaps a reference to the relevant documents.



> +	/* Some CPUs only report frequency in 16H */

Dito.


> +#if defined(HAVE__GET_CPUID)
> +	__get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
> +#elif defined(HAVE__CPUID)
> +	__cpuid(r, 0x16);
> +#else
> +#error cpuid instruction not available
> +#endif

Perhaps we could package the __get_cpuid / __cpuid thing in a wrapper, instead
of repeating the ifdefery three times?



> index 985b6b5af88..e7191c5d6cd 100644
> --- a/src/include/portability/instr_time.h
> +++ b/src/include/portability/instr_time.h
> @@ -4,9 +4,10 @@
>   *	  portable high-precision interval timing
>   *
>   * This file provides an abstraction layer to hide portability issues in
> - * interval timing.  On Unix we use clock_gettime(), and on Windows we use
> - * QueryPerformanceCounter().  These macros also give some breathing room to
> - * use other high-precision-timing APIs.
> + * interval timing. On x86 we use the RDTSC/RDTSCP instruction directly in
> + * certain cases, or alternatively clock_gettime() on Unix-like systems and
> + * QueryPerformanceCounter() on Windows. These macros also give some breathing
> + * room to use other high-precision-timing APIs.
>   *
>   * The basic data type is instr_time, which all callers should treat as an
>   * opaque typedef.  instr_time can store either an absolute time (of
> @@ -17,10 +18,11 @@
>   *
>   * INSTR_TIME_SET_ZERO(t)			set t to zero (memset is acceptable too)
>   *
> - * INSTR_TIME_SET_CURRENT(t)		set t to current time
> + * INSTR_TIME_SET_CURRENT_FAST(t)	set t to current time without waiting
> + * 									for instructions in out-of-order window
>   *
> - * INSTR_TIME_SET_CURRENT_LAZY(t)	set t to current time if t is zero,
> - *									evaluates to whether t changed
> + * INSTR_TIME_SET_CURRENT(t)		set t to current time while waiting for
> + * 									instructions in OOO to retire
>   *
>   * INSTR_TIME_ADD(x, y)				x += y
>   *

I'd probably remove INSTR_TIME_SET_CURRENT_LAZY in a prep commit.


> @@ -93,13 +95,54 @@ typedef struct instr_time
>  extern PGDLLIMPORT uint64 ticks_per_ns_scaled;
>  extern PGDLLIMPORT uint64 max_ticks_no_overflow;
>
> +#if defined(__x86_64__) || defined(_M_X64)
> +#include <immintrin.h>

Why do we need to include immintrin.h in instr_time.h?  Including immintrin.h
makes compilation a lot slower:

$ echo '#include <immintrin.h>'|gcc -ftime-report -xc -o /dev/null -c -

Time variable                                  wall           GGC
 phase setup                        :   0.00 (  1%)  1905k (  6%)
 phase parsing                      :   0.50 ( 99%)    30M ( 94%)
 preprocessing                      :   0.09 ( 18%)  5375k ( 16%)
 lexical analysis                   :   0.02 (  5%)     0  (  0%)
 parser (global)                    :   0.31 ( 61%)    12M ( 39%)
 parser inl. func. body             :   0.08 ( 15%)    12M ( 39%)
 TOTAL                              :   0.51           32M



>  int
> @@ -46,10 +46,47 @@ main(int argc, char *argv[])
>  	/* initialize timing infrastructure (required for INSTR_* calls) */
>  	pg_initialize_timing();
>
> -	loop_count = test_timing(test_duration);
> -
> +	/*
> +	 * First, test default (non-fast) timing code. A clock source for that is
> +	 * always available. Hence, we can unconditionally output the result.
> +	 */
> +	loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_SYSTEM, false);
>  	output(loop_count);
>
> +#if defined(__x86_64__) || defined(_M_X64)

I don't love that now test_timing.c has architecture specific checks.  Could
we abstract this a bit more?


> +	/*
> +	 * If on a supported architecture, test the RDTSC clock source. This clock
> +	 * source is not always available. In that case the loop count will be 0
> +	 * and we don't print.
> +	 *
> +	 * We first emit RDTSCP timings, which is slower, and gets used for higher
> +	 * precision measurements when the TSC clock source is enabled. We emit
> +	 * RDTSC second, which is used for faster timing measurements with lower
> +	 * precision.
> +	 */
> +	printf("\n");
> +	loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, false);
> +	if (loop_count > 0)
> +	{
> +		output(loop_count);
> +		printf("\n");
> +
> +		/* Now, emit fast timing measurements */
> +		loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, true);
> +		output(loop_count);
> +		printf("\n");
> +
> +		pg_set_timing_clock_source(TIMING_CLOCK_SOURCE_AUTO);
> +		if (pg_current_timing_clock_source() == TIMING_CLOCK_SOURCE_TSC)
> +			printf(_("TSC clock source will be used by default, unless timing_clock_source is set to 'system'.\n"));
> +		else
> +			printf(_("TSC clock source will not be used by default, unless timing_clock_source is set to 'tsc'.\n"));
> +	}
> +	else
> +		printf(_("TSC clock source is not usable. Likely unable to determine TSC frequency. are you running in an unsupported virtualized environment?.\n"));
> +#endif
> +

A bit weird that most of the output stuff is handled in output(), but then
some of it is handled directly in main() now, some of it in test_timing().


Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-02-24T02:01:51Z

On Tue, Feb 24, 2026 at 5:28 AM Andres Freund <andres@anarazel.de> wrote:
> On 2026-02-23 16:24:57 +0100, David Geier wrote:
> > The code wasn't compiling properly on Windows because __x86_64__ is not
> > defined in Visual C++. I've changed the code to use
> >
> >   #if defined(__x86_64__) || defined(_M_X64)
>
> Independently of this patchset I wonder if it'd be worth introducing a
> PG_ARCH_X64 or such, to avoid this kind of thing.

+1

I've already borrowed USE_SSE2 for this meaning in commit b9278871f,
but that's conflating two different things and I'd actually prefer the
above, plus one that includes 32-bit as well.

+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}

I'm hoping to centralize CPU-specific checks like the above:

https://www.postgresql.org/message-id/CANWCAZbKQ2im1r4ztcyfqNh_6gaJzyewabExYrkACNvMNyqxog%40mail.gmail.com

is_rdtscp_available() is an easy thing to delegate to my patch, but I
agree it would be easier if that was abstracted a bit more so that a
different leaf can be passed each time. The latter could also be used
to simplify the frequency and hypervisor stuff as well.

--
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-02-24T07:22:13Z

On Mon, Feb 23, 2026 at 6:02 PM John Naylor <johncnaylorls@gmail.com> wrote:
>
> On Tue, Feb 24, 2026 at 5:28 AM Andres Freund <andres@anarazel.de> wrote:
> > On 2026-02-23 16:24:57 +0100, David Geier wrote:
> > > The code wasn't compiling properly on Windows because __x86_64__ is not
> > > defined in Visual C++. I've changed the code to use
> > >
> > >   #if defined(__x86_64__) || defined(_M_X64)
> >
> > Independently of this patchset I wonder if it'd be worth introducing a
> > PG_ARCH_X64 or such, to avoid this kind of thing.
>
> +1
>
> I've already borrowed USE_SSE2 for this meaning in commit b9278871f,
> but that's conflating two different things and I'd actually prefer the
> above, plus one that includes 32-bit as well.

+1, would be good to have a consistent definition for this, I hadn't
realized this differs between platforms. John, do you want to take
care of adding that since you recently added USE_SSE2?

Its worth noting that we've previously analyzed when the TSC is
invariant (i.e. doesn't change frequency across power level changes)
and concluded that it is on basically all x86-64 CPUs - but I don't
think that necessarily holds true on 32-bit x86, rare as it is.

Since we now intend to have the GUC for controlling the timing clock
source this seems less problematic, so maybe we can allow (but not
default to) use on any x86 platform that has CPUID telling us both
RDTSC and RDTSCP are available, whether its 32-bit or 64-bit.

>
> +static bool
> +is_rdtscp_available()
> +{
> + uint32 r[4] = {0, 0, 0, 0};
> +
> +#if defined(HAVE__GET_CPUID)
> + if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
> + return false;
> +#elif defined(HAVE__CPUID)
> + __cpuid(r, 0x80000001);
> +#else
> +#error cpuid instruction not available
> +#endif
> +
> + return (r[3] & (1 << 27)) != 0;
> +}
>
> I'm hoping to centralize CPU-specific checks like the above:
>
> https://www.postgresql.org/message-id/CANWCAZbKQ2im1r4ztcyfqNh_6gaJzyewabExYrkACNvMNyqxog%40mail.gmail.com
>
> is_rdtscp_available() is an easy thing to delegate to my patch, but I
> agree it would be easier if that was abstracted a bit more so that a
> different leaf can be passed each time. The latter could also be used
> to simplify the frequency and hypervisor stuff as well.

Yeah, that makes sense, agreed it'd be nice to centralize the CPU
architecture specific code that utilizes cpuid/etc.

Looking at your v5/0002 over there, that should work well. As you
note, is_rdtscp_available is an easy delegation to your logic - I
think we can probably always fetch the 0x80000001 leaf to check for
RDTSCP presence in the proposed set_x86_features?

For set_tsc_frequency_khz, if we go forward with this route, I think
we should just turn that back into a get_... function (i.e. not
setting the global inside) and have that as an additional exported
method from pg_cpu.h that instr_time.c can consume. Generalizing use
of the __get_cpuid/__cpuid variants use makes sense to me.

I'll get back to the other patch feedback tomorrow.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-02-25T10:00:35Z

Hi Andres, thanks for the detailed review!

And thanks to David for testing on Windows (and patching what's
needed), great to have the confirmation this works there too.

See attached v9, rebased, with feedback addressed, unless otherwise noted.

0001 is the updated cpuidex patch, and should be good to go (though we
could also merge this into the TSC commit, if you prefer)
0002 is the INSTR_TIME_SET_CURRENT_LAZY removal pulled forward
0003 is the INSTR_TIME_LT => INSTR_TIME_GT naming fix from [0] because
I assume we'll push that shortly, and 0004 now uses that macro
0004 is the pg_test_timing change split out, with an additional
optimization (see below)
0005 is the same as v8/0002 (Streamline ticks to nanosecond
conversion), but needs some more brain energy to address your
suggestions (or feel free to drive that forward if you'd like)
0006 and 0007 are the two TSC related changes, with feedback
addressed, except for documenting the cpuid calls better (added TODO)

On Mon, Feb 23, 2026 at 2:28 PM Andres Freund <andres@anarazel.de> wrote:
> > diff --git a/meson.build b/meson.build
> > index ebfb85e93e5..312c919eaa4 100644
> > --- a/meson.build
> > +++ b/meson.build
> ...
> >      int main(int arg, char **argv)
> >      {
> >          unsigned int exx[4] = {0, 0, 0, 0};
>
> FWIW, this seems to trigger a warning locally:

Good catch, adjusted for both Meson and make - it looks like that was
wrong before already, since even MSVC defines this as a signed integer
[1].

> > From 2392d95626599a1b5562f9216eb8c334db99c932 Mon Sep 17 00:00:00 2001
> > From: Lukas Fittl <lukas@fittl.com>
> > Date: Fri, 25 Jul 2025 17:57:20 -0700
> > Subject: [PATCH v8 2/4] Timing: Streamline ticks to nanosecond conversion
> >  across platforms
> >
> > ...
> > To support this, pg_initialize_timing() is introduced, and is now
> > mandatory for both the backend and any frontend programs to call before
> > utilizing INSTR_* macros.
>
> I wonder if it's worth trying to transparently initialize in the overflow
> codepath. Probably not, but worth explicitly considering.

If I follow, you're thinking of something like:

- Initialize max_ticks_no_overflow to 0 by default
- In the overflow path (which we'd reach the first time around), do an
extra check if max_ticks_no_overflow == 0, and then call the
initialization function
- The initialization function sets max_ticks_no_overflow to a non-zero
value, so we don't get there the second time around

Is that right? (I think it could work, since its "just" an extra jump
instruction in an unlikely edge case)

> > diff --git a/src/bin/pg_test_timing/pg_test_timing.c b/src/bin/pg_test_timing/pg_test_timing.c
> > index a5621251afc..9fd630a490a 100644
>
> > @@ -182,9 +184,8 @@ test_timing(unsigned int duration)
> >                                       bits;
> >
> >               prev = cur;
> > -             INSTR_TIME_SET_CURRENT(temp);
> > -             cur = INSTR_TIME_GET_NANOSEC(temp);
> > -             diff = cur - prev;
> > +             INSTR_TIME_SET_CURRENT(cur);
> > +             diff = INSTR_TIME_DIFF_NANOSEC(cur, prev);
> >
> >               /* Did time go backwards? */
> >               if (unlikely(diff < 0))
>
> FWIW, I don't think this needs a special INSTR_TIME macro, it could just use
> INSTR_TIME_SUBTRACT() and INSTR_TIME_GET_NANOSEC().
>

I think that makes sense, but FWIW, its a bit inconvenient since
INSTR_TIME_SUBTRACT modifies the first argument (its "x -= y", not "x
- y"), and we re-use "cur" later. But we can just make a copy of "cur"
first that we pass to the macro.

Additionally, I've now adjusted this to calculate the target end time
when starting up test_timing, and simply compare the current time
against that (with INSTR_TIME_GT), avoiding a INSTR_TIME_SUBTRACT and
INSTR_TIME_GET_NANOSEC in the hot loop. That does add another macro
though (INSTR_TIME_SET_NANOSEC, to be able to initialize instr_time
from a user-defined interval value), but I think that's worth it.

> > +
> > +static void
> > +set_ticks_per_ns()
> > +{
> > +     ticks_per_ns_scaled = INT64CONST(1000000000) * TICKS_TO_NS_PRECISION / GetTimerFrequency();
>
>
> This should probably use NS_PER_S.

Done.

> I wonder whether we should use an explicit shift here and in pg_ticks_to_ns(),
> to avoid having to rely on the compiler to do so for us.

I've left this as-is for now since I lacked the brain space to write
out the shift logic - but fine doing this either way.

> > +     /*
> > +      * Would multiplication overflow? If so perform computation in two parts.
> > +      * Check overflow without actually overflowing via: a * b > max <=> a >
> > +      * max / b
> > +      */
> > +     if (unlikely(ticks > (int64) max_ticks_no_overflow))
>
> The "via" comment seems a bit misplaced, given that the transformation is not
> really utilized here (but at the point where max_ticks_no_overflow) is
> computed.

Hmm, yeah, I see your point. I wonder if we should move the "via ..."
part to the comment that's at the top of instr_time.c?

> > +     {
> > +             /*
> > +              * Compute how often the maximum number of ticks fits completely into
> > +              * the number of elapsed ticks and convert that number into
> > +              * nanoseconds. Then multiply by the count to arrive at the final
> > +              * value. In a 2nd step we adjust the number of elapsed ticks and
> > +              * convert the remaining ticks.
> > +              */
> > +             int64           count = ticks / max_ticks_no_overflow;
> > +             int64           max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
> > +
> > +             ns = max_ns * count;
> > +
> > +             /*
> > +              * Subtract the ticks that we now already accounted for, so that they
> > +              * don't get counted twice.
> > +              */
> > +             ticks -= count * max_ticks_no_overflow;
> > +             Assert(ticks >= 0);
>
> I think we could perhaps make the overflow case a good bit cheaper, by
> avoiding any divisions with a non-constant factor (assuming I haven't blown
> the logic below).  Instead of doing a division we can "transform back" into
> the non-scaled representation, I think?
>
> ns = (ticks * ticks_per_ns_scaled) / TICKS_TO_NS_PRECISION
>
>   equals, assuming arbitrary precision
>
> ns = (ticks / TICKS_TO_NS_PRECISION) * ticks_per_ns_scaled
>
>   and not assuming arbitrary precision:
>
> count = ticks // TICKS_TO_NS_PRECISION
> rem_ticks = ticks - (count * TICKS_TO_NS_PRECISION)
> ns = count * ticks_per_ns_scaled + rem_ticks * ticks_per_ns_scaled // TICKS_TO_NS_PRECISION
>
> None of which afaict would overflow?

I've left this as is for now since I didn't write the original logic
here (I think it was you in a prior version?), and I need a good
night's sleep to think through this. Additional help welcome to review
your proposal.

> > +{
> > +#if defined(__linux__)
> > +     FILE       *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
> > +     char            buf[128];
> > +
> > +     if (!fp)
> > +             return false;
> > +
> > +     if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
> > +     {
> > +             fclose(fp);
> > +             return true;
> > +     }
> > +
> > +     fclose(fp);
> > +#endif
> > +
> > +     return false;
> > +}
>
> I think this will often disable tsc on VMs, due to linux defaulting to
> kvm-clock in KVM VMs.
>
> Do we care about that?
>
>
> If the tsc is not actually viable, is it still listed in
> /sys/devices/system/clocksource/clocksource0/available_clocksource
> ?

I think unless we want to do additional checks ourselves (something
like in [2]), we need to be careful here, and can't rely on the
presence of "tsc" in available clock sources to mean its viable.

Specifically, my understanding is that the Kernel lists "tsc" as
available in more cases, and then if chosen in the beginning, has a
watchdog logic that observes the TSC and modifies it as needed if its
not viable. I think in such cases "tsc" would continue to be listed as
available, but the Kernel would have notified the user in the kernel
log that TSC is unstable.

>
> > +
> > +#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
> > +#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d)     /* KVMKVMKVM */
> > +
> > +static bool
> > +set_tsc_frequency_khz()
> > +{
> > +     uint32          r[4] = {0, 0, 0, 0};
> > +
> > +#if defined(HAVE__GET_CPUID)
> > +     __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
> > +#elif defined(HAVE__CPUID)
> > +     __cpuid(r, 0x15);
> > +#else
> > +#error cpuid instruction not available
> > +#endif
> > +
> > +     if (r[2] > 0)
> > +     {
> > +             if (r[0] == 0 || r[1] == 0)
> > +                     return false;
> > +
> > +             tsc_frequency_khz = r[2] / 1000 * r[1] / r[0];
> > +             return true;
> > +     }
>
> I think there should be some explanation about what this is testing.
> Including perhaps a reference to the relevant documents.
>
>
>
> > +     /* Some CPUs only report frequency in 16H */
>
> Dito.
>

Agreed - I've left this for a future revision to spell out, but added
a TODO for now so we don't forget.

>
> > @@ -93,13 +95,54 @@ typedef struct instr_time
> >  extern PGDLLIMPORT uint64 ticks_per_ns_scaled;
> >  extern PGDLLIMPORT uint64 max_ticks_no_overflow;
> >
> > +#if defined(__x86_64__) || defined(_M_X64)
> > +#include <immintrin.h>
>
> Why do we need to include immintrin.h in instr_time.h?  Including immintrin.h
> makes compilation a lot slower:

We previously had x86intrin.h there, I think David changed that to
immintrin.h in v8. I've adjusted this to use intrin.h on MSVC instead,
as that's noted as the correct file to include from [3], and back to
x86intrin.h for other platforms.

>
> >  int
> > @@ -46,10 +46,47 @@ main(int argc, char *argv[])
> >       /* initialize timing infrastructure (required for INSTR_* calls) */
> >       pg_initialize_timing();
> >
> > -     loop_count = test_timing(test_duration);
> > -
> > +     /*
> > +      * First, test default (non-fast) timing code. A clock source for that is
> > +      * always available. Hence, we can unconditionally output the result.
> > +      */
> > +     loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_SYSTEM, false);
> >       output(loop_count);
> >
> > +#if defined(__x86_64__) || defined(_M_X64)
>
> I don't love that now test_timing.c has architecture specific checks.  Could
> we abstract this a bit more?

That's fair. I think it might be best if we introduce a new define
that controls whether we're on an architecture that supports the TSC
logic.

I've added PG_INSTR_TSC_CLOCK for that purpose (naming feedback
welcome), and converted most "defined(__x86_64__) || defined(_M_X64)"
to utilize that instead, including the one in pg_test_timing.c. I've
also removed direct "RDTSC" and "RTDSCP" clock source names in
pg_test_timing.c and added separate defines for that, in the
theoretical case we add support for TSC-like mechanisms on another
architecture.

> > +     /*
> > +      * If on a supported architecture, test the RDTSC clock source. This clock
> > +      * source is not always available. In that case the loop count will be 0
> > +      * and we don't print.
> > +      *
> > +      * We first emit RDTSCP timings, which is slower, and gets used for higher
> > +      * precision measurements when the TSC clock source is enabled. We emit
> > +      * RDTSC second, which is used for faster timing measurements with lower
> > +      * precision.
> > +      */
> > +     printf("\n");
> > +     loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, false);
> > +     if (loop_count > 0)
> > +     {
> > +             output(loop_count);
> > +             printf("\n");
> > +
> > +             /* Now, emit fast timing measurements */
> > +             loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, true);
> > +             output(loop_count);
> > +             printf("\n");
> > +
> > +             pg_set_timing_clock_source(TIMING_CLOCK_SOURCE_AUTO);
> > +             if (pg_current_timing_clock_source() == TIMING_CLOCK_SOURCE_TSC)
> > +                     printf(_("TSC clock source will be used by default, unless timing_clock_source is set to 'system'.\n"));
> > +             else
> > +                     printf(_("TSC clock source will not be used by default, unless timing_clock_source is set to 'tsc'.\n"));
> > +     }
> > +     else
> > +             printf(_("TSC clock source is not usable. Likely unable to determine TSC frequency. are you running in an unsupported virtualized environment?.\n"));
> > +#endif
> > +
>
> A bit weird that most of the output stuff is handled in output(), but then
> some of it is handled directly in main() now, some of it in test_timing().

Hmm. I kind of see what you mean. I wonder if the main oddity there is
having lots of TSC logic in the main function, when other stuff lives
later in the file.

To explore an alternate structure, I've added test_system_timing() and
test_tsc_timing() methods, so its better abstracted. The new TSC
printfs are still directly in test_tsc_timing (moved from main),
because I don't see an easy way to have these happen in output(). We
could consider renaming "output" to "output_timings" for further
clarity?

Thanks,
Lukas

[0]: https://www.postgresql.org/message-id/flat/CAP53PkzGbyeJMLDAcvMRgzXPXYsYXZr3SBg0UwhfkYjqu8oK_g%40mail.gmail.com#7e007d1c3769c2755f0d98fa7f8b048a
[1]: https://learn.microsoft.com/en-us/cpp/intrinsics/cpuid-cpuidex?view=msvc-170
[2]: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/tsc.c#L1268
[3]: https://learn.microsoft.com/en-us/cpp/intrinsics/rdtsc?view=msvc-170

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-02-25T11:15:49Z

On Tue, Feb 24, 2026 at 2:22 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Mon, Feb 23, 2026 at 6:02 PM John Naylor <johncnaylorls@gmail.com> wrote:
> >
> > On Tue, Feb 24, 2026 at 5:28 AM Andres Freund <andres@anarazel.de> wrote:
> > > On 2026-02-23 16:24:57 +0100, David Geier wrote:
> > > > The code wasn't compiling properly on Windows because __x86_64__ is not
> > > > defined in Visual C++. I've changed the code to use
> > > >
> > > >   #if defined(__x86_64__) || defined(_M_X64)
> > >
> > > Independently of this patchset I wonder if it'd be worth introducing a
> > > PG_ARCH_X64 or such, to avoid this kind of thing.
> >
> > +1
> >
> > I've already borrowed USE_SSE2 for this meaning in commit b9278871f,
> > but that's conflating two different things and I'd actually prefer the
> > above, plus one that includes 32-bit as well.
>
> +1, would be good to have a consistent definition for this, I hadn't
> realized this differs between platforms. John, do you want to take
> care of adding that since you recently added USE_SSE2?

In the attached I tried like this:

/*
 * compiler-independent macros for CPU architecture
 */
#if defined(__x86_64__) || defined(_M_X64)
#define PG_ARCH_X64
#elif defined(__i386__) || defined(_M_IX86)
#define PG_ARCH_X32
#elif defined(__aarch64__) || defined(_M_ARM64)
#define PG_ARCH_ARM64
#elif defined(__arm__) || defined(__arm) || defined(_M_ARM)
#define PG_ARCH_ARM32
#endif

and adjusted a couple places to suit. The Arm ones aren't used yet so
I could leave them out for now.

> > is_rdtscp_available() is an easy thing to delegate to my patch, but I
> > agree it would be easier if that was abstracted a bit more so that a
> > different leaf can be passed each time. The latter could also be used
> > to simplify the frequency and hypervisor stuff as well.
>
> Yeah, that makes sense, agreed it'd be nice to centralize the CPU
> architecture specific code that utilizes cpuid/etc.
>
> Looking at your v5/0002 over there, that should work well. As you
> note, is_rdtscp_available is an easy delegation to your logic - I
> think we can probably always fetch the 0x80000001 leaf to check for
> RDTSCP presence in the proposed set_x86_features?

I think that would work. Your pg_cpuid() looks like the abstraction we
need. I think that would also allow my v5/0002 to avoid worrying about
ordering dependencies. (It resets to check leaf 7, but some AVX
features that we don't use need leaf 1)

--
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-02-25T11:57:19Z

On Wed, Feb 25, 2026 at 11:01 AM Lukas Fittl <lukas@fittl.com> wrote:
[..]
> See attached v9, rebased, with feedback addressed, unless otherwise noted.

FWIW, I've tried this on FreeBSD and I couldn't understand why I
couldn't enable TSC
(atlought) the cpuid tools reported RDTSCP being available. I have found out
that set_tsc_frequency_khz() returns false and tsc_frequency_khz==0 before even
inquiry for is_rdtscp_available() is done. As this was on virtualbox VM (so not
suppoprted by current code), I think this is pretty clear we'll need some way of
informing users why he cannot SET timing_clock_source to TSC. Possibly errhint()
could differentiate the reason? E.g.
  "failed to auto-detect TSC frequency (potential hypervisor issue)" or
  "CPU RDTSCP instructions are not available"

-J.

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-02-26T16:14:24Z

Hi,

On 2026-02-25 02:00:35 -0800, Lukas Fittl wrote:
> Hi Andres, thanks for the detailed review!
>
> And thanks to David for testing on Windows (and patching what's
> needed), great to have the confirmation this works there too.
>
> See attached v9, rebased, with feedback addressed, unless otherwise noted.
>
> 0001 is the updated cpuidex patch, and should be good to go (though we
> could also merge this into the TSC commit, if you prefer)

John, does this interact with your work at all?


> 0002 is the INSTR_TIME_SET_CURRENT_LAZY removal pulled forward
> 0003 is the INSTR_TIME_LT => INSTR_TIME_GT naming fix from [0] because
> I assume we'll push that shortly, and 0004 now uses that macro

Pushed these two.



> > > From 2392d95626599a1b5562f9216eb8c334db99c932 Mon Sep 17 00:00:00 2001
> > > From: Lukas Fittl <lukas@fittl.com>
> > > Date: Fri, 25 Jul 2025 17:57:20 -0700
> > > Subject: [PATCH v8 2/4] Timing: Streamline ticks to nanosecond conversion
> > >  across platforms
> > >
> > > ...
> > > To support this, pg_initialize_timing() is introduced, and is now
> > > mandatory for both the backend and any frontend programs to call before
> > > utilizing INSTR_* macros.
> >
> > I wonder if it's worth trying to transparently initialize in the overflow
> > codepath. Probably not, but worth explicitly considering.
>
> If I follow, you're thinking of something like:
>
> - Initialize max_ticks_no_overflow to 0 by default
> - In the overflow path (which we'd reach the first time around), do an
> extra check if max_ticks_no_overflow == 0, and then call the
> initialization function
> - The initialization function sets max_ticks_no_overflow to a non-zero
> value, so we don't get there the second time around
>
> Is that right? (I think it could work, since its "just" an extra jump
> instruction in an unlikely edge case)

Yes, that's what I was wondering about.


> > > diff --git a/src/bin/pg_test_timing/pg_test_timing.c b/src/bin/pg_test_timing/pg_test_timing.c
> > > index a5621251afc..9fd630a490a 100644
> >
> > > @@ -182,9 +184,8 @@ test_timing(unsigned int duration)
> > >                                       bits;
> > >
> > >               prev = cur;
> > > -             INSTR_TIME_SET_CURRENT(temp);
> > > -             cur = INSTR_TIME_GET_NANOSEC(temp);
> > > -             diff = cur - prev;
> > > +             INSTR_TIME_SET_CURRENT(cur);
> > > +             diff = INSTR_TIME_DIFF_NANOSEC(cur, prev);
> > >
> > >               /* Did time go backwards? */
> > >               if (unlikely(diff < 0))
> >
> > FWIW, I don't think this needs a special INSTR_TIME macro, it could just use
> > INSTR_TIME_SUBTRACT() and INSTR_TIME_GET_NANOSEC().
> >
>
> I think that makes sense, but FWIW, its a bit inconvenient since
> INSTR_TIME_SUBTRACT modifies the first argument (its "x -= y", not "x
> - y"), and we re-use "cur" later. But we can just make a copy of "cur"
> first that we pass to the macro.

Exactly.


> Additionally, I've now adjusted this to calculate the target end time
> when starting up test_timing, and simply compare the current time
> against that (with INSTR_TIME_GT), avoiding a INSTR_TIME_SUBTRACT and
> INSTR_TIME_GET_NANOSEC in the hot loop. That does add another macro
> though (INSTR_TIME_SET_NANOSEC, to be able to initialize instr_time
> from a user-defined interval value), but I think that's worth it.

Agreed.



> > > +     /*
> > > +      * Would multiplication overflow? If so perform computation in two parts.
> > > +      * Check overflow without actually overflowing via: a * b > max <=> a >
> > > +      * max / b
> > > +      */
> > > +     if (unlikely(ticks > (int64) max_ticks_no_overflow))
> >
> > The "via" comment seems a bit misplaced, given that the transformation is not
> > really utilized here (but at the point where max_ticks_no_overflow) is
> > computed.
>
> Hmm, yeah, I see your point. I wonder if we should move the "via ..."
> part to the comment that's at the top of instr_time.c?

Yea, probably some central part documenting this makes sense.


> > > +     {
> > > +             /*
> > > +              * Compute how often the maximum number of ticks fits completely into
> > > +              * the number of elapsed ticks and convert that number into
> > > +              * nanoseconds. Then multiply by the count to arrive at the final
> > > +              * value. In a 2nd step we adjust the number of elapsed ticks and
> > > +              * convert the remaining ticks.
> > > +              */
> > > +             int64           count = ticks / max_ticks_no_overflow;
> > > +             int64           max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
> > > +
> > > +             ns = max_ns * count;
> > > +
> > > +             /*
> > > +              * Subtract the ticks that we now already accounted for, so that they
> > > +              * don't get counted twice.
> > > +              */
> > > +             ticks -= count * max_ticks_no_overflow;
> > > +             Assert(ticks >= 0);
> >
> > I think we could perhaps make the overflow case a good bit cheaper, by
> > avoiding any divisions with a non-constant factor (assuming I haven't blown
> > the logic below).  Instead of doing a division we can "transform back" into
> > the non-scaled representation, I think?
> >
> > ns = (ticks * ticks_per_ns_scaled) / TICKS_TO_NS_PRECISION
> >
> >   equals, assuming arbitrary precision
> >
> > ns = (ticks / TICKS_TO_NS_PRECISION) * ticks_per_ns_scaled
> >
> >   and not assuming arbitrary precision:
> >
> > count = ticks // TICKS_TO_NS_PRECISION
> > rem_ticks = ticks - (count * TICKS_TO_NS_PRECISION)
> > ns = count * ticks_per_ns_scaled + rem_ticks * ticks_per_ns_scaled // TICKS_TO_NS_PRECISION
> >
> > None of which afaict would overflow?
>
> I've left this as is for now since I didn't write the original logic
> here (I think it was you in a prior version?), and I need a good
> night's sleep to think through this. Additional help welcome to review
> your proposal.

i don't remember writing the logic, but that doesn't say much :)


> > I think this will often disable tsc on VMs, due to linux defaulting to
> > kvm-clock in KVM VMs.
> >
> > Do we care about that?
> >
> >
> > If the tsc is not actually viable, is it still listed in
> > /sys/devices/system/clocksource/clocksource0/available_clocksource
> > ?
>
> I think unless we want to do additional checks ourselves (something
> like in [2]), we need to be careful here, and can't rely on the
> presence of "tsc" in available clock sources to mean its viable.
>
> Specifically, my understanding is that the Kernel lists "tsc" as
> available in more cases, and then if chosen in the beginning, has a
> watchdog logic that observes the TSC and modifies it as needed if its
> not viable. I think in such cases "tsc" would continue to be listed as
> available, but the Kernel would have notified the user in the kernel
> log that TSC is unstable.

We probably should verify that the kernel indeed behaves that way, otherwise
far fewer people will benefit from this improvement.



> > > @@ -93,13 +95,54 @@ typedef struct instr_time
> > >  extern PGDLLIMPORT uint64 ticks_per_ns_scaled;
> > >  extern PGDLLIMPORT uint64 max_ticks_no_overflow;
> > >
> > > +#if defined(__x86_64__) || defined(_M_X64)
> > > +#include <immintrin.h>
> >
> > Why do we need to include immintrin.h in instr_time.h?  Including immintrin.h
> > makes compilation a lot slower:
>
> We previously had x86intrin.h there, I think David changed that to
> immintrin.h in v8. I've adjusted this to use intrin.h on MSVC instead,
> as that's noted as the correct file to include from [3], and back to
> x86intrin.h for other platforms.

My concern is that instr_time.h is quite widely included, including through
executor/instrument.h and pgstat.h.

An -O0 build without including immintrin.h:
  $ ninja clean && rm -f .ninja_* && CCACHE_DISABLE=1 time ninja
  297.05user 53.54system 0:21.53elapsed 1628%CPU (0avgtext+0avgdata 493044maxresident)k
  584inputs+4215576outputs (36major+13819907minor)pagefaults 0swaps

just adding #include <immintrin.h> to instr_time.h:
  $ ninja clean && rm -f .ninja_* && CCACHE_DISABLE=1 time ninja
  529.83user 81.39system 0:47.85elapsed 1277%CPU (0avgtext+0avgdata 585492maxresident)k
  3504inputs+5232544outputs (31major+23905659minor)pagefaults 0swaps

I.e. the elapsed build time more than doubled.  That seems problematic to me.


I think we could either:

a) Avoid the expensive include, e.g. by including a narrower header or by just
   using the underlying builtin directly.

b) Introduce a separate header just defining the instr_time type, which then
   is included in headers like executor/instrument.h, pgstat.h, where we just
   need the type.


> > > +     /*
> > > +      * If on a supported architecture, test the RDTSC clock source. This clock
> > > +      * source is not always available. In that case the loop count will be 0
> > > +      * and we don't print.
> > > +      *
> > > +      * We first emit RDTSCP timings, which is slower, and gets used for higher
> > > +      * precision measurements when the TSC clock source is enabled. We emit
> > > +      * RDTSC second, which is used for faster timing measurements with lower
> > > +      * precision.
> > > +      */
> > > +     printf("\n");
> > > +     loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, false);
> > > +     if (loop_count > 0)
> > > +     {
> > > +             output(loop_count);
> > > +             printf("\n");
> > > +
> > > +             /* Now, emit fast timing measurements */
> > > +             loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, true);
> > > +             output(loop_count);
> > > +             printf("\n");
> > > +
> > > +             pg_set_timing_clock_source(TIMING_CLOCK_SOURCE_AUTO);
> > > +             if (pg_current_timing_clock_source() == TIMING_CLOCK_SOURCE_TSC)
> > > +                     printf(_("TSC clock source will be used by default, unless timing_clock_source is set to 'system'.\n"));
> > > +             else
> > > +                     printf(_("TSC clock source will not be used by default, unless timing_clock_source is set to 'tsc'.\n"));
> > > +     }
> > > +     else
> > > +             printf(_("TSC clock source is not usable. Likely unable to determine TSC frequency. are you running in an unsupported virtualized environment?.\n"));
> > > +#endif
> > > +
> >
> > A bit weird that most of the output stuff is handled in output(), but then
> > some of it is handled directly in main() now, some of it in test_timing().
>
> Hmm. I kind of see what you mean. I wonder if the main oddity there is
> having lots of TSC logic in the main function, when other stuff lives
> later in the file.
>
> To explore an alternate structure, I've added test_system_timing() and
> test_tsc_timing() methods, so its better abstracted. The new TSC
> printfs are still directly in test_tsc_timing (moved from main),
> because I don't see an easy way to have these happen in output().

Seems like an improvement.

Haven't yet had the bandwidth to review most of the new version.  Mostly
wanted to get out an explanation for the immintrin.h concern.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-02-27T14:00:37Z

On Thu, Feb 26, 2026 at 11:14 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-02-25 02:00:35 -0800, Lukas Fittl wrote:
> > See attached v9, rebased, with feedback addressed, unless otherwise noted.
> >
> > 0001 is the updated cpuidex patch, and should be good to go (though we
> > could also merge this into the TSC commit, if you prefer)
>
> John, does this interact with your work at all?

I just pushed the main piece of the centralization of runtime checks.
It conflicts, but it's easy to fix since now 0001 will have one less
place to change. I don't think my other patch will conflict. In any
case, I'll push it tomorrow if all goes well. I can help with the
other things we talked about.

-- 
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-03T10:29:06Z

On Thu, Feb 26, 2026 at 8:14 AM Andres Freund <andres@anarazel.de> wrote:
> > 0002 is the INSTR_TIME_SET_CURRENT_LAZY removal pulled forward
> > 0003 is the INSTR_TIME_LT => INSTR_TIME_GT naming fix from [0] because
> > I assume we'll push that shortly, and 0004 now uses that macro
>
> Pushed these two.

Thanks!

See attached rebased v10 with remaining feedback addressed (except
noted below), as well as:

- Moved the CPU feature bit detection and TSC frequency logic to the
new pg_cpu_x86.c (thanks John!)
- Explicitly check that the CPU has the TSC invariant bit set (this is
a safety measure, to go along with a change to the default handling as
described later in this email)
- Explicitly check if the CPU has the hypervisor bit set, and only do
the hypervisor TSC frequency logic in that case (I think that's
correct, based on my read how the Kernel TSC code handles this when
its in a VM itself)
- Added support for HyperV hypervisor by reading the TSC frequency
MSR. This allows Azure Linux VMs to work as well, and in my test gives
a similar speed up with RDTSC like reported on AWS. Only annoyance is
that to enable it you have to make /dev/cpu/0/msr readable ("setcap
cap_sys_rawio=ep" on the binary that accesses it + give the user/group
access to the device file)

FWIW, I have not incorporated John's patch re: architecture detection
- I think we could get that done independently of this patch set and
rebase as needed, or we can add it here if preferred.

Two outstanding questions this time around:

1) I have not added a better way to return the TSC error information
(i.e. telling the user why TSC can't be used, as requested by Jakub),
because its not clear to me what the best paradigm here is. I agree
it'd be useful, but I think we should have a way to show that both
when setting the GUC to "tsc" explicitly (and failing), and in
pg_test_timing. Maybe there should be a global "tsc_error_reason"
string that is set by the frequency initialization logic, and read by
the GUC check / pg_test_timing?

2) Another thing that came up whilst I was looking for reference
materials to add missing comments: I wonder if our use of RDTSCP (i.e.
the "slow" timing when TSC is enabled) needs an LFENCE instruction
following it, to ensure accuracy, per the comments in the abseil
library that uses it for timing [0]:

"The newer RDTSCP is sometimes described as serializing, but it
actually only serves as a half-fence with release semantics. Although
all instructions in the region will complete before the final
timestamp is captured, subsequent instructions may leak into the
region and increase the elapsed time. Inserting another fence after
the final RDTSCP would prevent such reordering without affecting the
measured region."

I haven't done any actual testing with this, but the argument seems
sound, so maybe its worth adding the instruction for accuracy with
RDTSCP?

> > > I wonder if it's worth trying to transparently initialize in the overflow
> > > codepath. Probably not, but worth explicitly considering.
> >
> > If I follow, you're thinking of something like:
> >
> > - Initialize max_ticks_no_overflow to 0 by default
> > - In the overflow path (which we'd reach the first time around), do an
> > extra check if max_ticks_no_overflow == 0, and then call the
> > initialization function
> > - The initialization function sets max_ticks_no_overflow to a non-zero
> > value, so we don't get there the second time around
> >
> > Is that right? (I think it could work, since its "just" an extra jump
> > instruction in an unlikely edge case)
>
> Yes, that's what I was wondering about.

Having thought more about this, I think this is a bad idea. First, it
would make a subsequent call to pg_ticks_to_ns with the same
instr_time input return incorrect data (since ticks was set before we
set use_tsc=true). Second, we have no guarantee today that the
instr_time value that pg_ticks_to_ns got called on was the only one
that was set and not read yet.

Put differently: Any call to pg_get_ticks/pg_get_ticks_fast before the
TSC is initialized would store ticks in the system frame of reference,
and once pg_ticks_to_ns is called any subsequent pg_get_ticks* calls
would store in the TSC frame of reference. But we don't know at
pg_ticks_to_ns read time whether the instr_time value was stored
before the TSC was initialized, or after.

> > > I think we could perhaps make the overflow case a good bit cheaper, by
> > > avoiding any divisions with a non-constant factor (assuming I haven't blown
> > > the logic below).  Instead of doing a division we can "transform back" into
> > > the non-scaled representation, I think?
> > >
> > > ns = (ticks * ticks_per_ns_scaled) / TICKS_TO_NS_PRECISION
> > >
> > >   equals, assuming arbitrary precision
> > >
> > > ns = (ticks / TICKS_TO_NS_PRECISION) * ticks_per_ns_scaled
> > >
> > >   and not assuming arbitrary precision:
> > >
> > > count = ticks // TICKS_TO_NS_PRECISION
> > > rem_ticks = ticks - (count * TICKS_TO_NS_PRECISION)
> > > ns = count * ticks_per_ns_scaled + rem_ticks * ticks_per_ns_scaled // TICKS_TO_NS_PRECISION
> > >
> > > None of which afaict would overflow?
> >
> > I've left this as is for now since I didn't write the original logic
> > here (I think it was you in a prior version?), and I need a good
> > night's sleep to think through this. Additional help welcome to review
> > your proposal.
>
> i don't remember writing the logic, but that doesn't say much :)

I've thought this through, and I think this seems sound. Adjusted, and
switched over to using explicit bit-shifts as well.

This code could probably still use another thorough read, just to
double check I didn't mess the math up now, especially in the overflow
case.

> > > I think this will often disable tsc on VMs, due to linux defaulting to
> > > kvm-clock in KVM VMs.
> > >
> > > Do we care about that?
> > >
> > >
> > > If the tsc is not actually viable, is it still listed in
> > > /sys/devices/system/clocksource/clocksource0/available_clocksource
> > > ?
> >
> > I think unless we want to do additional checks ourselves (something
> > like in [2]), we need to be careful here, and can't rely on the
> > presence of "tsc" in available clock sources to mean its viable.
> >
> > Specifically, my understanding is that the Kernel lists "tsc" as
> > available in more cases, and then if chosen in the beginning, has a
> > watchdog logic that observes the TSC and modifies it as needed if its
> > not viable. I think in such cases "tsc" would continue to be listed as
> > available, but the Kernel would have notified the user in the kernel
> > log that TSC is unstable.
>
> We probably should verify that the kernel indeed behaves that way, otherwise
> far fewer people will benefit from this improvement.

So I've verified that the Linux Kernel behaves this way, at least by
looking at the code again, i.e. "tsc" won't be removed from the
available clocksource list if the watchdog disqualifies it. I lack a
non-virtualized x86 system to test this with in practice, but I've
found nothing that contradicts this.

But, I also see your point that we want to broaden who can benefit by
default. And after further research, I think we can make a stronger
case for declaring the TSC safe to use ourselves, based on the Linux
kernel discussion in 2021 that stated [1]:

"We're finally at a point where TSC seems to be halfways reliable and
less abused by BIOS tinkerers. TSC_ADJUST was really key as we can now
detect even small modifications reliably and the important point is
that we can cure them as well (not pretty but better than all other
options). ... There is still no architecural guarantee for TSCs being
synchronized on machines with more than 4 sockets."

That then led the kernel folks to turning off the TSC watchdog at boot
when: the TSC is invariant, TSC_ADJUST is set (Intel only), and the
system has 4 sockets or less. No luck for AMD unfortunately, which the
kernel doesn't trust to the same level, so we probably shouldn't
either.

Based on this, I've adjusted our default choice logic to match the
Linux kernel logic (on any platform), but kept the previous Linux
clock source check as a fallback, which helps for AMD systems and 8+
socket systems where the Linux TSC watchdog did its work (presumably
shortly after boot, before Postgres starts), and confirmed the TSC is
safe to use.

On the cloud platforms I checked (AWS and GCP), both the TscInvariant
and TSC_ADJUST bits are set in the VM, so that should also help with
virtualized scenarios if the clock source is kvm-clock but the TSC is
in good shape.

> > > > @@ -93,13 +95,54 @@ typedef struct instr_time
> > > >  extern PGDLLIMPORT uint64 ticks_per_ns_scaled;
> > > >  extern PGDLLIMPORT uint64 max_ticks_no_overflow;
> > > >
> > > > +#if defined(__x86_64__) || defined(_M_X64)
> > > > +#include <immintrin.h>
> > >
> > > Why do we need to include immintrin.h in instr_time.h?  Including immintrin.h
> > > makes compilation a lot slower:
> >
> > We previously had x86intrin.h there, I think David changed that to
> > immintrin.h in v8. I've adjusted this to use intrin.h on MSVC instead,
> > as that's noted as the correct file to include from [3], and back to
> > x86intrin.h for other platforms.
>
> My concern is that instr_time.h is quite widely included, including through
> executor/instrument.h and pgstat.h.
>
> An -O0 build without including immintrin.h:
>   $ ninja clean && rm -f .ninja_* && CCACHE_DISABLE=1 time ninja
>   297.05user 53.54system 0:21.53elapsed 1628%CPU (0avgtext+0avgdata 493044maxresident)k
>   584inputs+4215576outputs (36major+13819907minor)pagefaults 0swaps
>
> just adding #include <immintrin.h> to instr_time.h:
>   $ ninja clean && rm -f .ninja_* && CCACHE_DISABLE=1 time ninja
>   529.83user 81.39system 0:47.85elapsed 1277%CPU (0avgtext+0avgdata 585492maxresident)k
>   3504inputs+5232544outputs (31major+23905659minor)pagefaults 0swaps
>
> I.e. the elapsed build time more than doubled.  That seems problematic to me.
>
>
> I think we could either:
>
> a) Avoid the expensive include, e.g. by including a narrower header or by just
>    using the underlying builtin directly.

Per discussing this with Andres off-list (since I was at first a bit
confused and incorrectly stated that using x86intrin.h helps here -
which it doesn't), I've adjusted this to use the built-ins for
RDTSC/RDTSCP like suggested. Those have been available in GCC for a
long time, and were added to clang in 3.5.0, released in September
2014. I think that means we can rely on them without an explicit
configure check. MSVC on the other hand requires the use of intrin.h
from my research.

Thanks,
Lukas

[0]: https://github.com/abseil/abseil-cpp/blob/20240116.2/absl/random/internal/nanobenchmark.cc#L178
[1]: https://lore.kernel.org/lkml/87eekfk8bd.fsf@nanos.tec.linutronix.de/

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-03-03T17:09:08Z

Hi,

Just a quick comment, haven't actually had time to look through the email/new
patch version with any care.

On 2026-03-03 02:29:06 -0800, Lukas Fittl wrote:
> - Added support for HyperV hypervisor by reading the TSC frequency
> MSR. This allows Azure Linux VMs to work as well, and in my test gives
> a similar speed up with RDTSC like reported on AWS. Only annoyance is
> that to enable it you have to make /dev/cpu/0/msr readable ("setcap
> cap_sys_rawio=ep" on the binary that accesses it + give the user/group
> access to the device file)

I rather doubt that giving even just read access to MSRs to unprivileged
userspace processes is a good idea.

But if we read files anyway, wouldn't just using
  /sys/devices/system/cpu/cpu0/cpufreq/base_frequency
work?

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-03T18:22:42Z

On Tue, Mar 3, 2026 at 9:09 AM Andres Freund <andres@anarazel.de> wrote:
> On 2026-03-03 02:29:06 -0800, Lukas Fittl wrote:
> > - Added support for HyperV hypervisor by reading the TSC frequency
> > MSR. This allows Azure Linux VMs to work as well, and in my test gives
> > a similar speed up with RDTSC like reported on AWS. Only annoyance is
> > that to enable it you have to make /dev/cpu/0/msr readable ("setcap
> > cap_sys_rawio=ep" on the binary that accesses it + give the user/group
> > access to the device file)
>
> I rather doubt that giving even just read access to MSRs to unprivileged
> userspace processes is a good idea.

Yeah, that's fair - the interface here is rather crude, and I agree
that MSR access is problematic for non-root.

> But if we read files anyway, wouldn't just using
>   /sys/devices/system/cpu/cpu0/cpufreq/base_frequency
> work?

I tested this just now on an Azure VM (Standard D2s v3), and its
close, but unfortunately CPU frequency doesn't match the TSC frequency
(cpuinfo_max_freq is 2800000, scaling_cur_freq is 2496279, and TSC
frequency via MSR is 2793438 -- note that I didn't have base_frequency
on this VM). My understanding is that the TSC clock is virtualized in
HyperV and does not directly match the CPU frequency.

I'm also happy to take this out again - maybe we can get the
HyperV/Azure Linux folks to improve the Kernel side here to pass down
the TSC frequency without needing the MSR, and just not support it for
now.

An alternate idea could be to allow overriding the TSC frequency via a
GUC - then one could use the root user (or a setuid program) to get
the TSC frequency on Azure/HyperV via the MSR and pass it to Postgres
at start. But not sure that's worth the trouble, since it won't help
with environments that don't have a reliable TSC (e.g. Virtualbox, I
think).

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-03-04T13:49:52Z

On Tue, Mar 3, 2026 at 5:29 PM Lukas Fittl <lukas@fittl.com> wrote:
> See attached rebased v10 with remaining feedback addressed (except
> noted below), as well as:
>
> - Moved the CPU feature bit detection and TSC frequency logic to the
> new pg_cpu_x86.c (thanks John!)

Sure thing! Integration with the new file looks good. I have a few
comments on that below, some of which are nice-to-haves which could be
split out into a separate refactoring patch. (I've not studied the
entire patch in detail):

v10-0001

--- a/src/port/pg_cpu_x86.c
+++ b/src/port/pg_cpu_x86.c
@@ -17,12 +17,12 @@

 #if defined(USE_SSE2) || defined(__i386__)

-#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT)
-#include <cpuid.h>
-#endif
-
-#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX)
+#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) ||
defined(HAVE__CPUIDEX)
+#if defined(_MSC_VER)
 #include <intrin.h>
+#else
+#include <cpuid.h>
+#endif /* defined(_MSC_VER) */

What if HAVE__CPUID is defined but not HAVE__CPUIDEX? Is that possible?

v10-0004:

+ pg_cpuid(1, exx);

Since some leaves later on use hex values, so maybe they should all be
hex for consistency.

- /* second cpuid call on leaf 7 to check extended AVX-512 support */
-
- memset(exx, 0, 4 * sizeof(exx[0]));
+ /* Use leaf 7 to check TSC_ADJUST and extended AVX-512 support */
+ memset(exx, 0, 4 * sizeof(exx[0]));

It wasn't previously stated outright, but I think the important point
here is we need to memset before this call so that platforms without
extended cpuid don't have stale values. It's not important at this
point what the goal of the call is. [thinking] Actually it's worse,
see below.

Speaking of initialization, I noticed we historically used "foo =
{0,0,0,0}" (lack of C99?), but "foo = {0}" is more concise. Not this
patch's responsibility, but it does add a few more of this pattern. It
would look nicer if we changed all those at some point.

+ X86Features[PG_RDTSCP] = (exx[3] & (1 << 27)) != 0;

The style I've used elsewhere for setting features is

X86Features[PG_SSE4_2] = exx[2] >> 20 & 1;

But maybe it'd actually be better to have a function that can be called like

x86_set_feature(PG_RDTSCP, EDX, 27);

...with the appropriate register name #define's. Again, not this
patches responsibility. Either way, it should be consistent.

Elsewhere in the patch series there are comments that refer to EBX etc
while the code looks like exx[1]. It hasn't been a problem up to now
since there were only a couple places that used cpuid. But this
patchset adds quite a few more, so now seems like a good time to make
it more readable with register name symbols.

- /* All these features depend on OSXSAVE */
- if (exx[2] & (1 << 27))
- {
- uint32 xcr0_val = 0;
-
- /* second cpuid call on leaf 7 to check extended AVX-512 support */
-
- memset(exx, 0, 4 * sizeof(exx[0]));
+ /* Use leaf 7 to check TSC_ADJUST and extended AVX-512 support */
+ memset(exx, 0, 4 * sizeof(exx[0]));

 #if defined(HAVE__GET_CPUID_COUNT)
- __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]);
+ __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]);
 #elif defined(HAVE__CPUIDEX)
- __cpuidex(exx, 7, 0);
+ __cpuidex(exx, 7, 0);
 #endif

+ X86Features[PG_TSC_ADJUST] = (exx[1] & (1 << 1)) != 0;
+
+ /* All relevant AVX-512 features depend on OSXSAVE */

This comment was deliberately general, since other things can depend
on it as well (e.g. AVX2 will be committed soon). Since this patch is
adding some calls with different leaves, maybe we can say something
like "/* leaf 7 features that depend on OSXSAVE */"?

+ if (exx[2] & (1 << 27))
+ {
+ uint32 xcr0_val = 0;
+

By moving the call to leaf 7 up, the results of leaf 1 just got blown
away, so isn't OXSAVE support now broken? Maybe we actually want 2
separate arrays? (Like "regs" and "ext_regs") That'll also avoid
having to memset.

+ level_type = (exx[2] >> 8) & 0xff;

The comment says "ECX[15:8]". This looks like ECX[7:0] but maybe I'm
missing something.

Looking at the above, the following could be in a separate patch(es)
to reduce the footprint and increase readability of 0004:

* add pg_cpuid
* call leaf 7 on separate array before checking OSXSAVE
* possibly the other cleanups mentioned above.

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-03-06T19:22:27Z

Hi,

On 2026-03-03 10:22:42 -0800, Lukas Fittl wrote:
> > But if we read files anyway, wouldn't just using
> >   /sys/devices/system/cpu/cpu0/cpufreq/base_frequency
> > work?
>
> I tested this just now on an Azure VM (Standard D2s v3), and its
> close, but unfortunately CPU frequency doesn't match the TSC frequency
> (cpuinfo_max_freq is 2800000, scaling_cur_freq is 2496279, and TSC
> frequency via MSR is 2793438 -- note that I didn't have base_frequency
> on this VM). My understanding is that the TSC clock is virtualized in
> HyperV and does not directly match the CPU frequency.

:(

It seems quite ridiculous that there's no cpuid to get the frequency of both
virtualized and "real" tsc.

> I'm also happy to take this out again - maybe we can get the
> HyperV/Azure Linux folks to improve the Kernel side here to pass down
> the TSC frequency without needing the MSR, and just not support it for
> now.

Yea, this doesn't seem worth it, it won't get used this way, I think.

> An alternate idea could be to allow overriding the TSC frequency via a
> GUC - then one could use the root user (or a setuid program) to get
> the TSC frequency on Azure/HyperV via the MSR and pass it to Postgres
> at start. But not sure that's worth the trouble, since it won't help
> with environments that don't have a reliable TSC (e.g. Virtualbox, I
> think).

I don't think manually specifying it makes sense either.

But maybe we should just do the stupid thing and figure out the multiplier as
such:

  ns_to_cycles = tsc_via_rdtsc / to_ns(clock_gettime(CLOCK_BOOTTIME))

in some quick experiments that ends up with a very good estimate.  There would
have to be an awful long gap between the rdtsc and clock_gettime() computation
for the frequency to be meaningfully inaccurate.

I was worried for a moment that the there would be issues with the tsc counter
overflowing after a long uptime, but that doesn't seem a real issue if I did
the math right (at a 10GHz tsc freq the time to overflow would be ~58 years).

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-06T19:47:10Z

On Fri, Mar 6, 2026 at 11:22 AM Andres Freund <andres@anarazel.de> wrote:
> On 2026-03-03 10:22:42 -0800, Lukas Fittl wrote:
> > > But if we read files anyway, wouldn't just using
> > >   /sys/devices/system/cpu/cpu0/cpufreq/base_frequency
> > > work?
> >
> > I tested this just now on an Azure VM (Standard D2s v3), and its
> > close, but unfortunately CPU frequency doesn't match the TSC frequency
> > (cpuinfo_max_freq is 2800000, scaling_cur_freq is 2496279, and TSC
> > frequency via MSR is 2793438 -- note that I didn't have base_frequency
> > on this VM). My understanding is that the TSC clock is virtualized in
> > HyperV and does not directly match the CPU frequency.
>
> :(
>
> It seems quite ridiculous that there's no cpuid to get the frequency of both
> virtualized and "real" tsc.

Agreed. I did find some efforts where folks tried to expose this
information to user space inspired by how Google apparently does it in
their internal kernel to support user space TSC programs (see [0] and
[1]), but I don't think that went anywhere, though I don't follow the
Kernel mailinglists, so maybe someone is working on that somewhere.

> > I'm also happy to take this out again - maybe we can get the
> > HyperV/Azure Linux folks to improve the Kernel side here to pass down
> > the TSC frequency without needing the MSR, and just not support it for
> > now.
>
> Yea, this doesn't seem worth it, it won't get used this way, I think.

Ack.

> But maybe we should just do the stupid thing and figure out the multiplier as
> such:
>
>   ns_to_cycles = tsc_via_rdtsc / to_ns(clock_gettime(CLOCK_BOOTTIME))
>
> in some quick experiments that ends up with a very good estimate.  There would
> have to be an awful long gap between the rdtsc and clock_gettime() computation
> for the frequency to be meaningfully inaccurate.

I think as long as the TSC counter and the clock boottime start at the
same moment, that should work. But I'm not sure if we can rely on that
to be the case in virtualized environments? I can do some more
testing.

Alternatively, we could consider doing it like the Kernel does it for
its calibration loop, and wait 1 second of wall time, and then see how
far the TSC counter has advanced.

FWIW, I ended up getting an x86 machine to be able to test these
things better, and got myself an AMD CPU. Well, turns out that my
non-virtualized AMD CPU ("AMD Ryzen™ AI Max+ 395") does not provide
the TSC frequency via CPUID, at all :(

Instead on newer AMD CPUs you can use an MSR to get the TSC frequency,
see [2] -- or calibrate against another time source (which I think we
should do).

Thanks,
Lukas

[0]: https://github.com/trailofbits/tsc_freq_khz
[1]: https://blog.trailofbits.com/2019/10/03/tsc-frequency-for-all-better-profiling-and-benchmarking/
[2]: https://github.com/jdmccalpin/low-overhead-timers/issues/1#issuecomment-1668472748

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-03-08T16:39:47Z

Hi,

On 2026-03-06 11:47:10 -0800, Lukas Fittl wrote:
> > But maybe we should just do the stupid thing and figure out the multiplier as
> > such:
> >
> >   ns_to_cycles = tsc_via_rdtsc / to_ns(clock_gettime(CLOCK_BOOTTIME))
> >
> > in some quick experiments that ends up with a very good estimate.  There would
> > have to be an awful long gap between the rdtsc and clock_gettime() computation
> > for the frequency to be meaningfully inaccurate.
> 
> I think as long as the TSC counter and the clock boottime start at the
> same moment, that should work. But I'm not sure if we can rely on that
> to be the case in virtualized environments? I can do some more
> testing.

I did some testing, and unfortunately it's not good enough. There are several
issues:

- The tsc counter starts earlier than the OS, by enough to make counter
  initially not quite right. It's not that bad on a laptop with a quick boot
  time, but on a server with slower bios time initialization (e.g. due to
  training of more memory) it's worse.

- If the server is rebooted not through a hard reset (the typical default),
  but through something like kexec (which does not go through bios again), the
  tsc counter is not reset.



> Alternatively, we could consider doing it like the Kernel does it for
> its calibration loop, and wait 1 second of wall time, and then see how
> far the TSC counter has advanced.

Yea, I think we need a calibration loop, unfortunately. But I think it should
be doable to make it a lot quicker than waiting one second.  I'm thinking of
something like a loop that measures the the clock cycles and relative time
(using clock_gettime()) since the start and does so until the frequency
estimate predicts the time results closely.  I think should be a few 10s of
milliseconds at most.


> FWIW, I ended up getting an x86 machine to be able to test these
> things better, and got myself an AMD CPU.

Dedication...


> Well, turns out that my
> non-virtualized AMD CPU ("AMD Ryzen™ AI Max+ 395") does not provide
> the TSC frequency via CPUID, at all :(

I can repro that on a somewhat older Zen 4 (7840U) laptop CPU.


> Instead on newer AMD CPUs you can use an MSR to get the TSC frequency,
> see [2]

:(

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-10T10:04:02Z

On Wed, Mar 4, 2026 at 5:50 AM John Naylor <johncnaylorls@gmail.com> wrote:
> -#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT)
> -#include <cpuid.h>
> -#endif
> -
> -#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX)
> +#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) ||
> defined(HAVE__CPUIDEX)
> +#if defined(_MSC_VER)
>  #include <intrin.h>
> +#else
> +#include <cpuid.h>
> +#endif /* defined(_MSC_VER) */
>
> What if HAVE__CPUID is defined but not HAVE__CPUIDEX? Is that possible?

Good point - I think technically our configure/meson checks allow the
case where only HAVE__CPUID is set.

> v10-0004:
>
> + pg_cpuid(1, exx);
>
> Since some leaves later on use hex values, so maybe they should all be
> hex for consistency.

Yep, agreed, will adjust.

>
> - /* second cpuid call on leaf 7 to check extended AVX-512 support */
> -
> - memset(exx, 0, 4 * sizeof(exx[0]));
> + /* Use leaf 7 to check TSC_ADJUST and extended AVX-512 support */
> + memset(exx, 0, 4 * sizeof(exx[0]));
>
> It wasn't previously stated outright, but I think the important point
> here is we need to memset before this call so that platforms without
> extended cpuid don't have stale values. It's not important at this
> point what the goal of the call is. [thinking] Actually it's worse,
> see below.

Hm, yeah, I misread the logic there - I think using more helper
functions will help here.

>
> Speaking of initialization, I noticed we historically used "foo =
> {0,0,0,0}" (lack of C99?), but "foo = {0}" is more concise. Not this
> patch's responsibility, but it does add a few more of this pattern. It
> would look nicer if we changed all those at some point.

Agreed, done in the next iteration.

> + X86Features[PG_RDTSCP] = (exx[3] & (1 << 27)) != 0;
>
> The style I've used elsewhere for setting features is
>
> X86Features[PG_SSE4_2] = exx[2] >> 20 & 1;
>
> But maybe it'd actually be better to have a function that can be called like
>
> x86_set_feature(PG_RDTSCP, EDX, 27);

I think your existing style is fine, I'll adjust my additions for consistency.

> ...with the appropriate register name #define's. Again, not this
> patches responsibility. Either way, it should be consistent.
>
> Elsewhere in the patch series there are comments that refer to EBX etc
> while the code looks like exx[1]. It hasn't been a problem up to now
> since there were only a couple places that used cpuid. But this
> patchset adds quite a few more, so now seems like a good time to make
> it more readable with register name symbols.

I'm not a fan of using macros for this, but what if we define
ourselves a struct like this:

typedef struct CPUIDResult
{
    unsigned int eax;
    unsigned int ebx;
    unsigned int ecx;
    unsigned int edx;
} CPUIDResult;

Then the code reads like:

pg_cpuid(0x80000001, &r);
X86Features[PG_RDTSCP] = r.edx >> 27 & 1;

>
> - /* All these features depend on OSXSAVE */
> ...
>
> + /* All relevant AVX-512 features depend on OSXSAVE */
>
> This comment was deliberately general, since other things can depend
> on it as well (e.g. AVX2 will be committed soon). Since this patch is
> adding some calls with different leaves, maybe we can say something
> like "/* leaf 7 features that depend on OSXSAVE */"?

Yeah, good point, will adjust.

>
> + if (exx[2] & (1 << 27))
> + {
> + uint32 xcr0_val = 0;
> +
>
> By moving the call to leaf 7 up, the results of leaf 1 just got blown
> away, so isn't OXSAVE support now broken? Maybe we actually want 2
> separate arrays? (Like "regs" and "ext_regs") That'll also avoid
> having to memset.

Yeah, I think using two result variables make sense here.
Alternatively we could save the result of the OXSAVE check in a
boolean.

> + level_type = (exx[2] >> 8) & 0xff;
>
> The comment says "ECX[15:8]". This looks like ECX[7:0] but maybe I'm
> missing something.

I think the code is right - bits 15 to 8 define the level type, and so
we shift that right by 8 bits, and then apply 0xff to drop anything to
the left of that.

> Looking at the above, the following could be in a separate patch(es)
> to reduce the footprint and increase readability of 0004:
>
> * add pg_cpuid
> * call leaf 7 on separate array before checking OSXSAVE
> * possibly the other cleanups mentioned above.

Yeah, makes sense to do that in an earlier patch - will post an
updated patch set later today that does that in a preparatory patch.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-11T09:11:17Z

Hi,

Attached v11, with the following changes:

0001 is a new patch that implements the refactorings of the CPUID code
suggested by John.
0002 is the existing patch for supporting using __cpuidex directly
(needed by the TSC hypervisor frequency code).
0003 is the existing patch as before to optimize pg_test_timing.

0004 is almost identical to the previous patch (v10/0003) that adds
the ticks to NS conversion, with a small improvement to use an
explicit define (PG_INSTR_TICKS_TO_NS) that controls whether we go
into the complex pg_ticks_to_ns logic at all.

0005 is the TSC patch with the following changes:

- Dropped the HyperV MSR read again, per Andres feedback
- Added a TSC calibration loop that is used if we can't get the
frequency from CPUID. This is based on a script that Andres shared
off-list, and works both on HyperV as well as my bare metal AMD CPU.
Note that we don't utilize this on Windows to avoid the delay for the
calibration to converge (< 50ms, typically less than 1ms) penalizing
connection start (since we don't get the tsc frequency global from
postmaster before the fork)
- Moved the GUC logic to instrument.c, because we shouldn't be
defining GUCs in a file that's built with front-end programs

Its worth noting that I have not yet included a way to pass debug
information back to the user (e.g. when the TSC calibration didn't
converge, the TSC is not invariant, etc), as Jakub suggested
previously - with the TSC calibration code in the picture I'm less
sure its really needed, since e.g. looking at the "cpuid" program will
tell you whether calibration runs or not, and then you could infer
that the calibration failed if you didn't get a usable TSC reported by
pg_test_timing.

0006 is the existing patch as before to add pg_test_timing debug output.

0007 is a new patch that shows how we could expand this to also be
used for ARM, by calling CNTVCT_EL0. I'm mainly adding this because I
think its the main evolution of this that we haven't talked about that
much yet, and even if we do this in a later release cycle it'll help
refine the design. This worked as expected for me on an AWS Graviton
instance, but failed on an Apple Silicon M3 due to quirks with its
Efficiency vs Performance core - dealt with in the patch by not using
the generic timer directly when we're on a homogeneous core system.

Looking at how an ARM implementation could work does make me wonder
one thing in general: Maybe we shouldn't be using the term "tsc" for
"timing_clock_source" (and internal defines), since "TSC" is an x86
term, that doesn't really make sense when we expand this to ARM in a
future release. Maybe we should use a generic name, like "hwtimer",
"direct" or "hardware"?

On Sun, Mar 8, 2026 at 9:39 AM Andres Freund <andres@anarazel.de> wrote:
> > Alternatively, we could consider doing it like the Kernel does it for
> > its calibration loop, and wait 1 second of wall time, and then see how
> > far the TSC counter has advanced.
>
> Yea, I think we need a calibration loop, unfortunately. But I think it should
> be doable to make it a lot quicker than waiting one second.  I'm thinking of
> something like a loop that measures the the clock cycles and relative time
> (using clock_gettime()) since the start and does so until the frequency
> estimate predicts the time results closely.  I think should be a few 10s of
> milliseconds at most.

That is implemented now in 0005, based on the script you shared off
list, which I hopefully translated correctly to the Postgres source.
The one part I wasn't sure about is whether we want to use RDTSCP for
the calibration, or RDTSC+LFENCE like you had it in your script (which
is closer to what the abseil library does, that I mentioned upthread)
- for now I went with RDTSCP to keep it simple.

We run up to 1 million RDTSCP instructions, for at most 50ms, and
terminate once the frequency stays stable for at least 3 iterations.
In testing this converges pretty quickly in practice (<1ms) and
closely matches the reported TSC frequency by the Linux kernel.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-11T19:52:35Z

+ /*
+ * Once freq_khz / prev_freq_khz is small, check if it stays that way.
+ * If it does for long enough, we've got a winner frequency.
+ */
+ if (prev_freq_khz != 0 && fabs(freq_khz / prev_freq_khz) < 1.0001)

Doesn't this accept any frequency decrease as stable? Shouldn't it
instead use something like

fabs(1 - freq_khz / prev_freq_khz) < 0.0001

which works in both directions?


- total_time = duration > 0 ? duration * INT64CONST(1000000000) : 0;
+ INSTR_TIME_SET_NANOSEC(duration_time, duration > 0 ? duration * NS_PER_S : 0);

  INSTR_TIME_SET_CURRENT(start_time);
- cur = INSTR_TIME_GET_NANOSEC(start_time);
+ cur = start_time;

- while (time_elapsed < total_time)
+ end_time = start_time;
+ INSTR_TIME_ADD(end_time, duration_time);

Is this correct? Won't INSTR_TIME_ADD add nanoseconds (duration_time)
to TSC ticks (end_time)? INSTR_TIME_GET_NANOSEC uses pg_ticks_to_ns,
but INSTR_TIME_SET_NANOSEC simply stores the value without any
conversion function.

Also, if the clock source is switched during statement we get totally
incorrect results. I don't think this has any real consequences in the
current use case, but maybe it's worth at least mentioning somewhere?

CREATE FUNCTION switch_timing_source_and_sleep()
RETURNS void AS $$
BEGIN
    SET timing_clock_source = 'system';
    PERFORM pg_sleep(0.01);
END;
$$ LANGUAGE plpgsql;

SET timing_clock_source = 'tsc';
EXPLAIN ANALYZE SELECT switch_timing_source_and_sleep();

On Wed, Mar 11, 2026 at 7:46 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> Hi,
>
> Attached v11, with the following changes:
>
> 0001 is a new patch that implements the refactorings of the CPUID code
> suggested by John.
> 0002 is the existing patch for supporting using __cpuidex directly
> (needed by the TSC hypervisor frequency code).
> 0003 is the existing patch as before to optimize pg_test_timing.
>
> 0004 is almost identical to the previous patch (v10/0003) that adds
> the ticks to NS conversion, with a small improvement to use an
> explicit define (PG_INSTR_TICKS_TO_NS) that controls whether we go
> into the complex pg_ticks_to_ns logic at all.
>
> 0005 is the TSC patch with the following changes:
>
> - Dropped the HyperV MSR read again, per Andres feedback
> - Added a TSC calibration loop that is used if we can't get the
> frequency from CPUID. This is based on a script that Andres shared
> off-list, and works both on HyperV as well as my bare metal AMD CPU.
> Note that we don't utilize this on Windows to avoid the delay for the
> calibration to converge (< 50ms, typically less than 1ms) penalizing
> connection start (since we don't get the tsc frequency global from
> postmaster before the fork)
> - Moved the GUC logic to instrument.c, because we shouldn't be
> defining GUCs in a file that's built with front-end programs
>
> Its worth noting that I have not yet included a way to pass debug
> information back to the user (e.g. when the TSC calibration didn't
> converge, the TSC is not invariant, etc), as Jakub suggested
> previously - with the TSC calibration code in the picture I'm less
> sure its really needed, since e.g. looking at the "cpuid" program will
> tell you whether calibration runs or not, and then you could infer
> that the calibration failed if you didn't get a usable TSC reported by
> pg_test_timing.
>
> 0006 is the existing patch as before to add pg_test_timing debug output.
>
> 0007 is a new patch that shows how we could expand this to also be
> used for ARM, by calling CNTVCT_EL0. I'm mainly adding this because I
> think its the main evolution of this that we haven't talked about that
> much yet, and even if we do this in a later release cycle it'll help
> refine the design. This worked as expected for me on an AWS Graviton
> instance, but failed on an Apple Silicon M3 due to quirks with its
> Efficiency vs Performance core - dealt with in the patch by not using
> the generic timer directly when we're on a homogeneous core system.
>
> Looking at how an ARM implementation could work does make me wonder
> one thing in general: Maybe we shouldn't be using the term "tsc" for
> "timing_clock_source" (and internal defines), since "TSC" is an x86
> term, that doesn't really make sense when we expand this to ARM in a
> future release. Maybe we should use a generic name, like "hwtimer",
> "direct" or "hardware"?
>
> On Sun, Mar 8, 2026 at 9:39 AM Andres Freund <andres@anarazel.de> wrote:
> > > Alternatively, we could consider doing it like the Kernel does it for
> > > its calibration loop, and wait 1 second of wall time, and then see how
> > > far the TSC counter has advanced.
> >
> > Yea, I think we need a calibration loop, unfortunately. But I think it should
> > be doable to make it a lot quicker than waiting one second.  I'm thinking of
> > something like a loop that measures the the clock cycles and relative time
> > (using clock_gettime()) since the start and does so until the frequency
> > estimate predicts the time results closely.  I think should be a few 10s of
> > milliseconds at most.
>
> That is implemented now in 0005, based on the script you shared off
> list, which I hopefully translated correctly to the Postgres source.
> The one part I wasn't sure about is whether we want to use RDTSCP for
> the calibration, or RDTSC+LFENCE like you had it in your script (which
> is closer to what the abseil library does, that I mentioned upthread)
> - for now I went with RDTSCP to keep it simple.
>
> We run up to 1 million RDTSCP instructions, for at most 50ms, and
> terminate once the frequency stays stable for at least 3 iterations.
> In testing this converges pretty quickly in practice (<1ms) and
> closely matches the reported TSC frequency by the Linux kernel.
>
> Thanks,
> Lukas
>
> --
> Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-11T21:02:11Z

Hi Zsolt,

Thanks for reviewing!

On Wed, Mar 11, 2026 at 12:52 PM Zsolt Parragi
<zsolt.parragi@percona.com> wrote:
>
> + /*
> + * Once freq_khz / prev_freq_khz is small, check if it stays that way.
> + * If it does for long enough, we've got a winner frequency.
> + */
> + if (prev_freq_khz != 0 && fabs(freq_khz / prev_freq_khz) < 1.0001)
>
> Doesn't this accept any frequency decrease as stable? Shouldn't it
> instead use something like
>
> fabs(1 - freq_khz / prev_freq_khz) < 0.0001
>
> which works in both directions?

Yeah, good point - I think you're correct.

I'll wait for Andres take another look at the TSC calibration logic as
written in the patch to see if there are further suggestions on how to
adjust, but otherwise will make that change in the next revision.

> - total_time = duration > 0 ? duration * INT64CONST(1000000000) : 0;
> + INSTR_TIME_SET_NANOSEC(duration_time, duration > 0 ? duration * NS_PER_S : 0);
>
>   INSTR_TIME_SET_CURRENT(start_time);
> - cur = INSTR_TIME_GET_NANOSEC(start_time);
> + cur = start_time;
>
> - while (time_elapsed < total_time)
> + end_time = start_time;
> + INSTR_TIME_ADD(end_time, duration_time);
>
> Is this correct? Won't INSTR_TIME_ADD add nanoseconds (duration_time)
> to TSC ticks (end_time)? INSTR_TIME_GET_NANOSEC uses pg_ticks_to_ns,
> but INSTR_TIME_SET_NANOSEC simply stores the value without any
> conversion function.

Ah, good catch - we basically have to do the inverse of pg_ticks_to_ns
when PG_INSTR_TICKS_TO_NS is active and ticks_per_ns_scaled is
non-zero.

I think having INSTR_TIME_SET_NANOSEC around to be able to set a
"target time" that then gets compared against in a loop seems more
generally useful, e.g. it could allow using the instr_time
infrastructure for cases like the one reported at [0], so I'll go add
that in the next revision, unless I hear feedback the other way.

> Also, if the clock source is switched during statement we get totally
> incorrect results. I don't think this has any real consequences in the
> current use case, but maybe it's worth at least mentioning somewhere?

Yeah, that's basically by design, because we can't invalidate prior
instr_time values when the timing clock source changes. I don't think
this is a big problem in practice, since changing the timing source
would be a rare thing to do - I think its mainly useful to do so
interactively when confirming numbers in case the values look off with
TSC.

Maybe we should emit a warning from the assign function when called
from inside a transaction? (since that'd be a clear cut case where its
not a good idea to do the SET)

Alternatively documenting this explicitly sounds good. Were you
thinking of user-facing documentation, or more on the code side?

Thanks,
Lukas

[0]: https://www.postgresql.org/message-id/flat/CAJhEC07OK8J7tLUbyiccnuOXRE7UKxBNqD2-pLfeFXa%3DtBoWtw%40mail.gmail.com

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-12T21:55:02Z

> Were you
> thinking of user-facing documentation, or more on the code side?

I'm not 100% sure, maybe code side is enough. With the current patch,
the behavior is only visible if somebody intentionally tries to make
it visible, and even then it's not really harmful.

In the code it's definitely worth mentioning since it's not trivial,
and stating it also makes it clear that it's not a bug/oversight. It
could be also useful info if another patch later wants to also use
timing_clock_source to keep in mind that users can change it
dynamically.

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-03-16T00:49:17Z

On Tue, Mar 10, 2026 at 5:04 PM Lukas Fittl <lukas@fittl.com> wrote:

> > Elsewhere in the patch series there are comments that refer to EBX etc
> > while the code looks like exx[1]. It hasn't been a problem up to now
> > since there were only a couple places that used cpuid. But this
> > patchset adds quite a few more, so now seems like a good time to make
> > it more readable with register name symbols.
>
> I'm not a fan of using macros for this, but what if we define
> ourselves a struct like this:
>
> typedef struct CPUIDResult
> {
>     unsigned int eax;
>     unsigned int ebx;
>     unsigned int ecx;
>     unsigned int edx;
> } CPUIDResult;
>
> Then the code reads like:
>
> pg_cpuid(0x80000001, &r);
> X86Features[PG_RDTSCP] = r.edx >> 27 & 1;

One objection is that __cpuid() takes an array of 4 integers as an
argument. I think it would technically happen to work to pass a
pointer to this struct, but it seems the wrong thing to do. If you're
not a fan of macros, the other way would be an enum of indices (and
named with all caps).

> > + if (exx[2] & (1 << 27))
> > + {
> > + uint32 xcr0_val = 0;
> > +
> >
> > By moving the call to leaf 7 up, the results of leaf 1 just got blown
> > away, so isn't OXSAVE support now broken? Maybe we actually want 2
> > separate arrays? (Like "regs" and "ext_regs") That'll also avoid
> > having to memset.
>
> Yeah, I think using two result variables make sense here.
> Alternatively we could save the result of the OXSAVE check in a
> boolean.

Actually I like the idea of a boolean better, since that would be less
churn, and maybe simpler to reason about. 0005 overwrites one of the
variables in subsequent calls, so it could be confusing for future
additions as to which one to use.

> > + level_type = (exx[2] >> 8) & 0xff;
> >
> > The comment says "ECX[15:8]". This looks like ECX[7:0] but maybe I'm
> > missing something.
>
> I think the code is right - bits 15 to 8 define the level type, and so
> we shift that right by 8 bits, and then apply 0xff to drop anything to
> the left of that.

Yes, I must have missed the shift, thanks.

v11-0001:

+static inline bool
+pg_cpuid_subleaf(int leaf, int subleaf, CPUIDResult *r)
+{
+#if defined(HAVE__GET_CPUID_COUNT)
+  return __get_cpuid_count(leaf, subleaf, &r->eax, &r->ebx, &r->ecx,
&r->edx) == 1;
+#elif defined(HAVE__CPUIDEX)
+  __cpuidex((int *) r, leaf, subleaf);
+  return true;
+#else
+  memset(r, 0, sizeof(CPUIDResult));
+  return false;
+#endif
+}

This needs a comment to explain the return value.


--
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-22T18:13:11Z

Hi John,

On Sun, Mar 15, 2026 at 5:49 PM John Naylor <johncnaylorls@gmail.com> wrote:
>
> On Tue, Mar 10, 2026 at 5:04 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> > > Elsewhere in the patch series there are comments that refer to EBX etc
> > > while the code looks like exx[1]. It hasn't been a problem up to now
> > > since there were only a couple places that used cpuid. But this
> > > patchset adds quite a few more, so now seems like a good time to make
> > > it more readable with register name symbols.
> >
> > I'm not a fan of using macros for this, but what if we define
> > ourselves a struct like this:
> >
> > typedef struct CPUIDResult
> > {
> >     unsigned int eax;
> >     unsigned int ebx;
> >     unsigned int ecx;
> >     unsigned int edx;
> > } CPUIDResult;
> >
> > Then the code reads like:
> >
> > pg_cpuid(0x80000001, &r);
> > X86Features[PG_RDTSCP] = r.edx >> 27 & 1;
>
> One objection is that __cpuid() takes an array of 4 integers as an
> argument. I think it would technically happen to work to pass a
> pointer to this struct, but it seems the wrong thing to do. If you're
> not a fan of macros, the other way would be an enum of indices (and
> named with all caps).

How about we deal with this in the pg_cpuid function by having
__cpuid() write into a separate array and then assign that to
CPUIDResult? See the attached 0001 patch.

I think the struct fields are the clearest approach here, but if you
disagree let me know and I can adjust further.

> > > + if (exx[2] & (1 << 27))
> > > + {
> > > + uint32 xcr0_val = 0;
> > > +
> > >
> > > By moving the call to leaf 7 up, the results of leaf 1 just got blown
> > > away, so isn't OXSAVE support now broken? Maybe we actually want 2
> > > separate arrays? (Like "regs" and "ext_regs") That'll also avoid
> > > having to memset.
> >
> > Yeah, I think using two result variables make sense here.
> > Alternatively we could save the result of the OXSAVE check in a
> > boolean.
>
> Actually I like the idea of a boolean better, since that would be less
> churn, and maybe simpler to reason about. 0005 overwrites one of the
> variables in subsequent calls, so it could be confusing for future
> additions as to which one to use.

Yeah, makes sense - switched it to a boolean in the 0005 patch that
adds the TSC logic.

> v11-0001:
>
> +static inline bool
> +pg_cpuid_subleaf(int leaf, int subleaf, CPUIDResult *r)
> +{
> +#if defined(HAVE__GET_CPUID_COUNT)
> +  return __get_cpuid_count(leaf, subleaf, &r->eax, &r->ebx, &r->ecx,
> &r->edx) == 1;
> +#elif defined(HAVE__CPUIDEX)
> +  __cpuidex((int *) r, leaf, subleaf);
> +  return true;
> +#else
> +  memset(r, 0, sizeof(CPUIDResult));
> +  return false;
> +#endif
> +}
>
> This needs a comment to explain the return value.

Good point, added a comment.

I think that should address all open feedback on the 0001 patch.

Attached v12, which also has these additional changes:

v12/0003 (pg_test_timing: Reduce per-loop overhead)

- Fixed the ns to ticks translation issue, per Zsolt's feedback
- Reworked this to use a INSTR_TIME_ADD_NANOSEC macro (instead of
INSTR_TIME_SET_NANOSEC) - I think that's best for the intent of "set a
target time to compare against using a time interval", and avoids
callers passing in fixed timestamps that'd be more expensive to
convert

v12/0004 (Streamline ticks to nanosecond conversion across platforms)

- Moved pg_initialize_timing to InitProcessGlobals instead of main. I
had previously placed this in main because it might have helped a
different TSC calibration technique that could have benefited from a
distance between two function calls, but the calibration function
appears fast enough that we can just run it all at once.
- Add assert that pg_initialize_timing was called before INSTR* macros
are called, and add a missing pg_initialize_timing in pg_regress, as
well as correct where we call it for pgbench

v12/0005 (Main TSC patch)

- Clarify that TSC frequency can technically change between instr_time
set and read (in code, next to the relevant variables), per Zsolt's
feedback
- Add pg_cpuidex helper that can be used to call _cpuidex (and
intentionally not __get_cpuid_count) for getting VM Hypervisor
information

TSC calibration improvements:
- Pass TSC frequency to child processes for EXEC_BACKEND, this makes
TSC calibration usable on Windows, per off-list suggestion from Andres
- To support the EXEC_BACKEND change, refactor tsc_frequency_khz to
have a sentinel value of -1 indicating the TSC was not initialized, vs
0 indicating it was initialized but isn't usable, and drop
has_usable_tsc variable
- Fix the TSC calibration logic to handle frequency decrease, per
Zsolt's feedback
- Run TSC calibration only once GUCs are being set, since we don't
need it earlier and we can simplify the client program logic that way
(if you don't call pg_set_timing_clock_source you won't get TSC)
- Based on further testing, measure the frequency only every 100
iterations, and require at least 10 stable cycles (i.e. effective min
iterations is 1000, max is 1 million as before) -- this improves the
accuracy on my two test machines (AMD bare metal + Azure HyperV)

v12/0006 (pg_test_timing patch)

- Show TSC frequency in pg_test_timing output

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-03-23T07:00:50Z

On Mon, Mar 23, 2026 at 1:13 AM Lukas Fittl <lukas@fittl.com> wrote:
>
> Hi John,
>
> On Sun, Mar 15, 2026 at 5:49 PM John Naylor <johncnaylorls@gmail.com> wrote:
> >
> > On Tue, Mar 10, 2026 at 5:04 PM Lukas Fittl <lukas@fittl.com> wrote:
> >
> > > > Elsewhere in the patch series there are comments that refer to EBX etc
> > > > while the code looks like exx[1]. It hasn't been a problem up to now
> > > > since there were only a couple places that used cpuid. But this
> > > > patchset adds quite a few more, so now seems like a good time to make
> > > > it more readable with register name symbols.
> > >
> > > I'm not a fan of using macros for this, but what if we define
> > > ourselves a struct like this:
> > >
> > > typedef struct CPUIDResult
> > > {
> > >     unsigned int eax;
> > >     unsigned int ebx;
> > >     unsigned int ecx;
> > >     unsigned int edx;
> > > } CPUIDResult;
> > >
> > > Then the code reads like:
> > >
> > > pg_cpuid(0x80000001, &r);
> > > X86Features[PG_RDTSCP] = r.edx >> 27 & 1;
> >
> > One objection is that __cpuid() takes an array of 4 integers as an
> > argument. I think it would technically happen to work to pass a
> > pointer to this struct, but it seems the wrong thing to do. If you're
> > not a fan of macros, the other way would be an enum of indices (and
> > named with all caps).
>
> How about we deal with this in the pg_cpuid function by having
> __cpuid() write into a separate array and then assign that to
> CPUIDResult? See the attached 0001 patch.
>
> I think the struct fields are the clearest approach here, but if you
> disagree let me know and I can adjust further.

I still don't see the value of the struct, since it doesn't get rid of
the array, which we still need for some APIs, so it's just adding
indirection:

+static inline void
+pg_cpuid(int leaf, CPUIDResult *r)
+{
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(leaf, &r->eax, &r->ebx, &r->ecx, &r->edx);
+#elif defined(HAVE__CPUID)
+ int exx[4] = {0};
+
+ __cpuid(exx, leaf);
+ r->eax = exx[0];
+ r->ebx = exx[1];
+ r->ecx = exx[2];
+ r->edx = exx[3];

+static inline bool
+pg_cpuid_subleaf(int leaf, int subleaf, CPUIDResult *r)
+{
+#if defined(HAVE__GET_CPUID_COUNT)
+ return __get_cpuid_count(leaf, subleaf, &r->eax, &r->ebx, &r->ecx,
&r->edx) == 1;
+#elif defined(HAVE__CPUIDEX)
+ int exx[4] = {0};
+
+ __cpuidex(exx, leaf, subleaf);
+ r->eax = exx[0];
+ r->ebx = exx[1];
+ r->ecx = exx[2];
+ r->edx = exx[3];
+ return true;

It's not that bad that the hard-coded indexes are in two places, but
it's also not necessary. I think "#define EAX 0 ...etc" is a fine,
straightforward increase in readability compared to what we have now,
and I don't see any downside. I suppose one argument in favor of the
struct is that it avoids declaring variables of type
array-of-4-unsigned in multiple places, but I think the array is fine,
and adding a new typedef is additional cognitive friction.

Speaking of signedness, why is the array of ints sometimes signed and
sometimes unsigned?

> > This needs a comment to explain the return value.
>
> Good point, added a comment.

+ * Returns false if the CPUID leaf/subleaf is not supported.

Okay, thanks. I'd suggest using "true if X is supported" or "true if X
is supported, false otherwise" phrasing.

-- 
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

David Geier <geidav.pg@gmail.com> — 2026-03-23T08:00:34Z

Hi Lukas!

Thanks for pushing this forward.

On 22.03.2026 19:13, Lukas Fittl wrote:

> TSC calibration improvements:
> - Pass TSC frequency to child processes for EXEC_BACKEND, this makes
> TSC calibration usable on Windows, per off-list suggestion from Andres
> - To support the EXEC_BACKEND change, refactor tsc_frequency_khz to
> have a sentinel value of -1 indicating the TSC was not initialized, vs
> 0 indicating it was initialized but isn't usable, and drop
> has_usable_tsc variable

Have you tested the latest patch set on Windows? Let me know if you want
me to do that.

--
David Geier

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-24T06:09:57Z

Hi David,

On Mon, Mar 23, 2026 at 1:00 AM David Geier <geidav.pg@gmail.com> wrote:
> > TSC calibration improvements:
> > - Pass TSC frequency to child processes for EXEC_BACKEND, this makes
> > TSC calibration usable on Windows, per off-list suggestion from Andres
> > - To support the EXEC_BACKEND change, refactor tsc_frequency_khz to
> > have a sentinel value of -1 indicating the TSC was not initialized, vs
> > 0 indicating it was initialized but isn't usable, and drop
> > has_usable_tsc variable
>
> Have you tested the latest patch set on Windows? Let me know if you want
> me to do that.

Not outside of CI - so if you could do testing on Windows, that'd be great!

FWIW, CI does now exercise the TSC calibration logic on Windows, which
is part why the last patch set added the handling of EXEC_BACKEND to
pass down the frequency to the client connections, because I otherwise
saw this impacting Windows CI runtimes - they're normal with the
version posted.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-24T07:16:07Z

Hi John,

On Mon, Mar 23, 2026 at 12:01 AM John Naylor <johncnaylorls@gmail.com> wrote:
> It's not that bad that the hard-coded indexes are in two places, but
> it's also not necessary. I think "#define EAX 0 ...etc" is a fine,
> straightforward increase in readability compared to what we have now,
> and I don't see any downside. I suppose one argument in favor of the
> struct is that it avoids declaring variables of type
> array-of-4-unsigned in multiple places, but I think the array is fine,
> and adding a new typedef is additional cognitive friction.

Sounds good, lets do it that way - adjusted to use macros instead of a struct.

> Speaking of signedness, why is the array of ints sometimes signed and
> sometimes unsigned?

The signedness is a MSVC-ism - I think its reasonable for us to work
with unsigned integers in our code, and pass them by casting to
__cpuid/__cpuidex (the MSVC variants).

> + * Returns false if the CPUID leaf/subleaf is not supported.
>
> Okay, thanks. I'd suggest using "true if X is supported" or "true if X
> is supported, false otherwise" phrasing.

Yup, agreed, that reads better, adjusted.

See attached v13 with your feedback addressed in 0001 and 0005,
otherwise the same as before.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-03-24T10:59:19Z

On Tue, Mar 24, 2026 at 2:16 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> Hi John,
>
> On Mon, Mar 23, 2026 at 12:01 AM John Naylor <johncnaylorls@gmail.com> wrote:
> > It's not that bad that the hard-coded indexes are in two places, but
> > it's also not necessary. I think "#define EAX 0 ...etc" is a fine,
> > straightforward increase in readability compared to what we have now,
> > and I don't see any downside. I suppose one argument in favor of the
> > struct is that it avoids declaring variables of type
> > array-of-4-unsigned in multiple places, but I think the array is fine,
> > and adding a new typedef is additional cognitive friction.
>
> Sounds good, lets do it that way - adjusted to use macros instead of a struct.

Looks good. The only thing that I would change is the single-letter
parameter/variable name "r". We could call it "reg" and it'd still be
pretty short. If you agree or have another suggestion, I can change
locally before pushing 0001 -- no need for a new patch set yet.

> > Speaking of signedness, why is the array of ints sometimes signed and
> > sometimes unsigned?
>
> The signedness is a MSVC-ism - I think its reasonable for us to work
> with unsigned integers in our code, and pass them by casting to
> __cpuid/__cpuidex (the MSVC variants).

CI works with this, so fine by me.

--
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-24T14:28:42Z

Hi John,

On Tue, Mar 24, 2026 at 3:59 AM John Naylor <johncnaylorls@gmail.com> wrote:
> Looks good. The only thing that I would change is the single-letter
> parameter/variable name "r". We could call it "reg" and it'd still be
> pretty short. If you agree or have another suggestion, I can change
> locally before pushing 0001 -- no need for a new patch set yet.

Sure, I mainly felt that "exx" would be odd to keep, but "reg" sounds good.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-03-25T05:38:58Z

On Tue, Mar 24, 2026 at 9:29 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> Hi John,
>
> On Tue, Mar 24, 2026 at 3:59 AM John Naylor <johncnaylorls@gmail.com> wrote:
> > Looks good. The only thing that I would change is the single-letter
> > parameter/variable name "r". We could call it "reg" and it'd still be
> > pretty short. If you agree or have another suggestion, I can change
> > locally before pushing 0001 -- no need for a new patch set yet.
>
> Sure, I mainly felt that "exx" would be odd to keep, but "reg" sounds good.

Okay, pushed that way. I also took the liberty of moving a couple
comment changes in 0005 to here. My hope is that will reduce merge
conflicts with the AVX checksums patch, since that one also
invalidates one of those comments and it's not clear which will be
committed first.

-- 
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-03-26T01:17:44Z

Hi John,

On Tue, Mar 24, 2026 at 10:39 PM John Naylor <johncnaylorls@gmail.com> wrote:
> Okay, pushed that way. I also took the liberty of moving a couple
> comment changes in 0005 to here. My hope is that will reduce merge
> conflicts with the AVX checksums patch, since that one also
> invalidates one of those comments and it's not clear which will be
> committed first.

Great, thank you!

Attached rebased v14 - no changes otherwise.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-02T00:54:57Z

Hi,

Lukas, could you add some preliminary Reviewed-by tags to the
not-yet-committed commitse?


On 2026-03-25 18:17:44 -0700, Lukas Fittl wrote:
> From 7cc7e230e05947892b2aa479eba45144beede48e Mon Sep 17 00:00:00 2001
> From: Lukas Fittl <lukas@fittl.com>
> Date: Sat, 31 Jan 2026 08:49:46 -0800
> Subject: [PATCH v14 1/6] Check for HAVE__CPUIDEX and HAVE__GET_CPUID_COUNT
>  separately

John, do you want to take this one, or should I take it?


> Subject: [PATCH v14 2/6] pg_test_timing: Reduce per-loop overhead

Pushed, after simplifying the following

> +	INSTR_TIME_ADD_NANOSEC(end_time, duration > 0 ? duration * NS_PER_S : 0);

by removing that ?:, since duration is unsigned and the caller checks for 0
and negative values. And even if a negative value or 0 were to be passed, it'd
still be harmless.



> From 8fbde9831fd89615a41af56917aed9faf619dbbb Mon Sep 17 00:00:00 2001
> From: Lukas Fittl <lukas@fittl.com>
> Date: Fri, 25 Jul 2025 17:57:20 -0700
> Subject: [PATCH v14 3/6] instrumentation: Streamline ticks to nanosecond
>  conversion across platforms
> 
> The timing infrastructure (INSTR_* macros) measures time elapsed using
> clock_gettime() on POSIX systems, which returns the time as nanoseconds,
> and QueryPerformanceCounter() on Windows, which is a specialized timing
> clock source that returns a tick counter that needs to be converted to
> nanoseconds using the result of QueryPerformanceFrequency().
> 
> This conversion currently happens ad-hoc on Windows, e.g. when calling
> INSTR_TIME_GET_NANOSEC, which calls QueryPerformanceFrequency() on every
> invocation, despite the frequency being stable after program start,
> incurring unnecessary overhead. It also causes a fractured implementation
> where macros are defined differently between platforms.
> 
> To ease code readability, and prepare for a future change that intends
> to use a ticks-to-nanosecond conversion on x86-64 for TSC use, introduce
> a new pg_ticks_to_ns() function that gets called on all platforms.
> 
> This function relies on a separately initialized ticks_per_ns_scaled
> value, that represents the conversion ratio. This value is initialized
> from QueryPerformanceFrequency() on Windows, and set to zero on x86-64
> POSIX systems, which results in the ticks being treated as nanoseconds.
> Other architectures always directly return the original ticks.

> To support this, pg_initialize_timing() is introduced, and is now
> mandatory for both the backend and any frontend programs to call before
> utilizing INSTR_* macros.

Are there assertions detecting missing initializations? Ah, yes, there is.


> +/*
> + * Stores what the number of ticks needs to be multiplied with to end up
> + * with nanoseconds using integer math.
> + *
> + * On certain platforms (currently Windows) the ticks to nanoseconds conversion
> + * requires floating point math because:
> + *
> + * sec = ticks / frequency_hz
> + * ns  = ticks / frequency_hz * 1,000,000,000
> + * ns  = ticks * (1,000,000,000 / frequency_hz)
> + * ns  = ticks * (1,000,000 / frequency_khz) <-- now in kilohertz
> + *
> + * Here, 'ns' is usually a floating number. For example for a 2.5 GHz CPU
> + * the scaling factor becomes 1,000,000 / 2,500,000 = 1.2.
> + *
> + * To be able to use integer math we work around the lack of precision. We
> + * first scale the integer up (left shift by TICKS_TO_NS_SHIFT) and after the
> + * multiplication by the number of ticks in pg_ticks_to_ns() we shift right by
> + * the same amount. We utilize unsigned integers even though ticks are stored
> + * as a signed value to encourage compilers to generate better assembly.
> + *
> + * We remember the maximum number of ticks that can be multiplied by the scale
> + * factor without overflowing so we can check via a * b > max <=> a > max / b.
> + *
> + * On all other platforms we are using clock_gettime(), which uses nanoseconds
> + * as ticks. Hence, we set the multiplier to zero, which causes pg_ticks_to_ns
> + * to return the original value.
> + */
> +uint64		ticks_per_ns_scaled = 0;
> +uint64		max_ticks_no_overflow = 0;
> +bool		timing_initialized = false;

Think it may be worth adding an example for a realistic max_ticks_no_overflow
to show that we almost always are going to be able to avoid the overflow case.


> +static void set_ticks_per_ns(void);
> +
> +void
> +pg_initialize_timing(void)
> +{
> +	if (timing_initialized)
> +		return;
> +
> +	set_ticks_per_ns();
> +	timing_initialized = true;
> +}
> +
> +#ifndef WIN32
> +
> +static void
> +set_ticks_per_ns(void)
> +{
> +	ticks_per_ns_scaled = 0;
> +	max_ticks_no_overflow = 0;
> +}
> +
> +#else							/* WIN32 */
> +
> +/* GetTimerFrequency returns counts per second */
> +static inline double
> +GetTimerFrequency(void)
> +{
> +	LARGE_INTEGER f;
> +
> +	QueryPerformanceFrequency(&f);
> +	return (double) f.QuadPart;
> +}
> +
> +static void
> +set_ticks_per_ns(void)
> +{
> +	ticks_per_ns_scaled = (NS_PER_S << TICKS_TO_NS_SHIFT) / GetTimerFrequency();
> +	max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
> +}
> +
> +#endif							/* WIN32 */

Seems a tiny bit odd to introduce them here with a name to then evolve the
naming pattern in the subsequent patches.

> +static inline int64
> +pg_ticks_to_ns(int64 ticks)
>  {
> -	LARGE_INTEGER f;
> +#if PG_INSTR_TICKS_TO_NS
> +	int64		ns = 0;
> +
> +	Assert(timing_initialized);
> +
> +	/*
> +	 * Avoid doing work if we don't use scaled ticks, e.g. system clock on
> +	 * Unix
> +	 */
> +	if (ticks_per_ns_scaled == 0)
> +		return ticks;
> +
> +	/*
> +	 * Would multiplication overflow? If so perform computation in two parts.
> +	 */
> +	if (unlikely(ticks > (int64) max_ticks_no_overflow))
> +	{
> +		/*
> +		 * To avoid overflow, first scale total ticks down by the fixed
> +		 * factor, and *afterwards* multiply them by the frequency-based scale
> +		 * factor.
> +		 *
> +		 * The remaining ticks can follow the regular formula, since they
> +		 * won't overflow.
> +		 */
> +		int64		count = ticks >> TICKS_TO_NS_SHIFT;
> +
> +		ns = count * ticks_per_ns_scaled;
> +		ticks -= (count << TICKS_TO_NS_SHIFT);
> +	}
> +
> +	ns += (ticks * ticks_per_ns_scaled) >> TICKS_TO_NS_SHIFT;
> +
> +	return ns;
> +#else
> +	Assert(timing_initialized);
>  
> -	QueryPerformanceFrequency(&f);
> -	return (double) f.QuadPart;
> +	return ticks;
> +#endif							/* PG_INSTR_TICKS_TO_NS */

Kinda wondering what the best way would be to verify that the overflow case
actually works.  What about adding a regress.c test that checks that we get
the right results for something like

   INSTR_TIME_SET_ZERO(t);
   INSTR_TIME_ADD_NANOSEC(t, some_number);
   EXPECT_EQ_U64(INSTR_TIME_GET_NANOSEC(t), some_number);

for a few different some_numbers?



> Subject: [PATCH v14 4/6] instrumentation: Use Time-Stamp Counter (TSC) on
>  x86-64 for faster measurements

> +bool
> +check_timing_clock_source(int *newval, void **extra, GucSource source)
> +{
> +	/*
> +	 * Ensure timing is initialized. On Windows (EXEC_BACKEND), GUC hooks can
> +	 * be called during InitializeGUCOptions() before InitProcessGlobals() has
> +	 * had a chance to run pg_initialize_timing().
> +	 */
> +	pg_initialize_timing();

So doesn't that mean the effort to synchronize the tsc freq on EXEC_BACKEND is futile?


> diff --git a/src/bin/pg_test_timing/pg_test_timing.c b/src/bin/pg_test_timing/pg_test_timing.c
> index 1d9ee4fb588..ee0e3a3b0ab 100644
> --- a/src/bin/pg_test_timing/pg_test_timing.c
> +++ b/src/bin/pg_test_timing/pg_test_timing.c
> @@ -43,7 +43,9 @@ main(int argc, char *argv[])
>  
>  	handle_args(argc, argv);
>  
> -	/* initialize timing infrastructure (required for INSTR_* calls) */
> +	/*
> +	 * Initialize timing infrastructure (required for INSTR_* calls)
> +	 */
>  	pg_initialize_timing();
>  
>  	loop_count = test_timing(test_duration);

This seems like a superfluous change?


> +bool
> +pg_set_timing_clock_source(TimingClockSourceType source)
> +{
> +	Assert(timing_initialized);
> +
> +#if PG_INSTR_TSC_CLOCK
> +	pg_initialize_timing_tsc();
> +#endif
> +
> +#if PG_INSTR_TSC_CLOCK
> +	switch (source)
> +	{
> +		case TIMING_CLOCK_SOURCE_AUTO:
> +			use_tsc = (tsc_frequency_khz > 0) && tsc_use_by_default();

A global named use_tsc seems a tad too short a name for my personal taste. Too
likely to also be used by something else.


> +int32		tsc_frequency_khz = -1;
> +
> +static uint32 tsc_calibrate(void);
> +
> +/*
> + * Detect the TSC frequency and whether RDTSCP is available on x86-64.
> + *
> + * This can't be reliably determined at compile time, since the
> + * availability of an "invariant" TSC (that is not affected by CPU
> + * frequency changes) is dependent on the CPU architecture. Additionally,
> + * there are cases where TSC availability is impacted by virtualization,
> + * where a simple cpuid feature check would not be enough.
> + */
> +static void
> +tsc_detect_frequency(void)
> +{
> +	tsc_frequency_khz = 0;

So we're using -1 for uninitialized and 0 for unknown? If so we should
document that.


> +	/* We require RDTSCP support, bail if not available */
> +	if (!x86_feature_available(PG_RDTSCP))
> +		return;
> +
> +	/* Determine speed at which the TSC advances */
> +	tsc_frequency_khz = x86_tsc_frequency_khz();
> +	if (tsc_frequency_khz > 0)
> +		return;
> +
> +	/*
> +	 * CPUID did not give us the TSC frequency. If TSC is invariant and RDTSCP
> +	 * is available, we can measure the frequency by comparing TSC ticks
> +	 * against walltime using a short calibration loop.
> +	 */
> +	if (x86_feature_available(PG_TSC_INVARIANT))
> +		tsc_frequency_khz = tsc_calibrate();
> +}

Why do we not need to always check if the TSC is invariant? Seems like a hard
requirement to me?



> +/*
> + * Calibrate the TSC frequency by comparing TSC ticks against walltime.
> + *
> + * Takes initial TSC and system clock snapshots, then loops, recomputing the
> + * frequency each TSC_CALIBRATION_SKIPS iterations from cumulative TSC
> + * ticks divided by elapsed time.
> + *
> + * Once the frequency estimate stabilizes (consecutive iterations agree), we
> + * consider it converged and the frequency in KHz is returned. If either too
> + * many iterations or a time limit passes without convergence, 0 is returned.
> + */
> +#define TSC_CALIBRATION_MAX_NS		(50 * NS_PER_MS)
> +#define TSC_CALIBRATION_ITERATIONS	1000000
> +#define TSC_CALIBRATION_SKIPS		100
> +#define TSC_CALIBRATION_STABLE_CYCLES	10
> +

> +static uint32
> +tsc_calibrate(void)
> +{
> +	instr_time	initial_wall;
> +	int64		initial_tsc;
> +	double		freq_khz = 0;
> +	double		prev_freq_khz = 0;
> +	int			stable_count = 0;
> +	int64		prev_tsc;
> +	uint32		unused;
> +
> +	/* Ensure INSTR_* time below work on system time */
> +	set_ticks_per_ns_system();
> +
> +	INSTR_TIME_SET_CURRENT(initial_wall);
> +
> +#ifdef _MSC_VER
> +	initial_tsc = __rdtscp(&unused);
> +#else
> +	initial_tsc = __builtin_ia32_rdtscp(&unused);
> +#endif
> +	prev_tsc = initial_tsc;
> +
> +	for (int i = 0; i < TSC_CALIBRATION_ITERATIONS; i++)
> +	{
> +		instr_time	now_wall;
> +		int64		now_tsc;
> +		int64		elapsed_ns;
> +		int64		elapsed_ticks;
> +
> +		INSTR_TIME_SET_CURRENT(now_wall);
> +
> +#ifdef _MSC_VER
> +		now_tsc = __rdtscp(&unused);
> +#else
> +		now_tsc = __builtin_ia32_rdtscp(&unused);
> +#endif

I'd put this in a helper, given that we already need it twice, and we might
later need to extend it for other architectures.


> +		/*
> +		 * Skip if this is not the Nth cycle where we measure, if TSC hasn't
> +		 * advanced, or we walked backwards for some reason.
> +		 */
> +		if (i % TSC_CALIBRATION_SKIPS != 0 || now_tsc == prev_tsc || elapsed_ns <= 0 || elapsed_ticks <= 0)
> +			continue;

What's the goal of TSC_CALIBRATION_SKIPS?



> +		freq_khz = ((double) elapsed_ticks / elapsed_ns) * 1000 * 1000;
> +
> +		/*
> +		 * Once freq_khz / prev_freq_khz is small, check if it stays that way.
> +		 * If it does for long enough, we've got a winner frequency.
> +		 */
> +		if (prev_freq_khz != 0 && fabs(1 - freq_khz / prev_freq_khz) < 0.0001)
> +		{
> +			stable_count++;
> +			if (stable_count >= TSC_CALIBRATION_STABLE_CYCLES)
> +				return (uint32) freq_khz;
> +		}
> +		else
> +			stable_count = 0;
> +
> +		prev_tsc = now_tsc;
> +		prev_freq_khz = freq_khz;
> +	}
> +
> +	/* did not converge */
> +	return 0;
> +}

I think it may be worth teaching pg_test_timing to display the tsc frequency
and show the frequency determined by calibration even if we got it via
cpuid. Otherwise it'll be hard to debug this.


> +/*
> + * Initialize the TSC clock source by determining its usability and frequency.
> + *
> + * This can be called multiple times, as tsc_frequency_khz will be set to 0
> + * if a prior call determined the TSC is not usable. On EXEC_BACKEND (Windows),
> + * the TSC frequency may also be set by restore_backend_variables.
> + */
> +void
> +pg_initialize_timing_tsc(void)
> +{
> +	if (tsc_frequency_khz < 0)
> +		tsc_detect_frequency();
> +}

Maybe we should add a debug log with the frequency in the < 0 case?

> +uint32
> +x86_tsc_frequency_khz(void)
> +{
> +	unsigned int r[4] = {0};
> +
> +	if (!x86_feature_available(PG_TSC_INVARIANT))
> +		return 0;
> +
> +	if (x86_feature_available(PG_HYPERVISOR))
> +		return x86_hypervisor_tsc_frequency_khz();
> +
> +	/*
> +	 * On modern Intel CPUs, the TSC is implemented by invariant timekeeping
> +	 * hardware, also called "Always Running Timer", or ART. The ART stays
> +	 * consistent even if the CPU changes frequency due to changing power
> +	 * levels.
> +	 *
> +	 * As documented in "Determining the Processor Base Frequency" in the
> +	 * "Intel® 64 and IA-32 Architectures Software Developer's Manual",
> +	 * February 2026 Edition, we can get the TSC frequency as follows:
> +	 *
> +	 * Nominal TSC frequency = ( CPUID.15H:ECX[31:0] * CPUID.15H:EBX[31:0] ) /
> +	 * CPUID.15H:EAX[31:0]
> +	 *
> +	 * With CPUID.15H:ECX representing the nominal core crystal clock
> +	 * frequency, and EAX/EBX representing values used to translate the TSC
> +	 * value to that frequency, see "Chapter 20.17 "Time-Stamp Counter" of
> +	 * that manual.
> +	 *
> +	 * Older Intel CPUs, and other vendors do not set CPUID.15H:ECX, and as
> +	 * such we fall back to alternate approaches.
> +	 */
> +	pg_cpuid(0x15, r);
> +	if (r[ECX] > 0)
> +	{
> +		/*
> +		 * EBX not being set indicates invariant TSC is not available. Require
> +		 * EAX being non-zero too, to avoid a theoretical divide by zero.
> +		 */
> +		if (r[EAX] == 0 || r[EBX] == 0)
> +			return 0;
> +
> +		return r[ECX] / 1000 * r[EBX] / r[EAX];
> +	}
> +
> +	/*
> +	 * When CPUID.15H is not available/incomplete, but we have verified an
> +	 * invariant TSC is used, we can instead get the processor base frequency
> +	 * in MHz from CPUID.16H:EAX, the "Processor Frequency Information Leaf".
> +	 */

s/can instead/can possibly instead/

As it's also optional...



> From 44292c17075264c659e03d784a6c7619e4c5bd3e Mon Sep 17 00:00:00 2001
> From: Lukas Fittl <lukas@fittl.com>
> Date: Tue, 10 Mar 2026 01:38:14 -0700
> Subject: [PATCH v14 6/6] instrumentation: ARM support for fast time
>  measurements
> 
> Similar to the RDTSC/RDTSCP instructions on x68-64, this introduces
> use of the cntvct_el0 instruction on ARM systems to access the generic
> timer that provides a synchronized ticks value across CPUs.
> 
> Note this adds an exception for Apple Silicon CPUs, due to the observed
> fact that M3 and newer has different timer frequencies for the Efficiency
> and the Performance cores, and we can't be sure where we get scheduled.

There are other such SOCs with different clock speeds for
performance/efficiency cores, I'm a bit worried that just denylisting Apple
Silicon will end up allowing other such systems.


Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-04-02T02:17:49Z

On Thu, Apr 2, 2026 at 7:54 AM Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-03-25 18:17:44 -0700, Lukas Fittl wrote:
> > From 7cc7e230e05947892b2aa479eba45144beede48e Mon Sep 17 00:00:00 2001
> > From: Lukas Fittl <lukas@fittl.com>
> > Date: Sat, 31 Jan 2026 08:49:46 -0800
> > Subject: [PATCH v14 1/6] Check for HAVE__CPUIDEX and HAVE__GET_CPUID_COUNT
> >  separately
>
> John, do you want to take this one, or should I take it?

I can. I took another look just now:

-# Check for __get_cpuid_count() and __cpuidex() in a similar fashion.
+# Check for __get_cpuid_count() and __cpuidex() separately, since we sometimes
+# need __cpuidex() even if __get_cpuid_count() is available.

I'm not sure we need to document here that we need them separately. It
seems more important to do that for the __cpuidex helper that's in a
later patch. I don't feel strongly, though.

I'd probably just do

# Check for __get_cpuid_count()
...

# Check for __cpuidex()
...

Either way, we do need a space between separate tests (ditto Meson).

+if cc.links('''
+    #ifdef _MSC_VER
     #include <intrin.h>
+    #else
+    #include <cpuid.h>
+    #endif
     int main(int arg, char **argv)
     {
-        unsigned int exx[4] = {0, 0, 0, 0};
+        int exx[4] = {0, 0, 0, 0};
         __cpuidex(exx, 7, 0);
     }

Hmm, maybe we can declare unsigned and cast to signed for MSVC as we
do where the code is used.

+# __cpuidex()
+AC_CACHE_CHECK([for __cpuidex], [pgac_cv__cpuidex],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#ifdef _MSC_VER
+    #include <intrin.h>
+    #else
+    #include <cpuid.h>
+    #endif],
+  [[int exx[4] = {0, 0, 0, 0};
+  __cpuidex(exx, 7, 0);
+  ]])],
+  [pgac_cv__cpuidex="yes"],
+  [pgac_cv__cpuidex="no"])])
+if test x"$pgac_cv__cpuidex" = x"yes"; then
+  AC_DEFINE(HAVE__CPUIDEX, 1, [Define to 1 if you have __cpuidex.])
 fi

MSVC doesn't use autoconf, so we can leave out the intrin.h and keep
using unsigned int.

-#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT)
-#include <cpuid.h>
-#endif
-
-#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX)
+#if defined(HAVE__CPUID) || defined(HAVE__GET_CPUID) ||
defined(HAVE__GET_CPUID_COUNT) || defined(HAVE__CPUIDEX)
+#if defined(_MSC_VER)
 #include <intrin.h>
+#else
+#include <cpuid.h>
+#endif /* defined(_MSC_VER) */
 #endif

Now I'm thinking at least one of these will be defined, otherwise we'd
have an #error, so we can just gate solely on _MSC_VER.

-- 
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-02T08:50:08Z

On Wed, Apr 1, 2026 at 7:18 PM John Naylor <johncnaylorls@gmail.com> wrote:
> -# Check for __get_cpuid_count() and __cpuidex() in a similar fashion.
> +# Check for __get_cpuid_count() and __cpuidex() separately, since we sometimes
> +# need __cpuidex() even if __get_cpuid_count() is available.
>
> I'm not sure we need to document here that we need them separately. It
> seems more important to do that for the __cpuidex helper that's in a
> later patch. I don't feel strongly, though.
>
> I'd probably just do
>
> # Check for __get_cpuid_count()
> ...
>
> # Check for __cpuidex()
> ...
>
> Either way, we do need a space between separate tests (ditto Meson).

Makes sense, updated.

>
> +if cc.links('''
> +    #ifdef _MSC_VER
>      #include <intrin.h>
> +    #else
> +    #include <cpuid.h>
> +    #endif
>      int main(int arg, char **argv)
>      {
> -        unsigned int exx[4] = {0, 0, 0, 0};
> +        int exx[4] = {0, 0, 0, 0};
>          __cpuidex(exx, 7, 0);
>      }
>
> Hmm, maybe we can declare unsigned and cast to signed for MSVC as we
> do where the code is used.

Sure, I think it works either way - updated.

>
> +# __cpuidex()
> +AC_CACHE_CHECK([for __cpuidex], [pgac_cv__cpuidex],
> +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#ifdef _MSC_VER
> +    #include <intrin.h>
> +    #else
> +    #include <cpuid.h>
> +    #endif],
> +  [[int exx[4] = {0, 0, 0, 0};
> +  __cpuidex(exx, 7, 0);
> +  ]])],
> +  [pgac_cv__cpuidex="yes"],
> +  [pgac_cv__cpuidex="no"])])
> +if test x"$pgac_cv__cpuidex" = x"yes"; then
> +  AC_DEFINE(HAVE__CPUIDEX, 1, [Define to 1 if you have __cpuidex.])
>  fi
>
> MSVC doesn't use autoconf, so we can leave out the intrin.h and keep
> using unsigned int.

I think that might be a problem for Clang on Windows, which has MSVC
compatibility headers, and will set _MSC_VER, see [0] and [1].

It probably works in practice, but it seems better to match the
autoconf checks to the actual code?

>
> -#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT)
> -#include <cpuid.h>
> -#endif
> -
> -#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX)
> +#if defined(HAVE__CPUID) || defined(HAVE__GET_CPUID) ||
> defined(HAVE__GET_CPUID_COUNT) || defined(HAVE__CPUIDEX)
> +#if defined(_MSC_VER)
>  #include <intrin.h>
> +#else
> +#include <cpuid.h>
> +#endif /* defined(_MSC_VER) */
>  #endif
>
> Now I'm thinking at least one of these will be defined, otherwise we'd
> have an #error, so we can just gate solely on _MSC_VER.

Yeah, makes sense, we would fail to compile in that case anyway.

I'm attaching v15 rebased, with the above changes made to 0001 to
unblock things, but feedback by Andres not addressed yet (will work on
that tomorrow).

John, feel free to adjust as you see fit in case there is any other
minor aspects to fix.

Thanks,
Lukas

[0]: https://clang.llvm.org/docs/UsersManual.html#microsoft-extensions
[1]: https://clang.llvm.org/doxygen/intrin_8h_source.html

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-04-02T12:55:31Z

On Thu, Apr 2, 2026 at 3:50 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Wed, Apr 1, 2026 at 7:18 PM John Naylor <johncnaylorls@gmail.com> wrote:

> > +AC_CACHE_CHECK([for __cpuidex], [pgac_cv__cpuidex],
> > +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#ifdef _MSC_VER
> > +    #include <intrin.h>
> > +    #else
> > +    #include <cpuid.h>
> > +    #endif],
> > +  [[int exx[4] = {0, 0, 0, 0};
> > +  __cpuidex(exx, 7, 0);
> > +  ]])],
> > +  [pgac_cv__cpuidex="yes"],
> > +  [pgac_cv__cpuidex="no"])])
> > +if test x"$pgac_cv__cpuidex" = x"yes"; then
> > +  AC_DEFINE(HAVE__CPUIDEX, 1, [Define to 1 if you have __cpuidex.])
> >  fi
> >
> > MSVC doesn't use autoconf, so we can leave out the intrin.h and keep
> > using unsigned int.
>
> I think that might be a problem for Clang on Windows, which has MSVC
> compatibility headers, and will set _MSC_VER, see [0] and [1].
>
> It probably works in practice, but it seems better to match the
> autoconf checks to the actual code?

Hmm, the previous coding had <intrin.h>, and this is just adding
<cpuid.h> as well, so I left it alone.

The only change I made was to make the #include stanza in pg_cpu_x86.c
match its neighbors.

I thought to add a note to the commit message about the casts to
signed int, but only after pushing did I notice that the explanation
looks out of place since no new code uses __cpuidex yet, and warnings
don't matter for configure checks. Hope it's not too confusing.

--
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-02T14:15:55Z

Hi,

On 2026-04-02 19:55:31 +0700, John Naylor wrote:
> The only change I made was to make the #include stanza in pg_cpu_x86.c
> match its neighbors.

Thanks for taking care of this one!

I'm wondering about splitting out some of the CPU detection stuff that's in
  "v15 3/5 instrumentation: Use Time-Stamp Counter (TSC) on x86-64 for faster measurements"
into its own commit.  As-is it's a pretty large commit doing lots of different things.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-02T23:16:13Z

Hi Andres,

On Wed, Apr 1, 2026 at 5:54 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> Lukas, could you add some preliminary Reviewed-by tags to the
> not-yet-committed commitse?
>

Sure - I've gone back through the thread and added preliminary review
tags for anyone who tested the patch/provided code feedback - if
anyone was left out incorrectly, my advance apologies. I've been
liberal in adding "(in an earlier version)" since I don't want to
claim that someone reviewed the latest variant incorrectly.

Also I wasn't sure how you wanted to deal with the mix of authorship
and reviews the three of us (Andres, David, Lukas) have done over the
lifetime of this thread, so I tried to put what feels right in terms
of which author to list first, and also listed ourselves as reviewers.
e.g. David and you were the ones who originally proposed the ticks
scaling logic, so I now put you first as authors on the commit that
introduces pg_ticks_to_ns (0001). For the TSC patch I've driven most
of the current lines of code including the GUC handling, so I put
myself first. FWIW, I'm certainly fine with putting my name in a
different order on one of the commits if that doesn't seem accurate,
since I think we all just want to see this in Postgres one way or the
other.

> > +/*
> > + * Stores what the number of ticks needs to be multiplied with to end up
> > + * with nanoseconds using integer math.
> > + *
> > + * On certain platforms (currently Windows) the ticks to nanoseconds conversion
> > + * requires floating point math because:
> > + *
> > + * sec = ticks / frequency_hz
> > + * ns  = ticks / frequency_hz * 1,000,000,000
> > + * ns  = ticks * (1,000,000,000 / frequency_hz)
> > + * ns  = ticks * (1,000,000 / frequency_khz) <-- now in kilohertz
> > + *
> > + * Here, 'ns' is usually a floating number. For example for a 2.5 GHz CPU
> > + * the scaling factor becomes 1,000,000 / 2,500,000 = 1.2.
> > + *
> > + * To be able to use integer math we work around the lack of precision. We
> > + * first scale the integer up (left shift by TICKS_TO_NS_SHIFT) and after the
> > + * multiplication by the number of ticks in pg_ticks_to_ns() we shift right by
> > + * the same amount. We utilize unsigned integers even though ticks are stored
> > + * as a signed value to encourage compilers to generate better assembly.
> > + *
> > + * We remember the maximum number of ticks that can be multiplied by the scale
> > + * factor without overflowing so we can check via a * b > max <=> a > max / b.
> > + *
> > + * On all other platforms we are using clock_gettime(), which uses nanoseconds
> > + * as ticks. Hence, we set the multiplier to zero, which causes pg_ticks_to_ns
> > + * to return the original value.
> > + */
> > +uint64               ticks_per_ns_scaled = 0;
> > +uint64               max_ticks_no_overflow = 0;
> > +bool         timing_initialized = false;
>
> Think it may be worth adding an example for a realistic max_ticks_no_overflow
> to show that we almost always are going to be able to avoid the overflow case.

I added a clarifying comment:

 * ...
 * We remember the maximum number of ticks that can be multiplied by the scale
 * factor without overflowing so we can check via a * b > max <=> a > max / b.
 *
 * However, as this is meant for interval measurements, it is unlikely that the
 * overflow path is actually taken in typical scenarios, since overflows would
 * only occur for intervals longer than 6.5 days.
 *...

Also, FWIW, earlier in that same comment we had bad math (I think), so
I corrected it as 1,000,000 / 2,500,000 = 0.4 (was 1.2).

> > +static inline int64
> > +pg_ticks_to_ns(int64 ticks)
> >  {
> > -     LARGE_INTEGER f;
> > +#if PG_INSTR_TICKS_TO_NS
> > +     int64           ns = 0;
> > +
> > +     Assert(timing_initialized);
> > +
> > +     /*
> > +      * Avoid doing work if we don't use scaled ticks, e.g. system clock on
> > +      * Unix
> > +      */
> > +     if (ticks_per_ns_scaled == 0)
> > +             return ticks;
> > +
> > +     /*
> > +      * Would multiplication overflow? If so perform computation in two parts.
> > +      */
> > +     if (unlikely(ticks > (int64) max_ticks_no_overflow))
> > +     {
> > +             /*
> > +              * To avoid overflow, first scale total ticks down by the fixed
> > +              * factor, and *afterwards* multiply them by the frequency-based scale
> > +              * factor.
> > +              *
> > +              * The remaining ticks can follow the regular formula, since they
> > +              * won't overflow.
> > +              */
> > +             int64           count = ticks >> TICKS_TO_NS_SHIFT;
> > +
> > +             ns = count * ticks_per_ns_scaled;
> > +             ticks -= (count << TICKS_TO_NS_SHIFT);
> > +     }
> > +
> > +     ns += (ticks * ticks_per_ns_scaled) >> TICKS_TO_NS_SHIFT;
> > +
> > +     return ns;
> > +#else
> > +     Assert(timing_initialized);
> >
> > -     QueryPerformanceFrequency(&f);
> > -     return (double) f.QuadPart;
> > +     return ticks;
> > +#endif                                                       /* PG_INSTR_TICKS_TO_NS */
>
> Kinda wondering what the best way would be to verify that the overflow case
> actually works.  What about adding a regress.c test that checks that we get
> the right results for something like
>
>    INSTR_TIME_SET_ZERO(t);
>    INSTR_TIME_ADD_NANOSEC(t, some_number);
>    EXPECT_EQ_U64(INSTR_TIME_GET_NANOSEC(t), some_number);
>
> for a few different some_numbers?
>

Yeah, that's a good idea. I added a test similar to what you proposed,
with some extra logic to handle the imprecision introduced by the
shifting. I also confirmed that the test correctly fails when removing
a part of the overflow logic.

>
> > Subject: [PATCH v14 4/6] instrumentation: Use Time-Stamp Counter (TSC) on
> >  x86-64 for faster measurements
>
> > +bool
> > +check_timing_clock_source(int *newval, void **extra, GucSource source)
> > +{
> > +     /*
> > +      * Ensure timing is initialized. On Windows (EXEC_BACKEND), GUC hooks can
> > +      * be called during InitializeGUCOptions() before InitProcessGlobals() has
> > +      * had a chance to run pg_initialize_timing().
> > +      */
> > +     pg_initialize_timing();
>
> So doesn't that mean the effort to synchronize the tsc freq on EXEC_BACKEND is futile?

Good catch. I think the best way out of that is for us to not
initialize the TSC in the GUC hooks for EXEC_BACKEND, and instead make
a call to pg_set_timing_clock_source in restore_backend_variables,
which will initialize the TSC if not already initialized.

Done that way for now, with an early return in both assign and check
hooks if EXEC_BACKEND and timing not initialized.

>
> > +bool
> > +pg_set_timing_clock_source(TimingClockSourceType source)
> > +{
> > +     Assert(timing_initialized);
> > +
> > +#if PG_INSTR_TSC_CLOCK
> > +     pg_initialize_timing_tsc();
> > +#endif
> > +
> > +#if PG_INSTR_TSC_CLOCK
> > +     switch (source)
> > +     {
> > +             case TIMING_CLOCK_SOURCE_AUTO:
> > +                     use_tsc = (tsc_frequency_khz > 0) && tsc_use_by_default();
>
> A global named use_tsc seems a tad too short a name for my personal taste. Too
> likely to also be used by something else.

Renamed this to "timing_tsc_enabled", and also renamed the frequency
variable to timing_tsc_frequency_khz for consistency.

>
> > +int32                tsc_frequency_khz = -1;
> > +
> > +static uint32 tsc_calibrate(void);
> > +
> > +/*
> > + * Detect the TSC frequency and whether RDTSCP is available on x86-64.
> > + *
> > + * This can't be reliably determined at compile time, since the
> > + * availability of an "invariant" TSC (that is not affected by CPU
> > + * frequency changes) is dependent on the CPU architecture. Additionally,
> > + * there are cases where TSC availability is impacted by virtualization,
> > + * where a simple cpuid feature check would not be enough.
> > + */
> > +static void
> > +tsc_detect_frequency(void)
> > +{
> > +     tsc_frequency_khz = 0;
>
> So we're using -1 for uninitialized and 0 for unknown? If so we should
> document that.

That's documented in instr_time.h:

/*
 * TSC frequency in kHz, set during initialization.
 *
 * -1 = not yet initialized, 0 = TSC not usable, >0 = frequency in kHz.
 */
extern PGDLLIMPORT int32 timing_tsc_frequency_khz;

Let me know if you think there is a better location.

> > +     /* We require RDTSCP support, bail if not available */
> > +     if (!x86_feature_available(PG_RDTSCP))
> > +             return;
> > +
> > +     /* Determine speed at which the TSC advances */
> > +     tsc_frequency_khz = x86_tsc_frequency_khz();
> > +     if (tsc_frequency_khz > 0)
> > +             return;
> > +
> > +     /*
> > +      * CPUID did not give us the TSC frequency. If TSC is invariant and RDTSCP
> > +      * is available, we can measure the frequency by comparing TSC ticks
> > +      * against walltime using a short calibration loop.
> > +      */
> > +     if (x86_feature_available(PG_TSC_INVARIANT))
> > +             tsc_frequency_khz = tsc_calibrate();
> > +}
>
> Why do we not need to always check if the TSC is invariant? Seems like a hard
> requirement to me?

Yeah, that's a good point - moved this up to check it together with
RDTSCP support.

> > +static uint32
> > +tsc_calibrate(void)
> > +{
> > +     instr_time      initial_wall;
> > +     int64           initial_tsc;
> > +     double          freq_khz = 0;
> > +     double          prev_freq_khz = 0;
> > +     int                     stable_count = 0;
> > +     int64           prev_tsc;
> > +     uint32          unused;
> > +
> > +     /* Ensure INSTR_* time below work on system time */
> > +     set_ticks_per_ns_system();
> > +
> > +     INSTR_TIME_SET_CURRENT(initial_wall);
> > +
> > +#ifdef _MSC_VER
> > +     initial_tsc = __rdtscp(&unused);
> > +#else
> > +     initial_tsc = __builtin_ia32_rdtscp(&unused);
> > +#endif
> > +     prev_tsc = initial_tsc;
> > +
> > +     for (int i = 0; i < TSC_CALIBRATION_ITERATIONS; i++)
> > +     {
> > +             instr_time      now_wall;
> > +             int64           now_tsc;
> > +             int64           elapsed_ns;
> > +             int64           elapsed_ticks;
> > +
> > +             INSTR_TIME_SET_CURRENT(now_wall);
> > +
> > +#ifdef _MSC_VER
> > +             now_tsc = __rdtscp(&unused);
> > +#else
> > +             now_tsc = __builtin_ia32_rdtscp(&unused);
> > +#endif
>
> I'd put this in a helper, given that we already need it twice, and we might
> later need to extend it for other architectures.

Yeah, that's fair. Since we have the same pattern in instr_time.h I
added both "pg_rdtsc" and "pg_rdtscp" as inline helpers.

I assume we want to avoid the extra include, otherwise we could also
put these in pg_cpu.h.

>
> > +             /*
> > +              * Skip if this is not the Nth cycle where we measure, if TSC hasn't
> > +              * advanced, or we walked backwards for some reason.
> > +              */
> > +             if (i % TSC_CALIBRATION_SKIPS != 0 || now_tsc == prev_tsc || elapsed_ns <= 0 || elapsed_ticks <= 0)
> > +                     continue;
>
> What's the goal of TSC_CALIBRATION_SKIPS?

This improved the measurement quality in my testing, by measuring the
time elapsed across 100 RDTSCP instructions, instead of a single
instruction. Without this we are more biased towards a later
measurement stabilizing based on just a handful of RDTSCP
instructions, since we're doing

freq_khz = ((double) elapsed_ticks / elapsed_ns) * 1000 * 1000;

i.e. the 1000th measurement and the 1001th measurement have a very
small difference in terms of ticks and ns, causing us to quickly
confirm subsequent measurements, causing an inaccurate TSC frequency.

> > +             freq_khz = ((double) elapsed_ticks / elapsed_ns) * 1000 * 1000;
> > +
> > +             /*
> > +              * Once freq_khz / prev_freq_khz is small, check if it stays that way.
> > +              * If it does for long enough, we've got a winner frequency.
> > +              */
> > +             if (prev_freq_khz != 0 && fabs(1 - freq_khz / prev_freq_khz) < 0.0001)
> > +             {
> > +                     stable_count++;
> > +                     if (stable_count >= TSC_CALIBRATION_STABLE_CYCLES)
> > +                             return (uint32) freq_khz;
> > +             }
> > +             else
> > +                     stable_count = 0;
> > +
> > +             prev_tsc = now_tsc;
> > +             prev_freq_khz = freq_khz;
> > +     }
> > +
> > +     /* did not converge */
> > +     return 0;
> > +}
>
> I think it may be worth teaching pg_test_timing to display the tsc frequency
> and show the frequency determined by calibration even if we got it via
> cpuid. Otherwise it'll be hard to debug this.

Displaying the TSC frequency is already done in the later patch that
adds extra RDTSC(P) reporting in pg_test_timing.

However, that later change only reports the frequency that is set in
the global, and doesn't call into the TSC calibration logic if we got
it via CPUID. We could add that if you think its important, but it'd
require exporting tsc_calibrate in instr_time.h.

> > +/*
> > + * Initialize the TSC clock source by determining its usability and frequency.
> > + *
> > + * This can be called multiple times, as tsc_frequency_khz will be set to 0
> > + * if a prior call determined the TSC is not usable. On EXEC_BACKEND (Windows),
> > + * the TSC frequency may also be set by restore_backend_variables.
> > + */
> > +void
> > +pg_initialize_timing_tsc(void)
> > +{
> > +     if (tsc_frequency_khz < 0)
> > +             tsc_detect_frequency();
> > +}
>
> Maybe we should add a debug log with the frequency in the < 0 case?
>

Makes sense, added, with a !FRONTEND check.

> > From 44292c17075264c659e03d784a6c7619e4c5bd3e Mon Sep 17 00:00:00 2001
> > From: Lukas Fittl <lukas@fittl.com>
> > Date: Tue, 10 Mar 2026 01:38:14 -0700
> > Subject: [PATCH v14 6/6] instrumentation: ARM support for fast time
> >  measurements
> >
> > Similar to the RDTSC/RDTSCP instructions on x68-64, this introduces
> > use of the cntvct_el0 instruction on ARM systems to access the generic
> > timer that provides a synchronized ticks value across CPUs.
> >
> > Note this adds an exception for Apple Silicon CPUs, due to the observed
> > fact that M3 and newer has different timer frequencies for the Efficiency
> > and the Performance cores, and we can't be sure where we get scheduled.
>
> There are other such SOCs with different clock speeds for
> performance/efficiency cores, I'm a bit worried that just denylisting Apple
> Silicon will end up allowing other such systems.

That makes sense. As described earlier, I'm mainly including that
patch to show that we can expand this to ARM (i.e. our general design
allows for that), but I'd say its probably best considered separately
in a new thread for commit in a future release.

Attached v16 with your requested changes applied, as well as:

- Split out the CPU detection logic into a separate commit (as
suggested in a separate email by Andres)
- Moved a documentation snippet referencing pg_test_timing to the
later commit that shows TSC details in pg_test_timing (if for some
reason we didn't commit that pg_test_timing change, that docs snippet
doesn't make sense)
- Removed the socket counting logic again to determine whether to use
TSC by default, its now only gated on having a frequency, and
PG_RDTSCP + PG_TSC_INVARIANT + PG_TSC_ADJUST. After some pondering and
testing on the one 8 socket machine I could get my hands on
("u-6tb1.56xlarge" instance in AWS), which had a consistent TSC
frequency across its 224 cores, I suspect this may have been more of
an edge case situation the Linux kernel folks were concerned about.
Since we have the GUC to turn off use of the TSC, it seems better to
reduce code complexity here for now.


Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-04-03T09:06:14Z

On Fri, Apr 3, 2026 at 6:16 AM Lukas Fittl <lukas@fittl.com> wrote:
> v16

Just some minor quibbles on 0002:

--- a/src/include/port/pg_cpu.h
+++ b/src/include/port/pg_cpu.h
@@ -23,6 +23,12 @@ typedef enum X86FeatureId
  /* scalar registers and 128-bit XMM registers */
  PG_SSE4_2,
  PG_POPCNT,
+ PG_HYPERVISOR,

The hypervisor flag doesn't really belong with an instruction family.
Maybe a separate category like "identification"?

+ /* TSC flags */
+ PG_RDTSCP,
+ PG_TSC_INVARIANT,
+ PG_TSC_ADJUST,

Maybe spell out time stamp counter in the comment, since this will be
the first time the reader encounters that in this file.

+ * For some other Hypervisors that have an invariant TSC, e.g. HyperV, we would
+ * need to access an MSR to get the frequency (which is typically not available

Maybe spell out MSR too, because I for one don't know what that is.

+ X86Features[PG_HYPERVISOR] = reg[ECX] >> 31 & 1;
+ have_osxsave = reg[ECX] & (1 << 27);
+
+ pg_cpuid_subleaf(0x07, 0, reg);
+
+ X86Features[PG_TSC_ADJUST] = (reg[EBX] & (1 << 1)) != 0;

Some inconsistency in shift style.

-- 
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-04T09:21:14Z

On Fri, Apr 3, 2026 at 2:06 AM John Naylor <johncnaylorls@gmail.com> wrote:
>
> On Fri, Apr 3, 2026 at 6:16 AM Lukas Fittl <lukas@fittl.com> wrote:
> > v16
>
> Just some minor quibbles on 0002:

Thanks for the review!

>
> --- a/src/include/port/pg_cpu.h
> +++ b/src/include/port/pg_cpu.h
> @@ -23,6 +23,12 @@ typedef enum X86FeatureId
>   /* scalar registers and 128-bit XMM registers */
>   PG_SSE4_2,
>   PG_POPCNT,
> + PG_HYPERVISOR,
>
> The hypervisor flag doesn't really belong with an instruction family.
> Maybe a separate category like "identification"?

Yeah, I've pondered different names here, but "identification" seems
good for now - I agree it doesn't belong with the earlier flags.

>
> + /* TSC flags */
> + PG_RDTSCP,
> + PG_TSC_INVARIANT,
> + PG_TSC_ADJUST,
>
> Maybe spell out time stamp counter in the comment, since this will be
> the first time the reader encounters that in this file.

Makes sense, done.

FWIW, TSC can be spelled out either as "Time Stamp Counter" or
"Time-Stamp Counter". I've gone with the latter for now since that's
what was done elsewhere, and is how the Intel manuals reference it as
well.

>
> + * For some other Hypervisors that have an invariant TSC, e.g. HyperV, we would
> + * need to access an MSR to get the frequency (which is typically not available
>
> Maybe spell out MSR too, because I for one don't know what that is.

Done. I've also removed the note re: TSC calibration in that same
comment, since that's in a later commit and I don't think its
necessary to talk about it in that comment anyway.

>
> + X86Features[PG_HYPERVISOR] = reg[ECX] >> 31 & 1;
> + have_osxsave = reg[ECX] & (1 << 27);
> +
> + pg_cpuid_subleaf(0x07, 0, reg);
> +
> + X86Features[PG_TSC_ADJUST] = (reg[EBX] & (1 << 1)) != 0;
>
> Some inconsistency in shift style.

Fixed.

See attached v17.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-04T18:59:41Z

On Sat, Apr 4, 2026 at 2:21 AM Lukas Fittl <lukas@fittl.com> wrote:
> See attached v17.

This needed a rebase after 5e13b0f240397b2, see attached v18.

Looking at the changes in that commit, I also ended up re-ordering the
feature flags in pg_cpu.h, so we have the 128/256/512 bit registers
first, and then hypervisor and TSC flags after the 512 bit registers.

No other changes.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-04-05T02:28:59Z

On Sun, Apr 5, 2026 at 2:00 AM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Sat, Apr 4, 2026 at 2:21 AM Lukas Fittl <lukas@fittl.com> wrote:
> > See attached v17.
>
> This needed a rebase after 5e13b0f240397b2, see attached v18.
>
> Looking at the changes in that commit, I also ended up re-ordering the
> feature flags in pg_cpu.h, so we have the 128/256/512 bit registers
> first, and then hypervisor and TSC flags after the 512 bit registers.

v18-0002 looks good to me as far as feature-detection, but aside from
that there are plenty of domain-specific details that I lack the means
to check on short notice. I did verify that OSXSAVE support is still
functional.

-- 
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-06T01:47:51Z

On Sat, Apr 4, 2026 at 7:29 PM John Naylor <johncnaylorls@gmail.com> wrote:
> v18-0002 looks good to me as far as feature-detection, but aside from
> that there are plenty of domain-specific details that I lack the means
> to check on short notice. I did verify that OSXSAVE support is still
> functional.

Thanks for your input and reviews!

Attached v19 rebased over the instrument.c changes in 5a79e78501f4,
and also a slight adjustment to the commit message in 0001 to mention
the inverse pg_ns_to_ticks() function that was added for
INSTR_TIME_ADD_NANOSEC. No other changes.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-06T05:15:04Z

Hi,

Random things I noticed while looking through v19:

- src/common/instr_time.c has the wrong IDENTIFICATION


- instr_time.c should do

#ifndef FRONTEND
#include "postgres.h"
#else
#include "postgres_fe.h"
#endif

  instead of always including postgres.h


- It's not PG style to put #includes in the middle of a file as done in
  instrument.c

- tsc_use_by_default() may be documenting things that aren't the case anymore


- It may be paranoia, but it seems like tsc_calibrate() should perhaps save
  the old clock source and restore it at the end?


- A brief comment in the source (rather than just the thread), why
  TSC_CALIBRATION_SKIPS exists would be nice


- pg_initialize_timing() references an allow_tsc_calibration argument that
  doesn't exist anymore


- Wonder if it would be clearer if pg_initialize_timing() callled
  set_ticks_per_ns_system()?


- Should pg_initialize_timing() allow repeated initialization? Seems like that
  would normally be a bug?


- It's nice that pg_test_timing shows the frequency.  I was thinking it were
  able to show the result of the calibration, even if we were able to
  determine the frequency without calibration.  That should make it easier to
  figure out whether the calibration works well.


- Wonder if some of the code would look a bit cleaner if timing_tsc_enabled,
  timing_tsc_frequency_khz were defined regardless of PG_INSTR_TSC_CLOCK.


- is it ok that we are doing pg_cpuid_subleaf() without checking the result?

  It's not clear to me if a failed __get_cpuid_count() would clear out the old
  reg or leave it in place.


- I don't think the elog() in pg_initialize_timing_tsc() works in any common
  scenario, as log_min_messages hasn't been initialized yet, so that message
  won't ever be emitted.

  If it used a different log level, it'd be emitted without knowing the
  log_line_prefix etc, because that hasn't yet been initialized.


- How much do we care about weird results when dynamically changing
  timing_clocksource?

postgres[182055][1]=# EXPLAIN ANALYZE SELECT set_config('timing_clock_source', 'tsc', true), pg_sleep(1), set_config('timing_clock_source', 'system', true), pg_sleep(1);
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                 QUERY PLAN                                                 │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Result  (cost=0.00..0.01 rows=1 width=72) (actual time=-6540570569.396..-6540570569.395 rows=1.00 loops=1) │
│ Planning Time: 0.184 ms                                                                                    │
│ Execution Time: -6540570569.355 ms                                                                         │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(3 rows)

Time: 2002.350 ms (00:02.002)


- SGML:

  - "modern CPUs" seems a bit pejorative, at least if we don't merge the
    aarch64 stuff.

  - 'tsc' describes just x86-64, even if there is a patch to support aarch64.
    Perhaps it'd be enough to sprinkle a few "E.g. on x86-64, ..." around.


- I wonder if it'd be useful to have a INSTR_TIME_SET_CURRENT_SYSTEM() that'd
  always use the "system" method, but convert it to to ticks if appropriate.

  E.g. test_tsc_timing() could measure the elapsed time with system and
  display the difference in how long the test_timing() takes between tsc based
  source and system time.

  This might be more of a thing for later.


Think we're getting there...

Greetings,

Andres

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-06T11:25:36Z

On Sun, Apr 5, 2026 at 10:15 PM Andres Freund <andres@anarazel.de> wrote:
>
> - tsc_use_by_default() may be documenting things that aren't the case anymore
>

I don't think there is a correctness issue, unless you mean the fact
that we're not doing the < 8 socket check anymore that is one of the
things mentioned on the LKML posts referenced. I think the LKML post
references are still helpful as evidence why we trust that Intel has
reliable TSC handling.

I did note a bit of a grammar oddity at the end of the comment, fixed that.

> - It may be paranoia, but it seems like tsc_calibrate() should perhaps save
>   the old clock source and restore it at the end?

Sure, seems reasonable. Done.

>
> - A brief comment in the source (rather than just the thread), why
>   TSC_CALIBRATION_SKIPS exists would be nice
>

Added. I've also split the two different kinds of skipping into their
own if conditions for clarity.

>
> - Should pg_initialize_timing() allow repeated initialization? Seems like that
>   would normally be a bug?

I'm trying to recall if restore_backend_variables might have a problem
if we didn't allow that? (since we call pg_initialize_timing there,
which I think is due to ordering)

>
> - It's nice that pg_test_timing shows the frequency.  I was thinking it were
>   able to show the result of the calibration, even if we were able to
>   determine the frequency without calibration.  That should make it easier to
>   figure out whether the calibration works well.

Added. I've renamed "tsc_calibrate" to "pg_tsc_calibrate_frequency"
and exported that, to support that.

FWIW, this now calibrates twice in pg_test_timing if we're on a system
that has to use calibration. If we wanted to avoid that, we could
introduce some kind of flag that indicated the TSC frequency was
already determined through calibration. Not sure if needed?

>
> - Wonder if some of the code would look a bit cleaner if timing_tsc_enabled,
>   timing_tsc_frequency_khz were defined regardless of PG_INSTR_TSC_CLOCK.

Yeah, I don't see harm in defining them always, and its easier on the
eyes. Done. Likewise, I've also made the timing_tsc_frequency_khz in
BackendParameters defined always.

I've left pg_initialize_timing_tsc gated by PG_INSTR_TSC_CLOCK for
now, but we could also consider making that always existing, and just
not do anything.

>
> - is it ok that we are doing pg_cpuid_subleaf() without checking the result?
>
>   It's not clear to me if a failed __get_cpuid_count() would clear out the old
>   reg or leave it in place.

Hm, maybe best if we just memset reg in pg_cpuid_subleaf
unconditionally, before calling __get_cpuid_count / __cpuidex.

With that, we can be sure to have zero values for our subsequent
checks, in case it failed.

> - I don't think the elog() in pg_initialize_timing_tsc() works in any common
>   scenario, as log_min_messages hasn't been initialized yet, so that message
>   won't ever be emitted.
>
>   If it used a different log level, it'd be emitted without knowing the
>   log_line_prefix etc, because that hasn't yet been initialized.

Ah. Hmm. If I follow correctly, that's because it gets called from a
GUC check hook, and log_min_messages will only get initialized later.

I've removed it again for now.

>
> - How much do we care about weird results when dynamically changing
>   timing_clocksource?
>
> postgres[182055][1]=# EXPLAIN ANALYZE SELECT set_config('timing_clock_source', 'tsc', true), pg_sleep(1), set_config('timing_clock_source', 'system', true), pg_sleep(1);
> ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
> │                                                 QUERY PLAN                                                 │
> ├────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
> │ Result  (cost=0.00..0.01 rows=1 width=72) (actual time=-6540570569.396..-6540570569.395 rows=1.00 loops=1) │
> │ Planning Time: 0.184 ms                                                                                    │
> │ Execution Time: -6540570569.355 ms                                                                         │
> └────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
> (3 rows)
>
> Time: 2002.350 ms (00:02.002)

That was brought up earlier in the thread as well, and I added a code
comment in response.

I think the trade-off here is that if we make this a more restrictive
GUC level (the main solution I can think of), we take away the ability
for users to confirm whether the new timing logic caused their timings
to be inaccurate. And it seems very unlikely that someone would
actually change the GUC within a query (or within a function).

That said, if we think that's a problem, we should just raise the GUC
level. I don't think it'd break anything, its just less convenient.

>
>
> - SGML:
>
>   - "modern CPUs" seems a bit pejorative, at least if we don't merge the
>     aarch64 stuff.

I've revised this to "supported x86-64 CPUs".

>   - 'tsc' describes just x86-64, even if there is a patch to support aarch64.
>     Perhaps it'd be enough to sprinkle a few "E.g. on x86-64, ..." around.

Hmm. I'm not sure how we can improve that really by adding "E.g."
somewhere, but maybe I don't follow.

What I could see us doing is explicitly calling out that TSC is not
supported on other architectures?

>
> - I wonder if it'd be useful to have a INSTR_TIME_SET_CURRENT_SYSTEM() that'd
>   always use the "system" method, but convert it to to ticks if appropriate.
>
>   E.g. test_tsc_timing() could measure the elapsed time with system and
>   display the difference in how long the test_timing() takes between tsc based
>   source and system time.
>
>   This might be more of a thing for later.

Sure, I think if the conversion happened right away (i.e. we keep the
stored internal ticks value consistent with the active source) that
would be safe to do. I agree we can defer this until later.

---

Attached v20 with feedback addressed.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Zsolt Parragi <zsolt.parragi@percona.com> — 2026-04-06T23:36:27Z

+uint64 max_ticks_no_overflow = 0;

Could this be int64? It is based on PG_INT64_MAX, and it's only
practical use is with a cast to int64.

> This allows the direct use of the Time-Stamp Counter (TSC) value retrieved
> from the CPU using RDTSC/RDTSC instructions,

Typo in commit message for 0003: RDTSC/RDTSC

+ else
+ printf(_("TSC clock source is not usable. Likely unable to determine
TSC frequency. are you running in an unsupported virtualized
environment?.\n"));
+}

Typo: are, ?.


-/* TSC specific logic */
+/* Hardware clock specific logic (x86 TSC / AArch64 CNTVCT) */

Shouldn't this ARM patch also update the documentation?

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-07T00:06:36Z

Hi Zsolt,

On Mon, Apr 6, 2026 at 4:36 PM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
>
> +uint64 max_ticks_no_overflow = 0;
>
> Could this be int64? It is based on PG_INT64_MAX, and it's only
> practical use is with a cast to int64.

Its intentionally uint64, per this comment above it:

 * Note we utilize unsigned integers even though ticks are stored as a signed
 * value to encourage compilers to generate better assembly, since we can be
 * sure these values are not negative.

In my earlier Compiler Explorer tests that did actually make a
difference for the generated assembly.

>
> > This allows the direct use of the Time-Stamp Counter (TSC) value retrieved
> > from the CPU using RDTSC/RDTSC instructions,
>
> Typo in commit message for 0003: RDTSC/RDTSC
>
> + else
> + printf(_("TSC clock source is not usable. Likely unable to determine
> TSC frequency. are you running in an unsupported virtualized
> environment?.\n"));
> +}
>
> Typo: are, ?.

Agreed those are typos. Since I know Andres is actively taking a look
at things right now I'll hold off on posting a new version for this,
but thanks for noting.

>
> -/* TSC specific logic */
> +/* Hardware clock specific logic (x86 TSC / AArch64 CNTVCT) */
>
> Shouldn't this ARM patch also update the documentation?

Yes, it should, but there are other open items for the ARM patch
(mainly the detection of different CPU core types causing different
CNTVCT frequencies needs to be OS agnostic), so the ARM patch is not
targeted for 19. Its just in this patch set to illustrate how we can
expand coverage for it later.

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-07T00:40:00Z

Hi,

On 2026-04-06 04:25:36 -0700, Lukas Fittl wrote:
> On Sun, Apr 5, 2026 at 10:15 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > - tsc_use_by_default() may be documenting things that aren't the case anymore
> >
>
> I don't think there is a correctness issue, unless you mean the fact
> that we're not doing the < 8 socket check anymore that is one of the
> things mentioned on the LKML posts referenced. I think the LKML post
> references are still helpful as evidence why we trust that Intel has
> reliable TSC handling.

Well:
    "Mirrors the Linux kernel's clocksource watchdog disable logic"
doesn't seem quite right, given that in that place we are just checking
TSC_ADJUST and we don't have the < 8 socket check.

I'd probably say something like 'inspired by ... ' and mention that the rest
of the check is in tsc_detect_frequency().


I wonder if the cpuid tests should be a bit further abstracted into
pg_cpu_x86.c.

E.g. instead of tsc_detect_frequency() checking for PG_RDTSCP,
PG_TSC_INVARIANT, PG_TSC_ADJUST we could have

PG_TSC_AVAILABLE /* RDTSCP & INVARIANT */
PG_TSC_KNOWN_RELIABLE /* PG_TSC_AVAILABLE && PG_TSC_ADJUST */
PG_TSC_FREQUENCY_KNOWN /* x86_tsc_frequency_khz works */

and always run all of that during set_x86_features().


> I did note a bit of a grammar oddity at the end of the comment, fixed that.
>
> > - It may be paranoia, but it seems like tsc_calibrate() should perhaps save
> >   the old clock source and restore it at the end?
>
> Sure, seems reasonable. Done.

Cool.


> > - Should pg_initialize_timing() allow repeated initialization? Seems like that
> >   would normally be a bug?
>
> I'm trying to recall if restore_backend_variables might have a problem
> if we didn't allow that? (since we call pg_initialize_timing there,
> which I think is due to ordering)

Yea, that'd make it problematic.



> > - It's nice that pg_test_timing shows the frequency.  I was thinking it were
> >   able to show the result of the calibration, even if we were able to
> >   determine the frequency without calibration.  That should make it easier to
> >   figure out whether the calibration works well.
>
> Added. I've renamed "tsc_calibrate" to "pg_tsc_calibrate_frequency"
> and exported that, to support that.

Nice.

Workstation idle:
TSC frequency in use: 2500000 kHz
TSC frequency from calibration: 2499519 kHz

Busy:
TSC frequency in use: 2500000 kHz
TSC frequency from calibration: 2499262 kHz

Completely overwhelmed (load >1200):
TSC frequency in use: 2500000 kHz
TSC frequency from calibration: 2499405 kHz

That's very much good enough.


> FWIW, this now calibrates twice in pg_test_timing if we're on a system
> that has to use calibration. If we wanted to avoid that, we could
> introduce some kind of flag that indicated the TSC frequency was
> already determined through calibration. Not sure if needed?

I have no concern whatsoever with doing it twice in pg_test_timing.


> > - Wonder if some of the code would look a bit cleaner if timing_tsc_enabled,
> >   timing_tsc_frequency_khz were defined regardless of PG_INSTR_TSC_CLOCK.
>
> Yeah, I don't see harm in defining them always, and its easier on the
> eyes. Done. Likewise, I've also made the timing_tsc_frequency_khz in
> BackendParameters defined always.

Nice.

One thing this reminded me of is that pg_set_timing_clock_source() does:

bool
pg_set_timing_clock_source(TimingClockSourceType source)
{
	Assert(timing_initialized);

#if PG_INSTR_TSC_CLOCK
	pg_initialize_timing_tsc();

	switch (source)
	{
		case TIMING_CLOCK_SOURCE_AUTO:
			timing_tsc_enabled = (timing_tsc_frequency_khz > 0) && tsc_use_by_default();
			break;
		case TIMING_CLOCK_SOURCE_SYSTEM:
			timing_tsc_enabled = false;
			break;
		case TIMING_CLOCK_SOURCE_TSC:
			/* Tell caller TSC is not usable */
			if (timing_tsc_frequency_khz <= 0)
				return false;
			timing_tsc_enabled = true;
			break;
	}
#endif

	set_ticks_per_ns();
	timing_clock_source = source;
	return true;
}


Which means that if building without PG_INSTR_TSC_CLOCK and called with
TIMING_CLOCK_SOURCE_TSC, it'd return true despite having done something bogus.


I was also wondering if there an argument for moving the
pg_initialize_timing_tsc() into the relevant switch() cases, so the
calibration doesn't run if configured with TIMING_CLOCK_SOURCE_SYSTEM. But I
think because during GUC initialization we'll be called with the builtin
value, that wouldn't change anything?


> >
> > - is it ok that we are doing pg_cpuid_subleaf() without checking the result?
> >
> >   It's not clear to me if a failed __get_cpuid_count() would clear out the old
> >   reg or leave it in place.
>
> Hm, maybe best if we just memset reg in pg_cpuid_subleaf
> unconditionally, before calling __get_cpuid_count / __cpuidex.

Yea, I think that'd be safer.


> > - How much do we care about weird results when dynamically changing
> >   timing_clocksource?
> >
> > postgres[182055][1]=# EXPLAIN ANALYZE SELECT set_config('timing_clock_source', 'tsc', true), pg_sleep(1), set_config('timing_clock_source', 'system', true), pg_sleep(1);
> > ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
> > │                                                 QUERY PLAN                                                 │
> > ├────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
> > │ Result  (cost=0.00..0.01 rows=1 width=72) (actual time=-6540570569.396..-6540570569.395 rows=1.00 loops=1) │
> > │ Planning Time: 0.184 ms                                                                                    │
> > │ Execution Time: -6540570569.355 ms                                                                         │
> > └────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
> > (3 rows)
> >
> > Time: 2002.350 ms (00:02.002)
>
> That was brought up earlier in the thread as well, and I added a code
> comment in response.

Should there be a comment in the docs about it?


> I think the trade-off here is that if we make this a more restrictive
> GUC level (the main solution I can think of), we take away the ability
> for users to confirm whether the new timing logic caused their timings
> to be inaccurate.

Yea, it is very useful.  I guess an inbetween could be to make it SUSET.

Is there an argument that a user could hide the cost of their queries from
things like pg_stat_statements and that therefore it should be SUSET?


> And it seems very unlikely that someone would actually change the GUC within
> a query (or within a function).

I agree.  I think it's worth mentioning and worth thinking about whether it
needs to be SUSET (re the question above), but I don't see an argument for
making it PGC_SIGHUP or such.

Not for now, but I think it'd be nice if the GUC framework had a way of
expressing that some settings can only be changed at the top-level.


> >   - 'tsc' describes just x86-64, even if there is a patch to support aarch64.
> >     Perhaps it'd be enough to sprinkle a few "E.g. on x86-64, ..." around.
>
> Hmm. I'm not sure how we can improve that really by adding "E.g."
> somewhere, but maybe I don't follow.

+            <literal>tsc</literal> (measures timing using the x86-64 Time-Stamp Counter (TSC)
+            by directly executing RDTSC/RDTSCP instructions, see below)

If that instead is something like 'tsc' (measures timing with a CPU
instruction, e.g. using RDTSC/RDTSCP on x86-64) it would not be wrong even
after adding aarch64 support.



> What I could see us doing is explicitly calling out that TSC is not
> supported on other architectures?

Yea, I think it'd be good to mention that.



> Subject: [PATCH v20 1/5] instrumentation: Streamline ticks to nanosecond
>  conversion across platforms

> +static inline int64
> +pg_ticks_to_ns(int64 ticks)
>  {
> -	LARGE_INTEGER f;
> +#if PG_INSTR_TICKS_TO_NS
> +	int64		ns = 0;
> +
> +	Assert(timing_initialized);
> +
> +	/*
> +	 * Avoid doing work if we don't use scaled ticks, e.g. system clock on
> +	 * Unix
> +	 */

Maybe add something like "(in that case ticks is counted in nanoseconds)"?


Leaving aside that I don't think it makes sense to push this without also
pushing 0002/0003, I think this is eady.



> Subject: [PATCH v20 2/5] Allow retrieving x86 TSC frequency/flags from CPUID
>
> This adds additional x86 specific CPUID checks for flags needed for
> determining whether the Time-Stamp Counter (TSC) is usable on a given
> system, as well as a helper function to retrieve the TSC frequency from
> CPUID.
>
> This is intended for a future patch that will utilize the TSC to lower
> the overhead of timing instrumentation.
>
> In passing, always make pg_cpuid_subleaf reset the variables used for its
> result, to avoid accidentally using stale results if __get_cpuid_count
> errors out.



> +/*
> + * Determine the TSC frequency of the CPU through CPUID, where supported.
> + *
> + * Needed to interpret the tick value returned by RDTSC/RDTSCP. Return value of
> + * 0 indicates the frequency information was not accessible via CPUID.
> + */
> +uint32
> +x86_tsc_frequency_khz(void)
> +{
> +	unsigned int reg[4] = {0};
> +
> +	if (x86_feature_available(PG_HYPERVISOR))
> +		return x86_hypervisor_tsc_frequency_khz();


Is there a point in checking whether the things below are present if the
hypervisor specific logic doesn't find a freq?  I think it can be configured
to be passed through on some hypervisor / cpu combinations.



I think this is also close to ready, except for the minor details I raised at
the start and just here.


> From c58ea726f9c1a89eed46ee45a182457cea737d79 Mon Sep 17 00:00:00 2001
> From: Lukas Fittl <lukas@fittl.com>
> Date: Thu, 2 Apr 2026 13:17:11 -0700
> Subject: [PATCH v20 3/5] instrumentation: Use Time-Stamp Counter (TSC) on
>  x86-64 for faster measurements
>
> This allows the direct use of the Time-Stamp Counter (TSC) value retrieved
> from the CPU using RDTSC/RDTSC instructions, instead of APIs like

Missing P at the end of RDTSC/RDTSC.


> clock_gettime() on POSIX systems. This reduces the overhead of EXPLAIN with
> ANALYZE and TIMING ON. Tests showed that runtime when instrumented can be
> reduced by up to 10% for queries moving lots of rows through the plan.

FWIW, I see considerably bigger gains in some cases. Mostly queries with many
query "levels". But even some simple ones:


Baseline:

SELECT * FROM pgbench_accounts LIMIT 1 OFFSET 10000000;
\timing reports 322.548 ms

Baseline with EXPLAIN ANALYZE overhead:


EXPLAIN (ANALYZE, BUFFERS 0, TIMING OFF) SELECT * FROM pgbench_accounts LIMIT 1 OFFSET 10000000;

QUERY PLAN
Limit  (cost=168370.00..168370.02 rows=1 width=97) (actual rows=0.00 loops=1)
  ->  Seq Scan on pgbench_accounts  (cost=0.00..168370.00 rows=10000000 width=97) (actual rows=10000000.00 loops=1)
Planning Time: 0.059 ms
Execution Time: 426.570 ms

1.32 x slowdown.


SET timing_clock_source = 'system';
EXPLAIN (ANALYZE, BUFFERS 0) SELECT * FROM pgbench_accounts LIMIT 1 OFFSET 10000000;

Limit  (cost=168370.00..168370.02 rows=1 width=97) (actual time=882.843..882.843 rows=0.00 loops=1)
  ->  Seq Scan on pgbench_accounts  (cost=0.00..168370.00 rows=10000000 width=97) (actual time=0.021..593.587 rows=10000000.00 loops=1)
Planning Time: 0.063 ms
Execution Time: 882.860 ms

2.06 x slowdown relative to TIMING OFF


SET timing_clock_source = 'tsc';
Limit  (cost=168370.00..168370.02 rows=1 width=97) (actual time=543.098..543.098 rows=0.00 loops=1)
  ->  Seq Scan on pgbench_accounts  (cost=0.00..168370.00 rows=10000000 width=97) (actual time=0.017..413.878 rows=10000000.00 loops=1)
Planning Time: 0.061 ms
Execution Time: 543.122 ms

1.27 x slowdown relative to TIMING OFF

1.63x speedup relative to system.


But I also see ~20% gains for some TPCH queries, for example.


> To control use of the TSC, the new "timing_clock_source" GUC is introduced,
> whose default ("auto") automatically uses the TSC when running on Linux/x86-64,
> in case the system clocksource is reported as "tsc". The use of the system
> APIs can be enforced by setting "system", or on x86-64 architectures the
> use of TSC can be enforced by explicitly setting "tsc".

It's more widely enabled by default now, right?


> In order to use the TSC the frequency is first determined by use of CPUID,
> and if not available, by running a short calibration loop at program start,
> falling back to the system time if TSC values are not stable.
>
> Note, that we split TSC usage into the RDTSC CPU instruction which does not
> wait for out-of-order execution (faster, less precise) and the RDTSCP instruction,
> which waits for outstanding instructions to retire. RDTSCP is deemed to have
> little benefit in the typical InstrStartNode() / InstrStopNode() use case of
> EXPLAIN, and can be up to twice as slow. To separate these use cases, the new
> macro INSTR_TIME_SET_CURRENT_FAST() is introduced, which uses RDTSC.
>
> The original macro INSTR_TIME_SET_CURRENT() uses RDTSCP and is supposed
> to be used when precision is more important than performance. When the
> system timing clock source is used both of these macros instead utilize
> the system APIs (clock_gettime / QueryPerformanceCounter) like before.

Maybe worth adding that there are other things that may be worth converting,
like track_io_timing/track_wal_io_timing.


> +const char *
> +show_timing_clock_source(void)
> +{
> +	switch (timing_clock_source)
> +	{
> +		case TIMING_CLOCK_SOURCE_AUTO:
> +#if PG_INSTR_TSC_CLOCK
> +			if (pg_current_timing_clock_source() == TIMING_CLOCK_SOURCE_TSC)
> +				return "auto (tsc)";
> +#endif
> +			return "auto (system)";
> +		case TIMING_CLOCK_SOURCE_SYSTEM:
> +			return "system";
> +#if PG_INSTR_TSC_CLOCK
> +		case TIMING_CLOCK_SOURCE_TSC:
> +			return "tsc";
> +#endif

For a moment I was wondering if we should have this display the frequency and
whether it's calibrated. But I think that's too cute by half.



> +static void
> +set_ticks_per_ns(void)
> +{
> +#if PG_INSTR_TSC_CLOCK
> +	if (timing_tsc_enabled)
> +		set_ticks_per_ns_for_tsc();
> +	else
> +		set_ticks_per_ns_system();
> +#else
> +	set_ticks_per_ns_system();
> +#endif
> +}

How about?

static void
set_ticks_per_ns(void)
{
#if PG_INSTR_TSC_CLOCK
	if (timing_tsc_enabled)
	{
		set_ticks_per_ns_for_tsc();
		return;
	}
#endif
	set_ticks_per_ns_system();
}


> @@ -83,27 +88,90 @@ typedef struct instr_time
>  /* Shift amount for fixed-point ticks-to-nanoseconds conversion. */
>  #define TICKS_TO_NS_SHIFT 14
>
> -#ifdef WIN32
> -#define PG_INSTR_TICKS_TO_NS 1
> -#else
> -#define PG_INSTR_TICKS_TO_NS 0
> -#endif
> -

I'd add it to the place it'll later be added.


Think this is quite close.



> Subject: [PATCH v20 4/5] pg_test_timing: Also test RDTSC/RDTSCP timing and
>  report time source and TSC frequency


> +		/* Now, emit fast timing measurements */
> +		loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, true);
> +		output(loop_count);
> +		printf("\n");
> +
> +		printf(_("TSC frequency in use: %u kHz\n"), timing_tsc_frequency_khz);
> +
> +		calibrated_freq = pg_tsc_calibrate_frequency();
> +		if (calibrated_freq > 0)
> +			printf(_("TSC frequency from calibration: %u kHz\n"), calibrated_freq);
> +		else
> +			printf(_("TSC calibration did not converge\n"));

If this were to indicate if the current frequency were from a non-calibration
source it'd be perfect, but that's definitely not required.



> Subject: [PATCH v20 5/5] instrumentation: ARM support for fast time
>  measurements
>
> Similar to the RDTSC/RDTSCP instructions on x68-64, this introduces
> use of the cntvct_el0 instruction on ARM systems to access the generic
> timer that provides a synchronized ticks value across CPUs.
>
> Note this adds an exception for Apple Silicon CPUs, due to the observed
> fact that M3 and newer has different timer frequencies for the Efficiency
> and the Performance cores, and we can't be sure where we get scheduled.
>
> To simplify the implementation this does not support Windows on ARM,
> since its quite rare and hard to test.
>
> Relies on the existing timing_clock_source GUC to control whether
> TSC-like timer gets used, instead of system timer.


> +/*
> + * Check whether this is a heterogeneous Apple Silicon P+E core system
> + * where CNTVCT_EL0 may tick at different rates on different core types.
> + */
> +static bool
> +aarch64_has_heterogeneous_cores(void)
> +{
> +#if defined(__APPLE__)
> +	int			nperflevels = 0;
> +	size_t		len = sizeof(nperflevels);
> +
> +	if (sysctlbyname("hw.nperflevels", &nperflevels, &len, NULL, 0) == 0)
> +		return nperflevels > 1;
> +#endif
> +
> +	return false;
> +}
> +
> +/*
> + * Detect the generic timer frequency on AArch64.
> + */
> +static void
> +tsc_detect_frequency(void)
> +{
> +	if (aarch64_has_heterogeneous_cores())
> +	{
> +		timing_tsc_frequency_khz = 0;
> +		return;
> +	}
> +
> +	timing_tsc_frequency_khz = aarch64_cntvct_frequency_khz();
> +}

> +/*
> + * The ARM generic timer is architecturally guaranteed to be monotonic and
> + * synchronized across cores of the same type, so we always use it by default
> + * when available and cores are homogenous.
> + */
> +static bool
> +tsc_use_by_default(void)
> +{
> +	return true;
> +}

I'm somewhat sceptical of that being viable, given that we only have support
for detect heteregenous cores on macos.  You e.g. can run linux on M*
hardware. And I wonder if other big.little heterogeneous architectures have
the same probem...


> +uint32
> +pg_tsc_calibrate_frequency(void)
> +{
> +	/* No calibration loop on AArch64; frequency comes from CNTFRQ_EL0 */
> +	return 0;
> +}

Think I'd advocate for support for that if/when we add ARM support, even if
it's just to be able to verify things are sane via pg_test_timing.


> @@ -144,7 +150,6 @@ extern bool pg_set_timing_clock_source(TimingClockSourceType source);
>  #define PG_INSTR_TICKS_TO_NS 0
>  #endif
>
> -
>  /* Whether to actually use TSC based on availability and GUC settings. */
>  extern PGDLLIMPORT bool timing_tsc_enabled;
>

Spurious line change.


> +#elif defined(__aarch64__) && !defined(WIN32)
> +
> +/*
> + * Read the ARM generic timer counter (CNTVCT_EL0).
> + *
> + * The "fast" variant reads the counter without a barrier, analogous to RDTSC
> + * on x86. The regular variant issues an ISB (Instruction Synchronization
> + * Barrier) first, which acts as a serializing instruction analogous to RDTSCP,
> + * ensuring all preceding instructions have completed before reading the
> + * counter.
> + */
> +static inline instr_time
> +pg_get_ticks_fast(void)
> +{
> +	if (likely(timing_tsc_enabled))
> +	{
> +		instr_time	now;
> +
> +		now.ticks = __builtin_arm_rsr64("cntvct_el0");
> +		return now;

Seems like this is about !msvc (or rather a gcc like compiler), rather than
about windows?


Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-07T03:41:46Z

On Mon, Apr 6, 2026 at 5:40 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-04-06 04:25:36 -0700, Lukas Fittl wrote:
> > On Sun, Apr 5, 2026 at 10:15 PM Andres Freund <andres@anarazel.de> wrote:
> > >
> > > - tsc_use_by_default() may be documenting things that aren't the case anymore
> > >
> >
> > I don't think there is a correctness issue, unless you mean the fact
> > that we're not doing the < 8 socket check anymore that is one of the
> > things mentioned on the LKML posts referenced. I think the LKML post
> > references are still helpful as evidence why we trust that Intel has
> > reliable TSC handling.
>
> Well:
>     "Mirrors the Linux kernel's clocksource watchdog disable logic"
> doesn't seem quite right, given that in that place we are just checking
> TSC_ADJUST and we don't have the < 8 socket check.
>
> I'd probably say something like 'inspired by ... ' and mention that the rest
> of the check is in tsc_detect_frequency().

Yeah, that makes sense. Reworded.

> I wonder if the cpuid tests should be a bit further abstracted into
> pg_cpu_x86.c.
>
> E.g. instead of tsc_detect_frequency() checking for PG_RDTSCP,
> PG_TSC_INVARIANT, PG_TSC_ADJUST we could have
>
> PG_TSC_AVAILABLE /* RDTSCP & INVARIANT */
> PG_TSC_KNOWN_RELIABLE /* PG_TSC_AVAILABLE && PG_TSC_ADJUST */
> PG_TSC_FREQUENCY_KNOWN /* x86_tsc_frequency_khz works */
>
> and always run all of that during set_x86_features().

I think that could work, but I kept the flags in features closer to
being direct mappings to CPUID bits since that seemed to be intent of
how John designed the facility originally.

John, do you have thoughts on this? (I've not changed it for now)

FWIW, I don't think having PG_TSC_KNOWN_RELIABLE makes sense in any
case, because that would tie together x86_tsc_frequency_khz and
set_x86_features, i.e. you'd either have the frequency return function
modify X86Features later, or always run x86_tsc_frequency_khz when
setting features (and that'd then require you to put the frequency
value somewhere, etc.)

> > > - It's nice that pg_test_timing shows the frequency.  I was thinking it were
> > >   able to show the result of the calibration, even if we were able to
> > >   determine the frequency without calibration.  That should make it easier to
> > >   figure out whether the calibration works well.
> >
> > Added. I've renamed "tsc_calibrate" to "pg_tsc_calibrate_frequency"
> > and exported that, to support that.
>
> Nice.
>
> Workstation idle:
> TSC frequency in use: 2500000 kHz
> TSC frequency from calibration: 2499519 kHz
>
> Busy:
> TSC frequency in use: 2500000 kHz
> TSC frequency from calibration: 2499262 kHz
>
> Completely overwhelmed (load >1200):
> TSC frequency in use: 2500000 kHz
> TSC frequency from calibration: 2499405 kHz
>
> That's very much good enough.
>

Nice, thanks for confirming!

> > > - Wonder if some of the code would look a bit cleaner if timing_tsc_enabled,
> > >   timing_tsc_frequency_khz were defined regardless of PG_INSTR_TSC_CLOCK.
> >
> > Yeah, I don't see harm in defining them always, and its easier on the
> > eyes. Done. Likewise, I've also made the timing_tsc_frequency_khz in
> > BackendParameters defined always.
>
> Nice.
>
> One thing this reminded me of is that pg_set_timing_clock_source() does:
>
> bool
> pg_set_timing_clock_source(TimingClockSourceType source)
> {
>         Assert(timing_initialized);
>
> #if PG_INSTR_TSC_CLOCK
>         pg_initialize_timing_tsc();
>
>         switch (source)
>         {
>                 case TIMING_CLOCK_SOURCE_AUTO:
>                         timing_tsc_enabled = (timing_tsc_frequency_khz > 0) && tsc_use_by_default();
>                         break;
>                 case TIMING_CLOCK_SOURCE_SYSTEM:
>                         timing_tsc_enabled = false;
>                         break;
>                 case TIMING_CLOCK_SOURCE_TSC:
>                         /* Tell caller TSC is not usable */
>                         if (timing_tsc_frequency_khz <= 0)
>                                 return false;
>                         timing_tsc_enabled = true;
>                         break;
>         }
> #endif
>
>         set_ticks_per_ns();
>         timing_clock_source = source;
>         return true;
> }
>
>
> Which means that if building without PG_INSTR_TSC_CLOCK and called with
> TIMING_CLOCK_SOURCE_TSC, it'd return true despite having done something bogus.

Right, that is slightly odd that we allow that.

I think the easiest way to make that very clear is to hide the
existence of the TIMING_CLOCK_SOURCE_TSC enum value behind an
PG_INSTR_TSC_CLOCK gate. I think that's permissible for this kind of
enum, and we also do this for the GUC value already anyway, and we had
no references to that value that weren't already behind a
PG_INSTR_TSC_CLOCK check.

Done that way.

That also required moving some of the #defines in instr_time.h around
so they are defined for defining TimingClockSourceType. I've opted to
have PG_INSTR_TSC_CLOCK/PG_INSTR_TICKS_TO_NS defined early in the
file, but put the clock source names now consistently next to the code
that implements pg_get_ticks. I think that flows nicely now.

>
> I was also wondering if there an argument for moving the
> pg_initialize_timing_tsc() into the relevant switch() cases, so the
> calibration doesn't run if configured with TIMING_CLOCK_SOURCE_SYSTEM. But I
> think because during GUC initialization we'll be called with the builtin
> value, that wouldn't change anything?

Yeah, unless the default was something different than "auto", we can't
get out of having to run "pg_initialize_timing_tsc". I think moving
pg_initialize_timing_tsc into the individual switch cases makes the
code more verbose without saving much, so I've left this as is for
now.

>
>
> > >
> > > - is it ok that we are doing pg_cpuid_subleaf() without checking the result?
> > >
> > >   It's not clear to me if a failed __get_cpuid_count() would clear out the old
> > >   reg or leave it in place.
> >
> > Hm, maybe best if we just memset reg in pg_cpuid_subleaf
> > unconditionally, before calling __get_cpuid_count / __cpuidex.
>
> Yea, I think that'd be safer.

FWIW, done that way in the v20 version already, I just didn't say that clearly.

>
> > > - How much do we care about weird results when dynamically changing
> > >   timing_clocksource?
> > >
> > > postgres[182055][1]=# EXPLAIN ANALYZE SELECT set_config('timing_clock_source', 'tsc', true), pg_sleep(1), set_config('timing_clock_source', 'system', true), pg_sleep(1);
> > > ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
> > > │                                                 QUERY PLAN                                                 │
> > > ├────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
> > > │ Result  (cost=0.00..0.01 rows=1 width=72) (actual time=-6540570569.396..-6540570569.395 rows=1.00 loops=1) │
> > > │ Planning Time: 0.184 ms                                                                                    │
> > > │ Execution Time: -6540570569.355 ms                                                                         │
> > > └────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
> > > (3 rows)
> > >
> > > Time: 2002.350 ms (00:02.002)
> >
> > That was brought up earlier in the thread as well, and I added a code
> > comment in response.
>
> Should there be a comment in the docs about it?

Lets do it. I've added the following note to the docs:

Changing the setting during query execution is not recommended
and may cause interval timings to jump significantly or produce negative values.

>
> > I think the trade-off here is that if we make this a more restrictive
> > GUC level (the main solution I can think of), we take away the ability
> > for users to confirm whether the new timing logic caused their timings
> > to be inaccurate.
>
> Yea, it is very useful.  I guess an inbetween could be to make it SUSET.
>
> Is there an argument that a user could hide the cost of their queries from
> things like pg_stat_statements and that therefore it should be SUSET?

Yeah, that's a valid point - per your test it could allow unprivileged
users to pretend their expensive operations are cheap, or potentially
corrupt aggregated timings intentionally. And I think the likely
person testing timing sources is an operator anyway (i.e. this is not
intended for an application to change, etc), so we're not losing much
value.

I've adjusted it to SUSET and added a note in the docs its superuser only.

>
> > And it seems very unlikely that someone would actually change the GUC within
> > a query (or within a function).
>
> I agree.  I think it's worth mentioning and worth thinking about whether it
> needs to be SUSET (re the question above), but I don't see an argument for
> making it PGC_SIGHUP or such.
>
> Not for now, but I think it'd be nice if the GUC framework had a way of
> expressing that some settings can only be changed at the top-level.

Agreed, that'd be useful to have.

>
> > >   - 'tsc' describes just x86-64, even if there is a patch to support aarch64.
> > >     Perhaps it'd be enough to sprinkle a few "E.g. on x86-64, ..." around.
> >
> > Hmm. I'm not sure how we can improve that really by adding "E.g."
> > somewhere, but maybe I don't follow.
>
> +            <literal>tsc</literal> (measures timing using the x86-64 Time-Stamp Counter (TSC)
> +            by directly executing RDTSC/RDTSCP instructions, see below)
>
> If that instead is something like 'tsc' (measures timing with a CPU
> instruction, e.g. using RDTSC/RDTSCP on x86-64) it would not be wrong even
> after adding aarch64 support.
>

Thanks, that is a good rewording - done.

>
> > What I could see us doing is explicitly calling out that TSC is not
> > supported on other architectures?
>
> Yea, I think it'd be good to mention that.
>

I've gone ahead and rewritten that whole paragraph for clarity, and
also split it into two. Feedback welcome:

<para>
  If enabled, the TSC clock source will use specialized CPU instructions
  when measuring time intervals. This lowers timing overhead compared to
  reading the OS system clock, and reduces the measurement error on top
  of the actual runtime, for example with EXPLAIN ANALYZE.
</para>
<para>
  On x86-64 CPUs the TSC clock source utilizes the Time-Stamp Counter (TSC)
  of the CPU. The RDTSC instruction is used to read the TSC for EXPLAIN ANALYZE.
  For timings that require higher precision the RDTSCP instruction is used,
  which avoids inaccuracies due to CPU instruction re-ordering. Use of
  RDTSC/RDTSCP is not supported on older x86-64 CPUs or hypervisors that don't
  pass the TSC frequency to guest VMs, and is not advised on systems that
  utilize an emulated TSC. The TSC clock source is currently not supported on
  other architectures.
</para>
<para>
  To help decide which clock source to use you can run the
<application>pg_test_timing</application>
  utility to check TSC availability, and perform timing measurements.
</para>

>
> > Subject: [PATCH v20 1/5] instrumentation: Streamline ticks to nanosecond
> >  conversion across platforms
>
> ...
>
> Leaving aside that I don't think it makes sense to push this without also
> pushing 0002/0003, I think this is eady.
>

Great!

>
> > Subject: [PATCH v20 2/5] Allow retrieving x86 TSC frequency/flags from CPUID
> >
>
> > +/*
> > + * Determine the TSC frequency of the CPU through CPUID, where supported.
> > + *
> > + * Needed to interpret the tick value returned by RDTSC/RDTSCP. Return value of
> > + * 0 indicates the frequency information was not accessible via CPUID.
> > + */
> > +uint32
> > +x86_tsc_frequency_khz(void)
> > +{
> > +     unsigned int reg[4] = {0};
> > +
> > +     if (x86_feature_available(PG_HYPERVISOR))
> > +             return x86_hypervisor_tsc_frequency_khz();
>
>
> Is there a point in checking whether the things below are present if the
> hypervisor specific logic doesn't find a freq?  I think it can be configured
> to be passed through on some hypervisor / cpu combinations.
>

Yeah, when I wrote that I pondered the same question. I don't see harm
in falling back to the below if the hypervisor frequency is 0.
Adjusted that way.

>
> I think this is also close to ready, except for the minor details I raised at
> the start and just here.
>

Great. Thank you for the thorough review, as always :)

> > clock_gettime() on POSIX systems. This reduces the overhead of EXPLAIN with
> > ANALYZE and TIMING ON. Tests showed that runtime when instrumented can be
> > reduced by up to 10% for queries moving lots of rows through the plan.
>
> FWIW, I see considerably bigger gains in some cases. Mostly queries with many
> query "levels". But even some simple ones:
>
>
> Baseline:
>
> SELECT * FROM pgbench_accounts LIMIT 1 OFFSET 10000000;
> \timing reports 322.548 ms
>
> Baseline with EXPLAIN ANALYZE overhead:
>
>
> EXPLAIN (ANALYZE, BUFFERS 0, TIMING OFF) SELECT * FROM pgbench_accounts LIMIT 1 OFFSET 10000000;
>
> QUERY PLAN
> Limit  (cost=168370.00..168370.02 rows=1 width=97) (actual rows=0.00 loops=1)
>   ->  Seq Scan on pgbench_accounts  (cost=0.00..168370.00 rows=10000000 width=97) (actual rows=10000000.00 loops=1)
> Planning Time: 0.059 ms
> Execution Time: 426.570 ms
>
> 1.32 x slowdown.
>
>
> SET timing_clock_source = 'system';
> EXPLAIN (ANALYZE, BUFFERS 0) SELECT * FROM pgbench_accounts LIMIT 1 OFFSET 10000000;
>
> Limit  (cost=168370.00..168370.02 rows=1 width=97) (actual time=882.843..882.843 rows=0.00 loops=1)
>   ->  Seq Scan on pgbench_accounts  (cost=0.00..168370.00 rows=10000000 width=97) (actual time=0.021..593.587 rows=10000000.00 loops=1)
> Planning Time: 0.063 ms
> Execution Time: 882.860 ms
>
> 2.06 x slowdown relative to TIMING OFF
>
>
> SET timing_clock_source = 'tsc';
> Limit  (cost=168370.00..168370.02 rows=1 width=97) (actual time=543.098..543.098 rows=0.00 loops=1)
>   ->  Seq Scan on pgbench_accounts  (cost=0.00..168370.00 rows=10000000 width=97) (actual time=0.017..413.878 rows=10000000.00 loops=1)
> Planning Time: 0.061 ms
> Execution Time: 543.122 ms
>
> 1.27 x slowdown relative to TIMING OFF
>
> 1.63x speedup relative to system.
>
>
> But I also see ~20% gains for some TPCH queries, for example.

Thanks for some fresh numbers! I've reworded this now as:

This reduces the overhead of EXPLAIN with ANALYZE and TIMING ON. Tests showed
that the overhead on top of actual runtime when instrumenting queries moving
lots of rows through the plan can be reduced from 2x as slow to 1.2x as slow
compared to the actual runtime. More complex workloads such as TPCH queries
have also shown ~20% gains when instrumented compared to before.

>
>
> > To control use of the TSC, the new "timing_clock_source" GUC is introduced,
> > whose default ("auto") automatically uses the TSC when running on Linux/x86-64,
> > in case the system clocksource is reported as "tsc". The use of the system
> > APIs can be enforced by setting "system", or on x86-64 architectures the
> > use of TSC can be enforced by explicitly setting "tsc".
>
> It's more widely enabled by default now, right?

Reworded as:

To control use of the TSC, the new "timing_clock_source" GUC is introduced,
whose default ("auto") automatically uses the TSC when reliable, for example
when running on modern Intel CPUs, or when running on Linux and the system
clocksource is reported as "tsc". The use of the operating system clock
source can be enforced by setting "system", or on x86-64 architectures
the use of TSC can be enforced by explicitly setting "tsc".

>
> > In order to use the TSC the frequency is first determined by use of CPUID,
> > and if not available, by running a short calibration loop at program start,
> > falling back to the system time if TSC values are not stable.
> >
> > Note, that we split TSC usage into the RDTSC CPU instruction which does not
> > wait for out-of-order execution (faster, less precise) and the RDTSCP instruction,
> > which waits for outstanding instructions to retire. RDTSCP is deemed to have
> > little benefit in the typical InstrStartNode() / InstrStopNode() use case of
> > EXPLAIN, and can be up to twice as slow. To separate these use cases, the new
> > macro INSTR_TIME_SET_CURRENT_FAST() is introduced, which uses RDTSC.
> >
> > The original macro INSTR_TIME_SET_CURRENT() uses RDTSCP and is supposed
> > to be used when precision is more important than performance. When the
> > system timing clock source is used both of these macros instead utilize
> > the system APIs (clock_gettime / QueryPerformanceCounter) like before.
>
> Maybe worth adding that there are other things that may be worth converting,
> like track_io_timing/track_wal_io_timing.

I've added the following:

Additional users of interval timing, such as track_io_timing and
track_wal_io_timing could also benefit from being converted to use
INSTR_TIME_SET_CURRENT_FAST() but are left for a future change.

>
> > +const char *
> > +show_timing_clock_source(void)
> > +{
> > +     switch (timing_clock_source)
> > +     {
> > +             case TIMING_CLOCK_SOURCE_AUTO:
> > +#if PG_INSTR_TSC_CLOCK
> > +                     if (pg_current_timing_clock_source() == TIMING_CLOCK_SOURCE_TSC)
> > +                             return "auto (tsc)";
> > +#endif
> > +                     return "auto (system)";
> > +             case TIMING_CLOCK_SOURCE_SYSTEM:
> > +                     return "system";
> > +#if PG_INSTR_TSC_CLOCK
> > +             case TIMING_CLOCK_SOURCE_TSC:
> > +                     return "tsc";
> > +#endif
>
> For a moment I was wondering if we should have this display the frequency and
> whether it's calibrated. But I think that's too cute by half.
>

Yeah, I already felt initially unsure whether the "auto (...)"
mechanism was a bit too novel. I don't think we should put the
frequency/calibration status into the show hook.

>
> > @@ -83,27 +88,90 @@ typedef struct instr_time
> >  /* Shift amount for fixed-point ticks-to-nanoseconds conversion. */
> >  #define TICKS_TO_NS_SHIFT 14
> >
> > -#ifdef WIN32
> > -#define PG_INSTR_TICKS_TO_NS 1
> > -#else
> > -#define PG_INSTR_TICKS_TO_NS 0
> > -#endif
> > -
>
> I'd add it to the place it'll later be added.

Addressed this by keeping it below TICKS_TO_NS_SHIFT always, and
setting PG_INSTR_TSC_CLOCK in the same spot as well, per the earlier
note of that being needed for the enum definition. I've also added a
comment explaining what these get used for.

> > Subject: [PATCH v20 4/5] pg_test_timing: Also test RDTSC/RDTSCP timing and
> >  report time source and TSC frequency
>
>
> > +             /* Now, emit fast timing measurements */
> > +             loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_TSC, true);
> > +             output(loop_count);
> > +             printf("\n");
> > +
> > +             printf(_("TSC frequency in use: %u kHz\n"), timing_tsc_frequency_khz);
> > +
> > +             calibrated_freq = pg_tsc_calibrate_frequency();
> > +             if (calibrated_freq > 0)
> > +                     printf(_("TSC frequency from calibration: %u kHz\n"), calibrated_freq);
> > +             else
> > +                     printf(_("TSC calibration did not converge\n"));
>
> If this were to indicate if the current frequency were from a non-calibration
> source it'd be perfect, but that's definitely not required.
>

I mostly didn't want to add yet another extra variable to keep that
information (and per my note above, I don't think that makes sense to
put in x86 features itself). I've left this as-is for now. I think if
people are unsure whether their CPUID gives the necessary information,
they could always run a separate utility like "cpuid" to confirm.

>
> > Subject: [PATCH v20 5/5] instrumentation: ARM support for fast time
> >  measurements
> >
> ...
>
> > +/*
> > + * The ARM generic timer is architecturally guaranteed to be monotonic and
> > + * synchronized across cores of the same type, so we always use it by default
> > + * when available and cores are homogenous.
> > + */
> > +static bool
> > +tsc_use_by_default(void)
> > +{
> > +     return true;
> > +}
>
> I'm somewhat sceptical of that being viable, given that we only have support
> for detect heteregenous cores on macos.  You e.g. can run linux on M*
> hardware. And I wonder if other big.little heterogeneous architectures have
> the same probem...

Ack. I think we could make this dependent on "were we able to
determine that its a homogeneous architecture", and if we weren't, we
default to off.

My understanding is that Apple did something funny here with Apple
Silicon on M3+ (specifically with how they did the timer handling for
the different core types), but I'm not an expert at the ARM
architecture to claim it won't be a problem on other ARM platforms.

>
> > +uint32
> > +pg_tsc_calibrate_frequency(void)
> > +{
> > +     /* No calibration loop on AArch64; frequency comes from CNTFRQ_EL0 */
> > +     return 0;
> > +}
>
> Think I'd advocate for support for that if/when we add ARM support, even if
> it's just to be able to verify things are sane via pg_test_timing.

We could, but the timer on ARM is actually more trustworthy in the
sense that the frequency is always fixed - until they decided to raise
that always fixed value for modern ARM, and Apple decided to mix the
two variants..

But there is nothing in the TSC frequency calibration that couldn't
allow ARM instructions to be used instead.

---

See attached v21.

I've also marked pg_get_ticks(_fast) as pg_attribute_always_inline,
per an off-list comment from Andres that he observed GCC not fully
inlining that function in pg_test_timing, presumably due to the
likely(..) in it.

Additionally, I reworded the 0001 commit as "instrumentation:
Standardize ticks to nanosecond conversion method" (instead of
"Streamline ticks to nanosecond conversion across platforms"), I think
that's a better headline of what's happening, since in that commit
itself we're still only doing ticks to nanoseconds on Windows.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-07T04:55:45Z

Hi,

On 2026-04-06 20:41:46 -0700, Lukas Fittl wrote:
> On Mon, Apr 6, 2026 at 5:40 PM Andres Freund <andres@anarazel.de> wrote:
> > I wonder if the cpuid tests should be a bit further abstracted into
> > pg_cpu_x86.c.
> >
> > E.g. instead of tsc_detect_frequency() checking for PG_RDTSCP,
> > PG_TSC_INVARIANT, PG_TSC_ADJUST we could have
> >
> > PG_TSC_AVAILABLE /* RDTSCP & INVARIANT */
> > PG_TSC_KNOWN_RELIABLE /* PG_TSC_AVAILABLE && PG_TSC_ADJUST */
> > PG_TSC_FREQUENCY_KNOWN /* x86_tsc_frequency_khz works */
> >
> > and always run all of that during set_x86_features().
> 
> I think that could work, but I kept the flags in features closer to
> being direct mappings to CPUID bits since that seemed to be intent of
> how John designed the facility originally.
> 
> John, do you have thoughts on this? (I've not changed it for now)

I'm ok either way.


> FWIW, I don't think having PG_TSC_KNOWN_RELIABLE makes sense in any
> case, because that would tie together x86_tsc_frequency_khz and
> set_x86_features, i.e. you'd either have the frequency return function
> modify X86Features later, or always run x86_tsc_frequency_khz when
> setting features (and that'd then require you to put the frequency
> value somewhere, etc.)

I was thinking the latter.

> I've gone ahead and rewritten that whole paragraph for clarity, and
> also split it into two. Feedback welcome:
> 
> <para>
>   If enabled, the TSC clock source will use specialized CPU instructions
>   when measuring time intervals. This lowers timing overhead compared to
>   reading the OS system clock, and reduces the measurement error on top
>   of the actual runtime, for example with EXPLAIN ANALYZE.
> </para>


> <para>
>   On x86-64 CPUs the TSC clock source utilizes the Time-Stamp Counter (TSC)

It's a bit weird that the third use of TSC in these paragraphs introduces
Time-Stamp Counter. I can see how you get there, but ...

Now I wonder if we should rename 'tsc' to 'cpu'...


>   of the CPU. The RDTSC instruction is used to read the TSC for EXPLAIN ANALYZE.
>   For timings that require higher precision the RDTSCP instruction is used,
>   which avoids inaccuracies due to CPU instruction re-ordering. Use of
>   RDTSC/RDTSCP is not supported on older x86-64 CPUs or hypervisors that don't
>   pass the TSC frequency to guest VMs, and is not advised on systems that

s/guest VMs/virtual machines/?

>   utilize an emulated TSC. The TSC clock source is currently not supported on
>   other architectures.

The not support bit about hypervisors isn't quite right though? We do even use
it automatically if TSC_ADJUST is set (and the calibration loop succeeds).



> </para>
> <para>
>   To help decide which clock source to use you can run the
> <application>pg_test_timing</application>
>   utility to check TSC availability, and perform timing measurements.
> </para>

How about a link to to the pg_test_timing page?  Hm, I guess that should also
be updated with new output.


I'd also sprinkle a few <acronym> and <command>s around.


Wonder if it's worth adding something like
        <indexterm><primary><acronym>RDTSC</acronym></primary></indexterm>
        <indexterm>
         <primary>Time-Stamp Counter</primary>
         <see><acronym>TSC</acronym></see>
        </indexterm>
        <indexterm><primary><acronym>TSC</acronym></primary></indexterm>

otherwise somebody seeing one of these in logs, pg_test_timing output or
whatever has even less of a chance to figure it out within our docs. They're
not hard to search for terms exactly, so ...


> I've also marked pg_get_ticks(_fast) as pg_attribute_always_inline,
> per an off-list comment from Andres that he observed GCC not fully
> inlining that function in pg_test_timing, presumably due to the
> likely(..) in it.

It's not the likely, I reproduced it even without that.  I mouthed off about
compilers on mastodon and was kindly asked to just open a bug report :)

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=124795


I think discussing which indexterms should be added signals that this is
pretty close.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

John Naylor <johncnaylorls@gmail.com> — 2026-04-07T04:56:04Z

On Tue, Apr 7, 2026 at 10:42 AM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Mon, Apr 6, 2026 at 5:40 PM Andres Freund <andres@anarazel.de> wrote:

> > I wonder if the cpuid tests should be a bit further abstracted into
> > pg_cpu_x86.c.
> >
> > E.g. instead of tsc_detect_frequency() checking for PG_RDTSCP,
> > PG_TSC_INVARIANT, PG_TSC_ADJUST we could have
> >
> > PG_TSC_AVAILABLE /* RDTSCP & INVARIANT */
> > PG_TSC_KNOWN_RELIABLE /* PG_TSC_AVAILABLE && PG_TSC_ADJUST */
> > PG_TSC_FREQUENCY_KNOWN /* x86_tsc_frequency_khz works */
> >
> > and always run all of that during set_x86_features().
>
> I think that could work, but I kept the flags in features closer to
> being direct mappings to CPUID bits since that seemed to be intent of
> how John designed the facility originally.
>
> John, do you have thoughts on this? (I've not changed it for now)

The intent is as how you described, but it's not because that way was
the best of all possible ways, more that it seemed to fit well
underneath the existing runtime checks. It's also already not quite a
direct mapping, since if OSXSAVE is not available, we don't bother
checking for/setting AVX at all.

--
John Naylor
Amazon Web Services

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-07T07:04:51Z

On Mon, Apr 6, 2026 at 9:55 PM Andres Freund <andres@anarazel.de> wrote:
> > FWIW, I don't think having PG_TSC_KNOWN_RELIABLE makes sense in any
> > case, because that would tie together x86_tsc_frequency_khz and
> > set_x86_features, i.e. you'd either have the frequency return function
> > modify X86Features later, or always run x86_tsc_frequency_khz when
> > setting features (and that'd then require you to put the frequency
> > value somewhere, etc.)
>
> I was thinking the latter.

Just for archives sake, my earlier note here had a typo - I was
referencing the proposed PG_TSC_FREQUENCY_KNOWN here, not
PG_TSC_KNOWN_RELIABLE. But I think you got what I meant.

I'd suggest we leave this be for now (i.e. keep the X86Features more
closely tied to CPUID bits), as that could always be refactored later.

>
> > I've gone ahead and rewritten that whole paragraph for clarity, and
> > also split it into two. Feedback welcome:
> >
> > <para>
> >   If enabled, the TSC clock source will use specialized CPU instructions
> >   when measuring time intervals. This lowers timing overhead compared to
> >   reading the OS system clock, and reduces the measurement error on top
> >   of the actual runtime, for example with EXPLAIN ANALYZE.
> > </para>
>
>
> > <para>
> >   On x86-64 CPUs the TSC clock source utilizes the Time-Stamp Counter (TSC)
>
> It's a bit weird that the third use of TSC in these paragraphs introduces
> Time-Stamp Counter. I can see how you get there, but ...

Mhm. I had the same feeling when writing it that the ordering was a bit off.

I'll adjust for now to "TSC clock source, named after the Time-Stamp
Counter on x86-64" in the earlier sentence, and drop it in the later
one.

> Now I wonder if we should rename 'tsc' to 'cpu'...

Yeah, I had proposed making it more generic in an earlier email, but I
don't think there are great names available (and nobody jumped at the
suggestion). I think "cpu" is a bit too unspecific, vs "tsc" is more
clearly referencing the kind of instruction being used directly, and
is unique enough that it makes people read on vs jumping to a
conclusion. I was previously thinking of something like "hwtimer" or
"hwclock", but I don't think those are great.

I think the worst case here is that we need to add a "Note this is
named after the TSC instructions on x86-64, but utilizes the similarly
functioning cntvct_el0 instruction on ARM." or something to the
documentation if/when we expand support to ARM.

>
>
> >   of the CPU. The RDTSC instruction is used to read the TSC for EXPLAIN ANALYZE.
> >   For timings that require higher precision the RDTSCP instruction is used,
> >   which avoids inaccuracies due to CPU instruction re-ordering. Use of
> >   RDTSC/RDTSCP is not supported on older x86-64 CPUs or hypervisors that don't
> >   pass the TSC frequency to guest VMs, and is not advised on systems that
>
> s/guest VMs/virtual machines/?

Yup.

>
> >   utilize an emulated TSC. The TSC clock source is currently not supported on
> >   other architectures.
>
> The not support bit about hypervisors isn't quite right though? We do even use
> it automatically if TSC_ADJUST is set (and the calibration loop succeeds).

Good catch, that's no longer true with the calibration in the picture - removed.

>
>
> > </para>
> > <para>
> >   To help decide which clock source to use you can run the
> > <application>pg_test_timing</application>
> >   utility to check TSC availability, and perform timing measurements.
> > </para>
>
> How about a link to to the pg_test_timing page?  Hm, I guess that should also
> be updated with new output.

Right. I've added this in the 0004 commit. I also copied out the
pg_test_timings from a FreeBSD CI run and put them in as the example
output.

>
> I'd also sprinkle a few <acronym> and <command>s around.

Ack.

>
> Wonder if it's worth adding something like
>         <indexterm><primary><acronym>RDTSC</acronym></primary></indexterm>
>         <indexterm>
>          <primary>Time-Stamp Counter</primary>
>          <see><acronym>TSC</acronym></see>
>         </indexterm>
>         <indexterm><primary><acronym>TSC</acronym></primary></indexterm>
>
> otherwise somebody seeing one of these in logs, pg_test_timing output or
> whatever has even less of a chance to figure it out within our docs. They're
> not hard to search for terms exactly, so ...

Sure, that makes sense. If I followed the idea correctly, you just
wanted those added next to the timing_clock_source indexterm
definition, so I added it there.

>
> > I've also marked pg_get_ticks(_fast) as pg_attribute_always_inline,
> > per an off-list comment from Andres that he observed GCC not fully
> > inlining that function in pg_test_timing, presumably due to the
> > likely(..) in it.
>
> It's not the likely, I reproduced it even without that.  I mouthed off about
> compilers on mastodon and was kindly asked to just open a bug report :)
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=124795

:-)

Attached v22 with documentation updates. I've also marked the ARM
patch "nocfbot" for now, so its clear we're doing that one later.

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Zsolt Parragi <zsolt.parragi@percona.com> — 2026-04-07T07:32:43Z

> Its intentionally uint64, per this comment above it:
>
> * Note we utilize unsigned integers even though ticks are stored as a signed
> * value to encourage compilers to generate better assembly, since we can be
> * sure these values are not negative.
>
> In my earlier Compiler Explorer tests that did actually make a
> difference for the generated assembly.

Isn't that comment more about ticks_per_ns_scaled?

For max_ticks_no_overflow the only use is with a cast to int64, so I
didn't expect much assembly difference. Now I actually checked
locally/godbolt, and I don't see any actual differences. Making
max_ticks_no_overflow int64 and removing that cast generates exactly
the same code.

For ticks_per_ns_scaled, gcc 9-10 actually generates +1 mov
instruction with int64, but that's not present in more recent
versions.

Recent compiler versions only have an idiv/div and shr/sar difference.
Idiv is slower than div on intel, so that is a point for keeping
ticks_per_ns_scaled unsigned.

For arm I see the same lsr/asr and udiv/sdiv difference.

https://godbolt.org/z/4r5GTbrs3

(the main gcc vs clang difference seems to be clang's 32 bit division
optimization)

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-07T08:13:17Z

On Tue, Apr 7, 2026 at 12:32 AM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
>
> > Its intentionally uint64, per this comment above it:
> >
> > * Note we utilize unsigned integers even though ticks are stored as a signed
> > * value to encourage compilers to generate better assembly, since we can be
> > * sure these values are not negative.
> >
> > In my earlier Compiler Explorer tests that did actually make a
> > difference for the generated assembly.
>
> Isn't that comment more about ticks_per_ns_scaled?
>
> For max_ticks_no_overflow the only use is with a cast to int64, so I
> didn't expect much assembly difference. Now I actually checked
> locally/godbolt, and I don't see any actual differences. Making
> max_ticks_no_overflow int64 and removing that cast generates exactly
> the same code.
>
> For ticks_per_ns_scaled, gcc 9-10 actually generates +1 mov
> instruction with int64, but that's not present in more recent
> versions.
>
> Recent compiler versions only have an idiv/div and shr/sar difference.
> Idiv is slower than div on intel, so that is a point for keeping
> ticks_per_ns_scaled unsigned.
>
> For arm I see the same lsr/asr and udiv/sdiv difference.
>
> https://godbolt.org/z/4r5GTbrs3
>
> (the main gcc vs clang difference seems to be clang's 32 bit division
> optimization)

Thanks for re-checking, and I think you're correct in your assessment
that max_ticks_no_overflow could be signed. But I also don't think it
does any harm for it to be unsigned, since we know it will never be
negative, and we're correctly using PG_INT64_MAX when initializing it
(i.e. we use the max that's valid for ticks, which is int64).

I don't feel strongly about this. I'll let Andres make the call
whether its worth changing.

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-07T17:59:48Z

Hi,

On 2026-04-07 00:04:51 -0700, Lukas Fittl wrote:
> Attached v22 with documentation updates. I've also marked the ARM
> patch "nocfbot" for now, so its clear we're doing that one later.

- changed comment in InitProcessGlobals to be consistent with other cases
- added a comment to the fallthrough to the generic logic in x86_tsc_frequency_khz
- started to wonder if we ought to report
	 * The similar __get_cpuid_count function does not work as expected since
	 * it contains a check for __get_cpuid_max, which has been observed to be
  as a bug.  That's obviously independent of committing this.

- a few typos (s/possibly/possible/, s/its/it's/, s/\. are/. Are/,
  INSTR_TIME_ADD_NANOSEC comment variable names) found with modern spell
  checking tools (i.e. an ai review)

- fixed include order in guc_tables.c

- slight rephrasing of pg_initialize_timing() comment about
  pg_set_timing_clock_source()

- Moved the indexterm to the relevant paragraphs. I don't think it really
  matters, but it seemed a tad clearer

- shortened some commit message titles ;)

- Added a comment about why always_inline is needed for pg_get_ticks, and as
  part of that swapped _fast() and plain versions around.

- I'm somewhat tired, because I was wondering whether TSC_CALIBRATION_SKIPS
  should be a power of 2, to make the check faster. But uh, IT DOESN'T MATTER
  :)

I've pushed 0001, 0002, 0003.

There's one minor documentation issue in 0004 that I wanted to look at before
pushing (and I need to switch to something else for a bit).  The rephrasing
gets rid of

-   [...] , with the worst case somewhere between 32768 and
-   65535 nanoseconds.  In the second block, we can see that typical loop
-   time is 16 nanoseconds, and the readings appear to have full nanosecond
-   precision.

I don't mind loosing the first sentence, but the second one might be useful to
somebody?


> > Now I wonder if we should rename 'tsc' to 'cpu'...
> 
> Yeah, I had proposed making it more generic in an earlier email, but I
> don't think there are great names available (and nobody jumped at the
> suggestion). I think "cpu" is a bit too unspecific, vs "tsc" is more
> clearly referencing the kind of instruction being used directly, and
> is unique enough that it makes people read on vs jumping to a
> conclusion. I was previously thinking of something like "hwtimer" or
> "hwclock", but I don't think those are great.
> 
> I think the worst case here is that we need to add a "Note this is
> named after the TSC instructions on x86-64, but utilizes the similarly
> functioning cntvct_el0 instruction on ARM." or something to the
> documentation if/when we expand support to ARM.

Yea, I think it's ok.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-07T18:24:20Z

On Tue, Apr 7, 2026 at 10:59 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-04-07 00:04:51 -0700, Lukas Fittl wrote:
> > Attached v22 with documentation updates. I've also marked the ARM
> > patch "nocfbot" for now, so its clear we're doing that one later.
>
> - changed comment in InitProcessGlobals to be consistent with other cases
> - added a comment to the fallthrough to the generic logic in x86_tsc_frequency_khz
> - started to wonder if we ought to report
>          * The similar __get_cpuid_count function does not work as expected since
>          * it contains a check for __get_cpuid_max, which has been observed to be
>   as a bug.  That's obviously independent of committing this.
>
> - a few typos (s/possibly/possible/, s/its/it's/, s/\. are/. Are/,
>   INSTR_TIME_ADD_NANOSEC comment variable names) found with modern spell
>   checking tools (i.e. an ai review)
>
> - fixed include order in guc_tables.c
>
> - slight rephrasing of pg_initialize_timing() comment about
>   pg_set_timing_clock_source()
>
> - Moved the indexterm to the relevant paragraphs. I don't think it really
>   matters, but it seemed a tad clearer
>
> - shortened some commit message titles ;)
>
> - Added a comment about why always_inline is needed for pg_get_ticks, and as
>   part of that swapped _fast() and plain versions around.
>
> - I'm somewhat tired, because I was wondering whether TSC_CALIBRATION_SKIPS
>   should be a power of 2, to make the check faster. But uh, IT DOESN'T MATTER
>   :)
>
> I've pushed 0001, 0002, 0003.

Yay! Thank you for pushing :)

And thank you everyone on this thread for countless reviews, and to
David for writing some essential parts of this earlier.

*cautiously pours a celebratory beverage*

>
> There's one minor documentation issue in 0004 that I wanted to look at before
> pushing (and I need to switch to something else for a bit).  The rephrasing
> gets rid of
>
> -   [...] , with the worst case somewhere between 32768 and
> -   65535 nanoseconds.  In the second block, we can see that typical loop
> -   time is 16 nanoseconds, and the readings appear to have full nanosecond
> -   precision.
>
> I don't mind loosing the first sentence, but the second one might be useful to
> somebody?

Hm, yeah, you're right. What if we word like this:

  <para>
   The example results below show system clock timing where 99.99% of loops
   took between 16 and 63 nanoseconds.  In the second block, we can see that
   the typical loop time is 40 nanoseconds, and the readings appear to have
   full nanosecond precision.  Following the system clock results, the
   <acronym>TSC</acronym> clock source results are shown.  The
   <command>RDTSCP</command> instruction shows most loops completing in
   20&ndash;30 nanoseconds, while the <command>RDTSC</command> instruction
   is the fastest with an average loop time of 20 nanoseconds.  In this
   example the <acronym>TSC</acronym> clock source will be used by default,
   but can be disabled by setting <varname>timing_clock_source</varname> to
   <literal>system</literal>.
  </para>

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-07T21:18:53Z

Hi,

On 2026-04-07 11:24:20 -0700, Lukas Fittl wrote:
> > I've pushed 0001, 0002, 0003.
> 
> Yay! Thank you for pushing :)
> 
> And thank you everyone on this thread for countless reviews, and to
> David for writing some essential parts of this earlier.

Seconded!


> > There's one minor documentation issue in 0004 that I wanted to look at before
> > pushing (and I need to switch to something else for a bit).  The rephrasing
> > gets rid of
> >
> > -   [...] , with the worst case somewhere between 32768 and
> > -   65535 nanoseconds.  In the second block, we can see that typical loop
> > -   time is 16 nanoseconds, and the readings appear to have full nanosecond
> > -   precision.
> >
> > I don't mind loosing the first sentence, but the second one might be useful to
> > somebody?
> 
> Hm, yeah, you're right. What if we word like this:
> 
>   <para>
>    The example results below show system clock timing where 99.99% of loops
>    took between 16 and 63 nanoseconds.  In the second block, we can see that
>    the typical loop time is 40 nanoseconds, and the readings appear to have
>    full nanosecond precision.  Following the system clock results, the
>    <acronym>TSC</acronym> clock source results are shown.  The
>    <command>RDTSCP</command> instruction shows most loops completing in
>    20&ndash;30 nanoseconds, while the <command>RDTSC</command> instruction
>    is the fastest with an average loop time of 20 nanoseconds.  In this
>    example the <acronym>TSC</acronym> clock source will be used by default,
>    but can be disabled by setting <varname>timing_clock_source</varname> to
>    <literal>system</literal>.
>   </para>

Works.

Before pushing I vaccilated a bit about whether to replace the track_io_timing
reference in

+   On platforms that support the <acronym>TSC</acronym> clock source,
+   additional output sections are shown for the <command>RDTSCP</command>
+   instruction (used for general timing needs, such as
+   <varname>track_io_timing</varname>) and the <command>RDTSC</command>
+   instruction (used for <command>EXPLAIN ANALYZE</command>).  At the end

given it's one of the more likely cases to be converted to the fast
timestamping.  But in a decision I may live to regret, I deferred coming up
with a good way to phrase the difference between "per node timing" and
"overall query duration".

Pushed.

Yay!

Took only 6 years :)

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-08T01:26:04Z

Hi,

Some minor follow up things worth noting:

(1) I discussed with Andres off-list today if there would be a problem
with TSC instructions being disabled for user mode, since there is
technically a way to restrict them to kernel mode only. In my testing
this won't be a practical problem on Linux, since vDSO already
directly calls RDTSCP, and so if you disallow TSC through
"prctl(PR_SET_TSC, PR_TSC_SIGSEGV)" pg_test_timing does crash - but it
crashes in the clock_gettime call, i.e. would do so even without any
direct TSC use in the picture. Presumably you could change the Linux
clocksource to avoid that. I also did some testing with containers
that use seccomp profiles (e.g. Docker), and there are no issues that
I can found beyond what would have already been a problem before.

If this was a practical problem, we could consider moving the
"pg_initialize_timing_tsc" call in pg_set_timing_clock_source to be
inside the switch cases (like Andres suggested upthread, but did not
change in the commit to be clear), that way a user could start
Postgres with timing_clock_source=system and not have any RDTSC(P)
instructions run at all.

(2) The pg_test_timing documentation has an existing link to this wiki
page: https://wiki.postgresql.org/wiki/Pg_test_timing -- it seems that
page does talk about certain things that are not covered in the main
documentation, but I'm not really sure why that can't be in the main
docs. Do we want to keep maintaining that wiki page? (maybe we should
just remove that link, and move anything we consider critical to the
main docs?)

(3) I will move the patch to add ARM support to a new thread sometime
before the PG20 branch opens, so we can discuss that further.

With that, I've marked the commitfest entry as committed.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-08T06:09:23Z

Hi,

It looks like this recent failure on buildfarm member drongo might be
related to the timing changes - mainly suspecting it because the
commits are in the (slightly larger) set that changed, and the error
is in a timing related module that uses INSTR_TIME_SET_CURRENT.

>From https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2026-04-08%2001%3A57%3A00

# diff --strip-trailing-cr -U3
C:/prog/bf/root/HEAD/pgsql/contrib/tsm_system_time/expected/tsm_system_time.out
C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/results/tsm_system_time.out
# --- C:/prog/bf/root/HEAD/pgsql/contrib/tsm_system_time/expected/tsm_system_time.out
   2023-01-23 04:39:00.533642000 +0000
# +++ C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/results/tsm_system_time.out
   2026-04-08 04:03:15.248127800 +0000
# @@ -15,7 +15,7 @@
#  SELECT count(*) FROM test_tablesample TABLESAMPLE system_time (100000);
#   count
#  -------
# -    31
# +    16
#  (1 row)
#
#  -- bad parameters should get through planning, but not execution:
# 1 of 1 tests failed.
# The differences that caused some tests to fail can be viewed in the
file "C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/regression.diffs".
# A copy of the test summary that you see above is saved in the file
"C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/regression.out".

Haven't had a chance to dig through it yet, just noting it to start.


Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-08T06:33:52Z

On Tue, Apr 7, 2026 at 11:09 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> Hi,
>
> It looks like this recent failure on buildfarm member drongo might be
> related to the timing changes - mainly suspecting it because the
> commits are in the (slightly larger) set that changed, and the error
> is in a timing related module that uses INSTR_TIME_SET_CURRENT.
>
> From https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2026-04-08%2001%3A57%3A00
>
> # diff --strip-trailing-cr -U3
> C:/prog/bf/root/HEAD/pgsql/contrib/tsm_system_time/expected/tsm_system_time.out
> C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/results/tsm_system_time.out
> # --- C:/prog/bf/root/HEAD/pgsql/contrib/tsm_system_time/expected/tsm_system_time.out
>    2023-01-23 04:39:00.533642000 +0000
> # +++ C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/results/tsm_system_time.out
>    2026-04-08 04:03:15.248127800 +0000
> # @@ -15,7 +15,7 @@
> #  SELECT count(*) FROM test_tablesample TABLESAMPLE system_time (100000);
> #   count
> #  -------
> # -    31
> # +    16
> #  (1 row)
> #
> #  -- bad parameters should get through planning, but not execution:
> # 1 of 1 tests failed.
> # The differences that caused some tests to fail can be viewed in the
> file "C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/regression.diffs".
> # A copy of the test summary that you see above is saved in the file
> "C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/regression.out".
>
> Haven't had a chance to dig through it yet, just noting it to start.
>

I wonder a bit if the problem here could be that
INSTR_TIME_GET_MILLISEC got slightly more computationally expensive
with 0022622c93d9 (due to the logic in pg_ticks_to_ns), and that
module effectively does that in a tight loop. And if I understood
drongo's configuration correctly, it runs under valgrind. Attached a
quick idea how we could rework that to avoid it.

Thoughts?

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-08T15:13:14Z

Hi,

Andrew, what's drongo running on? I assume it's some kind of VM? What
virtualization technology?

On 2026-04-07 23:33:52 -0700, Lukas Fittl wrote:
> On Tue, Apr 7, 2026 at 11:09 PM Lukas Fittl <lukas@fittl.com> wrote:
> > It looks like this recent failure on buildfarm member drongo might be
> > related to the timing changes - mainly suspecting it because the
> > commits are in the (slightly larger) set that changed, and the error
> > is in a timing related module that uses INSTR_TIME_SET_CURRENT.
> >
> > From https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2026-04-08%2001%3A57%3A00
> >
> > # diff --strip-trailing-cr -U3
> > C:/prog/bf/root/HEAD/pgsql/contrib/tsm_system_time/expected/tsm_system_time.out
> > C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/results/tsm_system_time.out
> > # --- C:/prog/bf/root/HEAD/pgsql/contrib/tsm_system_time/expected/tsm_system_time.out
> >    2023-01-23 04:39:00.533642000 +0000
> > # +++ C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/results/tsm_system_time.out
> >    2026-04-08 04:03:15.248127800 +0000
> > # @@ -15,7 +15,7 @@
> > #  SELECT count(*) FROM test_tablesample TABLESAMPLE system_time (100000);
> > #   count
> > #  -------
> > # -    31
> > # +    16
> > #  (1 row)
> > #
> > #  -- bad parameters should get through planning, but not execution:
> > # 1 of 1 tests failed.
> > # The differences that caused some tests to fail can be viewed in the
> > file "C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/regression.diffs".
> > # A copy of the test summary that you see above is saved in the file
> > "C:/prog/bf/root/HEAD/pgsql.build/testrun/tsm_system_time/regress/regression.out".
> >
> > Haven't had a chance to dig through it yet, just noting it to start.
> >
> 
> I wonder a bit if the problem here could be that
> INSTR_TIME_GET_MILLISEC got slightly more computationally expensive
> with 0022622c93d9 (due to the logic in pg_ticks_to_ns), and that
> module effectively does that in a tight loop. And if I understood
> drongo's configuration correctly, it runs under valgrind.

It doesn't, that's just options for valgrind if it were enabled, that are in
the default config.

I think there must be something different going on.

SELECT count(*) FROM test_tablesample TABLESAMPLE system_time (100000);

> This table sampling method accepts a single floating-point argument that is
> the maximum number of milliseconds to spend reading the table. This gives
> you direct control over how long the query takes, at the price that the size
> of the sample becomes hard to predict. The resulting sample will contain as
> many rows as could be read in the specified time, unless the whole table has
> been read first.

So that's waiting for 100 seconds.

But the whole test only took 18.88s.  So something else than overhead is going
wrong here.

Oh.

I think it has a tsc clock source returning bogus results. Look at the
pg_test_timing output.

# System clock source: QueryPerformanceCounter
# Average loop time including overhead: 34.54 ns
...

# Clock source: RDTSCP
# Average loop time including overhead: 8179723.50 ns
...
# Fast clock source: RDTSC
# Average loop time including overhead: 4196799.05 ns
...

# TSC frequency in use: 7 kHz
# TSC frequency from calibration: 2500044 kHz
# TSC clock source will be used by default, unless timing_clock_source is set to 'system'.

Sooo, this system claims to have an invariant tsc but the frequency
we are getting from cpuid is completely out of whack.

Of course that could be for different reasons. It could be that we have a
portability issue around cpuids; we could calculate the frequency incorrectly;
the virtualization technology used might have configured wrong results...

I think we might need some sanity checking of the timing results in
pg_test_timing, so that we can pick up this kind of craziness directly in
the tests for pg_test_timing, rather than indirectly like here.

We probably should do some basic range checking in the cpuid based frequency
too, clearly 7khz can never be sane.

But I don't want to add that before we have figured out why we're seeing the
frequency, if it's e.g. that something in the cpuid infrastructure (cpuidex
not working right), or is the vmware logic wrong, ...

Maybe we should add a char **source_details argument to
pg_tsc_calibrate_frequency that pg_test_timing can report?

I wonder if we also should add a pg_timing_clock_source_info() function that
returns frequency_khz, calibrated_frequency_khz, frequency_source_info or
such?

> Attached a quick idea how we could rework that to avoid it.
> 
> Thoughts?

Maybe maybe it's worth doing that for 20, but I don't think it's related to
the problem at hand.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-08T19:25:54Z

On Wed, Apr 8, 2026 at 8:13 AM Andres Freund <andres@anarazel.de> wrote:
>
> So that's waiting for 100 seconds.
>
> But the whole test only took 18.88s.  So something else than overhead is going
> wrong here.
>
> Oh.
>
> I think it has a tsc clock source returning bogus results. Look at the
> pg_test_timing output.
>
>
> # System clock source: QueryPerformanceCounter
> # Average loop time including overhead: 34.54 ns
> ...
>
> # Clock source: RDTSCP
> # Average loop time including overhead: 8179723.50 ns
> ...
> # Fast clock source: RDTSC
> # Average loop time including overhead: 4196799.05 ns
> ...
>
> # TSC frequency in use: 7 kHz
> # TSC frequency from calibration: 2500044 kHz
> # TSC clock source will be used by default, unless timing_clock_source is set to 'system'.
>
> Sooo, this system claims to have an invariant tsc but the frequency
> we are getting from cpuid is completely out of whack.

Huh. Yeah, I think this is a case of getting a bad TSC frequency from CPUID.

>
> Of course that could be for different reasons. It could be that we have a
> portability issue around cpuids; we could calculate the frequency incorrectly;
> the virtualization technology used might have configured wrong results...
>
>
> I think we might need some sanity checking of the timing results in
> pg_test_timing, so that we can pick up this kind of craziness directly in
> the tests for pg_test_timing, rather than indirectly like here.
>
>
> We probably should do some basic range checking in the cpuid based frequency
> too, clearly 7khz can never be sane.
>
> But I don't want to add that before we have figured out why we're seeing the
> frequency, if it's e.g. that something in the cpuid infrastructure (cpuidex
> not working right), or is the vmware logic wrong, ...

Agreed.

I half wonder if this could be a case of a Hypervisor, but not KVM or
VMware, and so we fall through to the regular CPUID information (which
AFAIR is different from how Linux itself handles that case, where
it'll always do calibration in such cases). I think the solution might
be to use the TSC calibration always on other hypervisors.

But lets wait for Andrew to confirm the configuration of the machine /
have a run with the additional information.

>
>
> Maybe we should add a char **source_details argument to
> pg_tsc_calibrate_frequency that pg_test_timing can report?
>
> I wonder if we also should add a pg_timing_clock_source_info() function that
> returns frequency_khz, calibrated_frequency_khz, frequency_source_info or
> such?

See attached a patch that adds that and shows its output in
pg_test_timing. Here is an example from an AWS instance:

TSC frequency in use: 2899943 kHz
TSC frequency source: x86, hypervisor (kvm), cpuid 0x40000010
TSC frequency from calibration: 2899063 kHz
TSC clock source will be used by default, unless timing_clock_source
is set to 'system'.

And from Azure (HyperV):

TSC frequency in use: 2791936 kHz
TSC frequency source: x86, calibration
TSC frequency from calibration: 2793379 kHz
TSC clock source will be used by default, unless timing_clock_source
is set to 'system'.

Note it doesn't emit the fact that its a hypervisor if calibration was
used to keep the code a bit simpler, but would emit it as "hypervisor
(other)" or "hypervisor (unknown)" (if cpuidex wasn't available) if
cpuid 0x15/0x16 get used.

>
> > Attached a quick idea how we could rework that to avoid it.
> >
> > Thoughts?
>
> Maybe maybe it's worth doing that for 20, but I don't think it's related to
> the problem at hand.

Ack, agreed this is unrelated to the issue.

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-08T19:44:02Z

Hi,

On 2026-04-08 12:25:54 -0700, Lukas Fittl wrote:
> > # TSC frequency in use: 7 kHz
> > # TSC frequency from calibration: 2500044 kHz
> > # TSC clock source will be used by default, unless timing_clock_source is set to 'system'.
> >
> > Sooo, this system claims to have an invariant tsc but the frequency
> > we are getting from cpuid is completely out of whack.
> 
> Huh. Yeah, I think this is a case of getting a bad TSC frequency from CPUID.

Yea.  Not sure yet why...


> I half wonder if this could be a case of a Hypervisor, but not KVM or
> VMware, and so we fall through to the regular CPUID information (which
> AFAIR is different from how Linux itself handles that case, where
> it'll always do calibration in such cases). I think the solution might
> be to use the TSC calibration always on other hypervisors.

Plausible.


> But lets wait for Andrew to confirm the configuration of the machine /
> have a run with the additional information.

Yep.


> > I wonder if we also should add a pg_timing_clock_source_info() function that
> > returns frequency_khz, calibrated_frequency_khz, frequency_source_info or
> > such?
> 
> See attached a patch that adds that and shows its output in
> pg_test_timing. Here is an example from an AWS instance:
> 
> TSC frequency in use: 2899943 kHz
> TSC frequency source: x86, hypervisor (kvm), cpuid 0x40000010
> TSC frequency from calibration: 2899063 kHz
> TSC clock source will be used by default, unless timing_clock_source
> is set to 'system'.
> 
> And from Azure (HyperV):
> 
> TSC frequency in use: 2791936 kHz
> TSC frequency source: x86, calibration
> TSC frequency from calibration: 2793379 kHz
> TSC clock source will be used by default, unless timing_clock_source
> is set to 'system'.

Nice.


> Note it doesn't emit the fact that its a hypervisor if calibration was
> used to keep the code a bit simpler, but would emit it as "hypervisor
> (other)" or "hypervisor (unknown)" (if cpuidex wasn't available) if
> cpuid 0x15/0x16 get used.

That seems ok for now, I think.


What do you think about making pg_test_timing warn and return 1 if there is a
tsc clocksource but the calibrated frequency differs by more than, idk, 10%?
I'm worried that there might be other problems like this lurking and we
wouldn't know about them unless the issue is of a similar magnitude.

>  
>  #if PG_INSTR_TSC_CLOCK
> +static const char *timing_tsc_frequency_source = NULL;

Think this ought to document what the lifetime of the memory is.


>  /*
>   * Initialize the TSC clock source by determining its usability and frequency.
> @@ -202,13 +205,14 @@ static void
>  tsc_detect_frequency(void)
>  {
>  	timing_tsc_frequency_khz = 0;
> +	timing_tsc_frequency_source = NULL;
>  
>  	/* We require RDTSCP support and an invariant TSC, bail if not available */
>  	if (!x86_feature_available(PG_RDTSCP) || !x86_feature_available(PG_TSC_INVARIANT))
>  		return;
>  
>  	/* Determine speed at which the TSC advances */
> -	timing_tsc_frequency_khz = x86_tsc_frequency_khz();
> +	timing_tsc_frequency_khz = x86_tsc_frequency_khz(&timing_tsc_frequency_source);
>  	if (timing_tsc_frequency_khz > 0)
>  		return;

So, if we were called again, we'd leak the old content of the memory.

But what I think is more problematic is that x86_tsc_frequency_khz() allocates
memory with palloc(), via psprintf(), but there's no guarantee that it's
allocated in a sufficiently long lived allocation.

timing_tsc_frequency_source were a char[128] and you passed the string and
length to x86_tsc_frequency_khz and then used snprintf().  That would also
make it a bit easier to iteratively populate the string.



> +/*
> + * Returns TSC clock source information for diagnostic purposes.
> + *
> + * Note: This always runs the TSC calibration loop which may take up to
> + * TSC_CALIBRATION_MAX_NS.
> + */
> +TscClockSourceInfo
> +pg_timing_tsc_clock_source_info(void)
> +{
> +	TscClockSourceInfo info;
> +
> +	info.frequency_khz = timing_tsc_frequency_khz;
> +	info.frequency_source = timing_tsc_frequency_source;
> +	info.calibrated_frequency_khz = pg_tsc_calibrate_frequency();

Seems we should also store the calibration somewhere and only run it if it
hadn't already done?

What if we just made this TscClockSourceInfo a struct in the file, populated
it during normal initialization, and then returned it as a const
TscClockSourceInfo *from pg_timing_tsc_clock_source_info?

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-09T04:36:48Z

On Wed, Apr 8, 2026 at 12:44 PM Andres Freund <andres@anarazel.de> wrote:
>
> > I half wonder if this could be a case of a Hypervisor, but not KVM or
> > VMware, and so we fall through to the regular CPUID information (which
> > AFAIR is different from how Linux itself handles that case, where
> > it'll always do calibration in such cases). I think the solution might
> > be to use the TSC calibration always on other hypervisors.
>
> Plausible.
>

And that is indeed the problem. Looks like we can't trust CPUID
0x15/0x16 when we're under a Hypervisor and its not KVM or VMware.

I was able to reproduce this on an m5.xlarge instance on AWS with
Windows Server 2019 installed, building with MSVC 2019:

TSC frequency in use: 7 kHz
TSC frequency source: x86, hypervisor (other), cpuid 0x15
TSC frequency from calibration: 2499879 kHz
TSC clock source will be used by default, unless timing_clock_source
is set to 'system'.

And when we return after we realize the Hypervisor can't give us the
frequency and use calibration instead:

TSC frequency in use: 2500357 kHz
TSC frequency source: x86, calibration
TSC frequency from calibration: 2499793 kHz
TSC clock source will be used by default, unless timing_clock_source
is set to 'system'.

It appears the problem is related to however AWS implements the
"nitro" virtualization layer for Windows. Interestingly enough, a
m5.xlarge instance with Linux works just fine, and shows a KVM
hypervisor. For Linux, it gets the correct information from the
special CPUID register for KVM/VMware, which matches the calibrated
TSC frequency shown under Windows.

FWIW, earlier deprecated EC2 instance types that used Xen (e.g.
m4.xlarge) just report "TSC unusable" on Windows, presumably because
its not an invariant TSC, but I haven't dug into it since it seems
fine to automatically use the system clock source in that case.

> > > I wonder if we also should add a pg_timing_clock_source_info() function that
> > > returns frequency_khz, calibrated_frequency_khz, frequency_source_info or
> > > such?
> >
> > See attached a patch that adds that and shows its output in
> > pg_test_timing. Here is an example from an AWS instance:
> >
> > TSC frequency in use: 2899943 kHz
> > TSC frequency source: x86, hypervisor (kvm), cpuid 0x40000010
> > TSC frequency from calibration: 2899063 kHz
> > TSC clock source will be used by default, unless timing_clock_source
> > is set to 'system'.
> >
> > And from Azure (HyperV):
> >
> > TSC frequency in use: 2791936 kHz
> > TSC frequency source: x86, calibration
> > TSC frequency from calibration: 2793379 kHz
> > TSC clock source will be used by default, unless timing_clock_source
> > is set to 'system'.
>
> Nice.
>

FWIW, I also ran tests today on VMware (AMD CPU):

TSC frequency in use: 2994329 kHz
TSC frequency source: x86, hypervisor (vmware), cpuid 0x40000010
TSC frequency from calibration: 2994008 kHz
TSC clock source will be used by default, unless timing_clock_source
is set to 'system'.

And Virtualbox (Intel CPU, thanks Maciek):

TSC frequency in use: 2989463 kHz
TSC frequency source: x86, calibration
TSC frequency from calibration: 2989619 kHz
TSC clock source will be used by default, unless timing_clock_source
is set to 'system'.

This was without the early return, i.e. Virtualbox doesn't pass
through CPUID even if the host has it on Intel CPUs.

> What do you think about making pg_test_timing warn and return 1 if there is a
> tsc clocksource but the calibrated frequency differs by more than, idk, 10%?
> I'm worried that there might be other problems like this lurking and we
> wouldn't know about them unless the issue is of a similar magnitude.

Yeah, that seems like a good idea. If I understand correctly you're
thinking we could tell the user to switch to
timing_clock_source=system in that case? (i.e. this is only a
pg_test_timing notice, not something "smarter" in the backend itself)

> >
> >  #if PG_INSTR_TSC_CLOCK
> > +static const char *timing_tsc_frequency_source = NULL;
>
> Think this ought to document what the lifetime of the memory is.

Per your point below, I agree it seems best to just use a fixed buffer.

> >  /*
> >   * Initialize the TSC clock source by determining its usability and frequency.
> > @@ -202,13 +205,14 @@ static void
> >  tsc_detect_frequency(void)
> >  {
> >       timing_tsc_frequency_khz = 0;
> > +     timing_tsc_frequency_source = NULL;
> >
> >       /* We require RDTSCP support and an invariant TSC, bail if not available */
> >       if (!x86_feature_available(PG_RDTSCP) || !x86_feature_available(PG_TSC_INVARIANT))
> >               return;
> >
> >       /* Determine speed at which the TSC advances */
> > -     timing_tsc_frequency_khz = x86_tsc_frequency_khz();
> > +     timing_tsc_frequency_khz = x86_tsc_frequency_khz(&timing_tsc_frequency_source);
> >       if (timing_tsc_frequency_khz > 0)
> >               return;
>
> So, if we were called again, we'd leak the old content of the memory.
>
> But what I think is more problematic is that x86_tsc_frequency_khz() allocates
> memory with palloc(), via psprintf(), but there's no guarantee that it's
> allocated in a sufficiently long lived allocation.
>
> timing_tsc_frequency_source were a char[128] and you passed the string and
> length to x86_tsc_frequency_khz and then used snprintf().  That would also
> make it a bit easier to iteratively populate the string.

Ack, good point.

> > +/*
> > + * Returns TSC clock source information for diagnostic purposes.
> > + *
> > + * Note: This always runs the TSC calibration loop which may take up to
> > + * TSC_CALIBRATION_MAX_NS.
> > + */
> > +TscClockSourceInfo
> > +pg_timing_tsc_clock_source_info(void)
> > +{
> > +     TscClockSourceInfo info;
> > +
> > +     info.frequency_khz = timing_tsc_frequency_khz;
> > +     info.frequency_source = timing_tsc_frequency_source;
> > +     info.calibrated_frequency_khz = pg_tsc_calibrate_frequency();
>
> Seems we should also store the calibration somewhere and only run it if it
> hadn't already done?
>
> What if we just made this TscClockSourceInfo a struct in the file, populated
> it during normal initialization, and then returned it as a const
> TscClockSourceInfo *from pg_timing_tsc_clock_source_info?

Yeah, that seems reasonable.

Attached 0001 fixes the issue for me on my test instance, and
presumably will fix drongo as well.

0002 is the updated version of emitting the additional debug info. I
think this is certainly less critical to have in 19 now, but could
still be useful if there are any future oddities.

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andrew Dunstan <andrew@dunslane.net> — 2026-04-09T09:27:36Z

On 2026-04-08 We 11:13 AM, Andres Freund wrote:
> Hi,
>
> Andrew, what's drongo running on? I assume it's some kind of VM? What
> virtualization technology?


It's an AWS EC2 instance.

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-09T16:02:28Z

Hi,

On 2026-04-08 21:36:48 -0700, Lukas Fittl wrote:
> On Wed, Apr 8, 2026 at 12:44 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > > I half wonder if this could be a case of a Hypervisor, but not KVM or
> > > VMware, and so we fall through to the regular CPUID information (which
> > > AFAIR is different from how Linux itself handles that case, where
> > > it'll always do calibration in such cases). I think the solution might
> > > be to use the TSC calibration always on other hypervisors.
> >
> > Plausible.
> >
> 
> And that is indeed the problem. Looks like we can't trust CPUID
> 0x15/0x16 when we're under a Hypervisor and its not KVM or VMware.

Why you'd report a TSC frequency but populate it with a distinct frequency
from the actual tsc is beyond me, but oh well, we gotta deal.

Pushed the fix.


> FWIW, earlier deprecated EC2 instance types that used Xen (e.g.
> m4.xlarge) just report "TSC unusable" on Windows, presumably because
> its not an invariant TSC, but I haven't dug into it since it seems
> fine to automatically use the system clock source in that case.

Yea, that's not worth investigating.


> This was without the early return, i.e. Virtualbox doesn't pass
> through CPUID even if the host has it on Intel CPUs.

Seems good.


> > What do you think about making pg_test_timing warn and return 1 if there is a
> > tsc clocksource but the calibrated frequency differs by more than, idk, 10%?
> > I'm worried that there might be other problems like this lurking and we
> > wouldn't know about them unless the issue is of a similar magnitude.
> 
> Yeah, that seems like a good idea. If I understand correctly you're
> thinking we could tell the user to switch to
> timing_clock_source=system in that case? (i.e. this is only a
> pg_test_timing notice, not something "smarter" in the backend itself)

I'd even just say "investigate your system an/or report a bug to postgres" :)


> Attached 0001 fixes the issue for me on my test instance, and
> presumably will fix drongo as well.
> 
> 0002 is the updated version of emitting the additional debug info. I
> think this is certainly less critical to have in 19 now, but could
> still be useful if there are any future oddities.

I think we should do something, probably together with the test enhancement I
described, because otherwise we won't actually find potential breakage before
it hits production environments.


> @@ -161,10 +165,13 @@ static uint32 x86_hypervisor_tsc_frequency_khz(void);
>   * 0 indicates the frequency information was not accessible via CPUID.
>   */
>  uint32
> -x86_tsc_frequency_khz(void)
> +x86_tsc_frequency_khz(char *source, size_t source_len)
>  {
>  	unsigned int reg[4] = {0};
>  
> +	if (source)
> +		strlcpy(source, "x86", source_len);
> +
>  	/*
>  	 * If we're inside a virtual machine, try to fetch the TSC frequency from
>  	 * the Hypervisor itself using specialized CPUID registers.
> @@ -173,7 +180,11 @@ x86_tsc_frequency_khz(void)
>  	 * a virtual machine, as it has been observed to be wildly incorrect.
>  	 */
>  	if (x86_feature_available(PG_HYPERVISOR))
> +	{
> +		if (source)
> +			strlcat(source, ", hypervisor, cpuid 0x40000010", source_len);
>  		return x86_hypervisor_tsc_frequency_khz();
> +	}
>  
>  	/*
>  	 * On modern Intel CPUs, the TSC is implemented by invariant
>  	timekeeping

Any reason you didn't include the hypervisor like in the prior version? Just
simplicity?

I think this actually ends up getting overwritten if
x86_hypervisor_tsc_frequency_khz() then "fails" to detect a frequency.  Feels
like it'd be good to continue reporting that it's in a hypervisor, because
hypervisors can set tsc frequency multipliers and stuff.



What do you think about the attached incremental patch?

If I e.g. intentionally force the hypervisor path being taken, on a non-VM, I
get:
    TSC frequency source: x86, hypervisor, cpuid 0x40000010, calibration
    TSC frequency in use: 2497902 kHz
    TSC frequency from calibration: 2497902 kHz
    TSC clock source will be used by default, unless timing_clock_source is set to 'system'.

And if rdtscp is not available:
    TSC frequency source: x86, no rdtscp
    TSC frequency in use: 0 kHz
    TSC frequency from calibration: 2500040 kHz
    TSC clock source is not usable. Likely unable to determine TSC frequency. Are you running in an unsupported virtualized environment?

It's not perfect, but seems like it might be good enough?


Note to future self: Need to consider update the sgml docs example. Probably
just fudge it, to avoid having to update the numbers too.



Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-10T07:12:00Z

On Thu, Apr 9, 2026 at 9:02 AM Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-04-08 21:36:48 -0700, Lukas Fittl wrote:
> >
> > And that is indeed the problem. Looks like we can't trust CPUID
> > 0x15/0x16 when we're under a Hypervisor and its not KVM or VMware.
>
> Why you'd report a TSC frequency but populate it with a distinct frequency
> from the actual tsc is beyond me, but oh well, we gotta deal.
>
> Pushed the fix.

Thanks!

> > > What do you think about making pg_test_timing warn and return 1 if there is a
> > > tsc clocksource but the calibrated frequency differs by more than, idk, 10%?
> > > I'm worried that there might be other problems like this lurking and we
> > > wouldn't know about them unless the issue is of a similar magnitude.
> >
> > Yeah, that seems like a good idea. If I understand correctly you're
> > thinking we could tell the user to switch to
> > timing_clock_source=system in that case? (i.e. this is only a
> > pg_test_timing notice, not something "smarter" in the backend itself)
>
> I'd even just say "investigate your system an/or report a bug to postgres" :)
>

Sure, seems reasonable. I went ahead and added that in the attached
v27 (squashed with your other change).

Example how that looks like (tested without the fix in place):

---

TSC frequency source: x86, hypervisor, cpuid 0x15
TSC frequency in use: 7 kHz
TSC frequency from calibration: 2500260 kHz
WARNING: Calibrated TSC frequency differs by 35717900.0% from the TSC
frequency in use
HINT: Consider setting timing_clock_source to 'system'. Report bugs to
<pgsql-bugs@lists.postgresql.org>.

TSC clock source will be used by default, unless timing_clock_source
is set to 'system'.

---

I also added the extra newline before the "will be used by default"
message, because I felt its too much information bunched together
otherwise.

> > Attached 0001 fixes the issue for me on my test instance, and
> > presumably will fix drongo as well.
> >
> > 0002 is the updated version of emitting the additional debug info. I
> > think this is certainly less critical to have in 19 now, but could
> > still be useful if there are any future oddities.
>
> I think we should do something, probably together with the test enhancement I
> described, because otherwise we won't actually find potential breakage before
> it hits production environments.

Ack, makes sense to me.

> Any reason you didn't include the hypervisor like in the prior version? Just
> simplicity?
>
> I think this actually ends up getting overwritten if
> x86_hypervisor_tsc_frequency_khz() then "fails" to detect a frequency.  Feels
> like it'd be good to continue reporting that it's in a hypervisor, because
> hypervisors can set tsc frequency multipliers and stuff.
>

Agreed that seems reasonable.

>
> What do you think about the attached incremental patch?
>
> If I e.g. intentionally force the hypervisor path being taken, on a non-VM, I
> get:
>     TSC frequency source: x86, hypervisor, cpuid 0x40000010, calibration
>     TSC frequency in use: 2497902 kHz
>     TSC frequency from calibration: 2497902 kHz
>     TSC clock source will be used by default, unless timing_clock_source is set to 'system'.
>
> And if rdtscp is not available:
>     TSC frequency source: x86, no rdtscp
>     TSC frequency in use: 0 kHz
>     TSC frequency from calibration: 2500040 kHz
>     TSC clock source is not usable. Likely unable to determine TSC frequency. Are you running in an unsupported virtualized environment?
>
> It's not perfect, but seems like it might be good enough?

Yeah, I think that looks good. On an m4.xlarge instance (Linux / xen)
with its very slow clock I get the following:

---

System clock source: clock_gettime (CLOCK_MONOTONIC)
Average loop time including overhead: 570.09 ns
Histogram of timing durations:
...
TSC frequency source: x86, not invariant
TSC frequency in use: 0 kHz
TSC frequency from calibration: 2299714 kHz

TSC clock source is not usable. Likely unable to determine TSC
frequency. Are you running in an unsupported virtualized environment?

---

FWIW, Linux has current_clocksource "xen" instead of "tsc" on that instance.

I assume we're okay with not reporting "hypervisor" in the source
string in the early failure case? If we wanted to, it'd make the diff
a bit larger since we'd need an extra hypervisor feature check.

> Note to future self: Need to consider update the sgml docs example. Probably
> just fudge it, to avoid having to update the numbers too.

Yeah, I wouldn't update the numbers in the docs. I've added an example
of the new output in the attached.

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-10T07:28:56Z

On Fri, Apr 10, 2026 at 12:12 AM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Thu, Apr 9, 2026 at 9:02 AM Andres Freund <andres@anarazel.de> wrote:
> >
> > On 2026-04-08 21:36:48 -0700, Lukas Fittl wrote:
> > >
> > > > What do you think about making pg_test_timing warn and return 1 if there is a
> > > > tsc clocksource but the calibrated frequency differs by more than, idk, 10%?
> > > > I'm worried that there might be other problems like this lurking and we
> > > > wouldn't know about them unless the issue is of a similar magnitude.
> > >
> > > Yeah, that seems like a good idea. If I understand correctly you're
> > > thinking we could tell the user to switch to
> > > timing_clock_source=system in that case? (i.e. this is only a
> > > pg_test_timing notice, not something "smarter" in the backend itself)
> >
> > I'd even just say "investigate your system an/or report a bug to postgres" :)
> >
>
> Sure, seems reasonable. I went ahead and added that in the attached
> v27 (squashed with your other change).
>
> Example how that looks like (tested without the fix in place):
>
> ---
>
> TSC frequency source: x86, hypervisor, cpuid 0x15
> TSC frequency in use: 7 kHz
> TSC frequency from calibration: 2500260 kHz
> WARNING: Calibrated TSC frequency differs by 35717900.0% from the TSC
> frequency in use
> HINT: Consider setting timing_clock_source to 'system'. Report bugs to
> <pgsql-bugs@lists.postgresql.org>.
>
> TSC clock source will be used by default, unless timing_clock_source
> is set to 'system'.
>
> ---
>
> I also added the extra newline before the "will be used by default"
> message, because I felt its too much information bunched together
> otherwise.

I just realized that you initially suggested to do a "return 1", but I
did not include that in the version I sent. I think we could easily do
an "exit(1)" after that warning is printed - seems sensible to get
CI/buildfarm to fail clearly.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Haibo Yan <tristan.yim@gmail.com> — 2026-04-12T06:04:42Z

On Fri, Apr 10, 2026 at 12:29 AM Lukas Fittl <lukas@fittl.com> wrote:

> On Fri, Apr 10, 2026 at 12:12 AM Lukas Fittl <lukas@fittl.com> wrote:
> >
> > On Thu, Apr 9, 2026 at 9:02 AM Andres Freund <andres@anarazel.de> wrote:
> > >
> > > On 2026-04-08 21:36:48 -0700, Lukas Fittl wrote:
> > > >
> > > > > What do you think about making pg_test_timing warn and return 1 if
> there is a
> > > > > tsc clocksource but the calibrated frequency differs by more than,
> idk, 10%?
> > > > > I'm worried that there might be other problems like this lurking
> and we
> > > > > wouldn't know about them unless the issue is of a similar
> magnitude.
> > > >
> > > > Yeah, that seems like a good idea. If I understand correctly you're
> > > > thinking we could tell the user to switch to
> > > > timing_clock_source=system in that case? (i.e. this is only a
> > > > pg_test_timing notice, not something "smarter" in the backend itself)
> > >
> > > I'd even just say "investigate your system an/or report a bug to
> postgres" :)
> > >
> >
> > Sure, seems reasonable. I went ahead and added that in the attached
> > v27 (squashed with your other change).
> >
> > Example how that looks like (tested without the fix in place):
> >
> > ---
> >
> > TSC frequency source: x86, hypervisor, cpuid 0x15
> > TSC frequency in use: 7 kHz
> > TSC frequency from calibration: 2500260 kHz
> > WARNING: Calibrated TSC frequency differs by 35717900.0% from the TSC
> > frequency in use
> > HINT: Consider setting timing_clock_source to 'system'. Report bugs to
> > <pgsql-bugs@lists.postgresql.org>.
> >
> > TSC clock source will be used by default, unless timing_clock_source
> > is set to 'system'.
> >
> > ---
> >
> > I also added the extra newline before the "will be used by default"
> > message, because I felt its too much information bunched together
> > otherwise.
>
> I just realized that you initially suggested to do a "return 1", but I
> did not include that in the version I sent. I think we could easily do
> an "exit(1)" after that warning is printed - seems sensible to get
> CI/buildfarm to fail clearly.
>
> Thanks,
> Lukas
>
> --
> Lukas Fittl
>
>
>
Hi Lukas,
Thanks for the patch. I may be missing something, but I wonder if the new
debug path can still end up calling pg_tsc_calibrate_frequency() after
tsc_detect_frequency() already bailed out with no rdtscp or not invariant.
If so, it seems the diagnostics path is no longer following the same gate
as the normal TSC-usability path, and could still execute pg_rdtscp() while
only trying to print debug info.
Maybe I am overlooking a guard somewhere, but I think it would be safer
either to use the same prerequisites here, or keep this helper purely
passive.
Best,
Haibo

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-12T16:38:15Z

Hi Haibo,

On Sat, Apr 11, 2026 at 11:04 PM Haibo Yan <tristan.yim@gmail.com> wrote:
>
> Thanks for the patch. I may be missing something, but I wonder if the new debug path can still end up calling pg_tsc_calibrate_frequency() after tsc_detect_frequency() already bailed out with no rdtscp or not invariant.
> If so, it seems the diagnostics path is no longer following the same gate as the normal TSC-usability path, and could still execute pg_rdtscp() while only trying to print debug info.
> Maybe I am overlooking a guard somewhere, but I think it would be safer either to use the same prerequisites here, or keep this helper purely passive.

Yes, that's a good point. I don't think the TSC not being invariant
would be a problem on its own, but the RDTSCP instruction not being
available would presumably make the frequency function fail.

I think the easiest way to approach that is to only run the
calibration function in the debug path when we have a frequency, but
it wasn't determined through calibration. Per earlier note from Andres
running calibration in those cases is intentional to help us compare
CPUID data with what we got from calibration.

Attached v28, which also adds the exit(1) mentioned earlier.

FWIW, for archive's sake, drongo is green again now, thanks to commit
7fc36c5db550 (Avoid CPUID 0x15/0x16 for Hypervisor TSC frequency).

Thanks,
Lukas

--
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-12T18:20:31Z

Lukas Fittl <lukas@fittl.com> writes:
> FWIW, for archive's sake, drongo is green again now, thanks to commit
> 7fc36c5db550 (Avoid CPUID 0x15/0x16 for Hypervisor TSC frequency).

drongo may be happy, but Coverity is not:

166     	uint64		loop_count;
167     
168     	loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_SYSTEM, false);
>>>     CID 1691465:         Incorrect expression  (DIVIDE_BY_ZERO)
>>>     In function call "output", division by expression "loop_count" which may be zero has undefined behavior.
169     	output(loop_count);

AFAICS it's correct to complain.  test_timing() visibly can return zero,
but of the three places where test_timing() is followed by output()
only one has a defense against that.

			regards, tom lane

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-04-12T18:30:39Z

Hi,

On 2026-04-12 14:20:31 -0400, Tom Lane wrote:
> Lukas Fittl <lukas@fittl.com> writes:
> > FWIW, for archive's sake, drongo is green again now, thanks to commit
> > 7fc36c5db550 (Avoid CPUID 0x15/0x16 for Hypervisor TSC frequency).
>
> drongo may be happy, but Coverity is not:
>
> 166     	uint64		loop_count;
> 167
> 168     	loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_SYSTEM, false);
> >>>     CID 1691465:         Incorrect expression  (DIVIDE_BY_ZERO)
> >>>     In function call "output", division by expression "loop_count" which may be zero has undefined behavior.
> 169     	output(loop_count);
>
> AFAICS it's correct to complain.  test_timing() visibly can return zero,
> but of the three places where test_timing() is followed by output()
> only one has a defense against that.

I think it should be unreachable as-is (but we should fix it anyway): If the
system clock source doesn't work, we have much bigger issues. If rdtscp works,
rdtsc should better work as well...

Maybe it's enough to add an Assert() to clarify this?  But I guess just
printing a message in that unreachable case would also work.

Greetings,

Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-12T19:58:36Z

On Sun, Apr 12, 2026 at 11:30 AM Andres Freund <andres@anarazel.de> wrote:
>
> On 2026-04-12 14:20:31 -0400, Tom Lane wrote:
> > 166           uint64          loop_count;
> > 167
> > 168           loop_count = test_timing(test_duration, TIMING_CLOCK_SOURCE_SYSTEM, false);
> > >>>     CID 1691465:         Incorrect expression  (DIVIDE_BY_ZERO)
> > >>>     In function call "output", division by expression "loop_count" which may be zero has undefined behavior.
> > 169           output(loop_count);
> >
> > AFAICS it's correct to complain.  test_timing() visibly can return zero,
> > but of the three places where test_timing() is followed by output()
> > only one has a defense against that.
>
> I think it should be unreachable as-is (but we should fix it anyway): If the
> system clock source doesn't work, we have much bigger issues. If rdtscp works,
> rdtsc should better work as well...

Yeah, agreed, in practice this won't be reached, but good to fix.

> Maybe it's enough to add an Assert() to clarify this?  But I guess just
> printing a message in that unreachable case would also work.

I think either is fine. If we did it with a message, how about this at
the beginning of the output function?

if (loop_count == 0)
{
  printf(_("WARNING: No timing measurements collected. Report this as
a bug to <%s>.\n"), PACKAGE_BUGREPORT);
  return;
}

That'd align with the proposed patch to warn about calibrated TSC
frequency diverging more than 10%.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-12T20:24:42Z

Lukas Fittl <lukas@fittl.com> writes:
> I think either is fine. If we did it with a message, how about this at
> the beginning of the output function?

> if (loop_count == 0)
> {
>   printf(_("WARNING: No timing measurements collected. Report this as a bug to <%s>.\n"), PACKAGE_BUGREPORT);
>   return;
> }

WFM.

			regards, tom lane

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Lukas Fittl <lukas@fittl.com> — 2026-04-24T06:16:07Z

On Sun, Apr 12, 2026 at 1:24 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Lukas Fittl <lukas@fittl.com> writes:
> > I think either is fine. If we did it with a message, how about this at
> > the beginning of the output function?
>
> > if (loop_count == 0)
> > {
> >   printf(_("WARNING: No timing measurements collected. Report this as a bug to <%s>.\n"), PACKAGE_BUGREPORT);
> >   return;
> > }
>
> WFM.

Thanks for confirming!

I've checked with Andres off-list whether he wants to fix this
separately, or together with the earlier proposed patch to show TSC
debug info in pg_test_timing. Andres proposed we take care of this in
the same commit.

Attached v29 patch that contains those earlier changes, plus this
Coverity fix. I've also adjusted x86_tsc_frequency_khz slightly to
clarify the new arguments, and wordsmithed the commit message a bit
for clarity plus explain why we're making this change now.

FWIW, I don't think the TSC debug info change violates any feature
freeze guidelines, and it would have helped investigate the issue on
drongo / have it fail independently of the tablesample tests being
affected by the issue. I think any TSC frequency related bug reports
during beta period will also be greatly aided by having this.

Thanks,
Lukas

-- 
Lukas Fittl

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

Andres Freund <andres@anarazel.de> — 2026-05-16T15:53:49Z

Hi,

On 2026-04-23 23:16:07 -0700, Lukas Fittl wrote:
> Thanks for confirming!
> 
> I've checked with Andres off-list whether he wants to fix this
> separately, or together with the earlier proposed patch to show TSC
> debug info in pg_test_timing. Andres proposed we take care of this in
> the same commit.
> 
> Attached v29 patch that contains those earlier changes, plus this
> Coverity fix. I've also adjusted x86_tsc_frequency_khz slightly to
> clarify the new arguments, and wordsmithed the commit message a bit
> for clarity plus explain why we're making this change now.
> 
> FWIW, I don't think the TSC debug info change violates any feature
> freeze guidelines, and it would have helped investigate the issue on
> drongo / have it fail independently of the tablesample tests being
> affected by the issue. I think any TSC frequency related bug reports
> during beta period will also be greatly aided by having this.

Agreed.


Sorry for loosing track of this for the last couple weeks, I've been
travelling somewhat crazily.  Pushed now.  Let's see whether the BF finds
other problems.

Greetings,

Andres