Horton

Re: Pluggable cumulative statistics

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: Andrei Lepikhov <lepihov@gmail.com>
Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Date: 2024-07-04T04:25:14Z
Lists: pgsql-hackers 
On Thu, Jul 04, 2024 at 10:11:02AM +0700, Andrei Lepikhov wrote:
> It is a feature my extensions (which usually change planning behaviour)
> definitely need. It is a problem to show the user if the extension does
> something or not because TPS smooths the execution time of a single query
> and performance cliffs.

Yeah, I can get that.  pgstat.c is quite good regarding that as it
delays stats flushes until commit by holding pending entries (see the
pgStatPending business for variable-size stats).  Custom stats kinds
registered would just rely on these facilities, including snapshot
APIs, etc.

> BTW, we have 'labelled DSM segments', which allowed extensions to be
> 'lightweight' - not necessarily be loaded on startup, stay backend-local and
> utilise shared resources. It was a tremendous win for me.
>
> Is it possible to design this extension in the same way?

I am not sure how this would be useful when it comes to cumulative
statistics, TBH.  These stats are global by design, and especially
since these most likely need to be flushed at shutdown (as of HEAD)
and read at startup, the simplest way to achieve that to let the
checkpointer and the startup process know about them is to restrict
the registration of custom stats types via _PG_init when loading
shared libraries.  That's what we do for custom WAL RMGRs, for
example.

I would not be against a new flag in KindInfo to state that a given
stats type should not be flushed, as much as a set of callbacks that
offers the possibility to redirect some stats kinds to somewhere else
than pgstat.stat, like pg_stat_statements.  That would be a separate
patch than what's proposed here.
--
Michael

Commits

  1. injection_points: Add some fixed-numbered statistics

  2. injection_points: Add some cumulative stats for injection points

  3. Add helper routines to retrieve data for custom fixed-numbered pgstats

  4. Introduce pluggable APIs for Cumulative Statistics

  5. Switch PgStat_Kind from an enum to a uint32 type

  6. Improve comment of pgstat_read_statsfile()

  7. Add a new 'F' entry type for fixed-numbered stats in pgstats file

  8. Add PgStat_KindInfo.init_shmem_cb

  9. Use pgstat_kind_infos to write fixed shared statistics

  10. Replace hardcoded identifiers of pgstats file by #defines