Re: per backend I/O statistics

On Tue, Nov 05, 2024 at 05:37:15PM +0000, Bertrand Drouvot wrote:
> I'm starting working on option 2, I think it will be easier to discuss with
> a patch proposal to look at.
> 
> If in the meantime, one strongly disagree with option 2 (means implement a brand
> new PGSTAT_KIND_BACKEND and keep PGSTAT_KIND_IO), please let me know.

Sorry for the late reply, catching up a bit.

As you are quoting in [1], you do not expect the backend-io stats and
the more global pg_stat_io to achieve the same level of consistency as
the backend stats would be gone at restart, and wiped out when a
backend shuts down.  So, splitting them with a different stats kind
feels more natural because it would be possible to control how each
stat kind behaves depending on the code shutdown and reset paths
within their own callbacks rather than making the callbacks of
PGSTAT_KIND_IO more complex than they already are.  And pg_stat_io is
a fixed-numbered stats kind because of the way it aggregates its stats
with a number states defined at compile-time.

Is the structure you have in mind different than PgStat_BktypeIO?
Perhaps a split is better anyway with that in mind.

The amount of memory required to store the snapshots of backend-IO
does not worry me much, TBH, but you are worried about a high turnover
of connections that could cause a lot of bloat in the backend-IO
snapshots because of the persistency that these stats would have,
right?  Actually, we already have cases with other stats kinds where
it is possible for a backend to hold references to some stats on an
object that has been dropped concurrently.  At least, that's possible
when a backend shuts down.  If possible, supporting snapshots would be
more consistent with the other stats.

Just to be clear, I am not in favor of making PgStat_HashKey larger
than it already is.  With 8 bytes allocated for the object ID, the
chances of conflicts in the dshash for a single variable-numbered
stats kind play in our favor already even with a couple of million
entries.

[1]: https://www.postgresql.org/message-id/ZyMRJIbUpNPoCXUe@ip-10-97-1-34.eu-west-3.compute.internal
--
Michael