WAL-logging facility for pgstats kinds
Michael Paquier <michael@paquier.xyz>
From: Michael Paquier <michael@paquier.xyz>
To: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Date: 2024-12-27T03:32:25Z
Lists: pgsql-hackers
Attachments
- v1-0001-injection_points-Tweak-variable-numbered-stats-to.patch (text/x-diff) patch v1-0001
- v1-0002-Move-information-about-pgstats-kinds-into-its-own.patch (text/x-diff) patch v1-0002
- v1-0003-Add-RMGR-and-WAL-logging-API-for-pgstats.patch (text/x-diff) patch v1-0003
- v1-0004-injection_points-Add-option-and-tests-for-WAL-log.patch (text/x-diff) patch v1-0004
Hi all, While brainstorming on the contents of the thread I have posted a couple of days ago, I have been looking at what could be done so as pgstats and WAL-logging could work together. This was point 2) from this message: https://www.postgresql.org/message-id/Z2tblEmfuOfZy4zx@paquier.xyz While considering the point that not all stats data is worth replicating, I have fallen down to the following properties that are nice to have across the board: - A pgstats kind should be able to WAL-log some data that is should be able to decide. Including versioning of the data. - The data kind should be able to decide how this data is handled at recovery (different persistency depending on standby vs crash recovery, for one). - Having to add one RMGR for each stats kind is not going to scale, especially as we can control the redo path using a callback part of PgStat_KindInfo. For custom kinds, this enforces some validation based on if the stats library has been really loaded at startup with shared_preload_library. - It is nice for each out-of-core stats kind to not have to load and define a custom RMGR, duplicating what this central facility is doing. - The persistency of the stats data is better across crashes: this approach makes the cleanup of orphaned pgstats entries easier as this can be enforced at replay by each stats kind, if necessary, and it is also possible to force the creation of stats data. - For variable-numbered stats, the WAL-logging can be timed with the timing where any pending stats are flushed, if there is some. Table stats to prevent autovacuum from committing Seppuku for tables without stats after a crash is the first use case I can think about, where we'd want to split and expose some DML stats. Scans stats should remain untouched on standbys, for example. There may be other stats data that is worth replicating, all these could use this new facility. Attached is the result of this investigation, where I have finished by implementing a new backend facility where pgstats kinds can WAL-log data at will using a new RMGR called "PgStat". The most interesting part is pgstat_xlog_data() in a new file called pgstat_xlog.c, that stats kinds can use as an entry point to WAL-log data structures with a XLogRegisterData() managed by a new record structure called xl_pgstat_data. For clarity, the patch set has been split into several pieces, I hope rather edible: - 0001, a fix I've posted on a different thread [1], used in patch 0004 to test this new facility. - 0002, a refactoring piece to be able to expose stats kind data into the frontend (for pg_waldump). - 0003 is the core backend piece, introducing the WAL-logging routine and the new RMGR. - 0004, that provides tests for the new backend facility via a custom stats kind. Requires 0001. I am going to attach it to the next CF. Comments or opinions are welcome. [1]: https://www.postgresql.org/message-id/Z23zcE4w1djukkva@paquier.xyz -- Michael
Commits
-
Move information about pgstats kinds into its own header pgstat_kind.h
- d35ea27e51c0 18.0 landed