Re: Introduce XID age and inactive timeout based replication slot invalidation

Peter Smith <smithpb2250@gmail.com>

From: Peter Smith <smithpb2250@gmail.com>

To: Nisha Moond <nisha.moond412@gmail.com>

Cc: vignesh C <vignesh21@gmail.com>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, Amit Kapila <amit.kapila16@gmail.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, shveta malik <shveta.malik@gmail.com>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Ajin Cherian <itsajin@gmail.com>, Bertrand Drouvot <bertranddrouvot.pg@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, Nathan Bossart <nathandbossart@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2025-02-11T21:23:37Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Add a test for commit ac0e33136a using the injection point.
- 8a695d7998be 18.0 landed
Invalidate inactive replication slots.
- ac0e33136abc 18.0 landed
Fix incorrect slot type in BuildTupleHashTableExt
- d96d1d5152f3 18.0 cited
Allow synced slots to have their inactive_since.
- 6f132ed693b6 17.0 landed
Change last_inactive_time to inactive_since in pg_replication_slots.
- 6d49c8d4b4f4 17.0 landed
Track last_inactive_time in pg_replication_slots.
- a11f330b5584 17.0 landed
Track invalidation_reason in pg_replication_slots.
- 6ae701b4378d 17.0 landed
Add option force_initdb to PostgreSQL::Test::Cluster:init()
- ff9e1e764fcc 17.0 cited
Add a failover option to subscriptions.
- 776621a5e479 17.0 cited
Allow setting failover property in the replication command.
- 73292404370c 17.0 cited
Allow to enable failover property for replication slots via SQL API.
- c393308b69d2 17.0 cited
Track conflict_reason in pg_replication_slots.
- 007693f2a3ac 17.0 cited
Log messages for replication slot acquisition and release.
- 7c3fb505b14e 17.0 cited
Remove vacuum_defer_cleanup_age
- 1118cd37eb61 16.0 cited
Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids
- be504a3e974d 16.0 cited
meson: Add initial version of meson based build system
- e6927270cd18 16.0 cited

On Wed, Feb 12, 2025 at 12:36 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Tue, Feb 11, 2025 at 8:49 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Nisha.
> >
> > Some review comments about v74-0001
> >
> > ======
> > src/backend/replication/slot.c
> >
> > 1.
> >  /* Maximum number of invalidation causes */
> > -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
> > -
> > -StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
> > - "array length mismatch");
> > +#define RS_INVAL_MAX_CAUSES (lengthof(InvalidationCauses)-1)
> >
> > The static assert was here to protect against dev mistakes in keeping
> > the lookup table up-to-date with the enum of slot.h. So it's not a
> > good idea to remove it...
> >
> > IMO the RS_INVAL_MAX_CAUSES should be relocated to slot.h where the
> > enum is defined and where the devs know exactly how many invalidation
> > types there are. Then this static assert can be put back in to do its
> > job of ensuring the integrity properly again for this lookup table.
> >
>
> How about keeping RS_INVAL_MAX_CAUSES dynamic in slot.c (as it was)
> and updating the static assert to ensure the lookup table stays
> up-to-date with the enums?
> The change has been implemented in v75.
>

Latest v75-001 patch code looks like:

+static const SlotInvalidationCauseMap InvalidationCauses[] = {
+ {RS_INVAL_NONE, "none"},
+ {RS_INVAL_WAL_REMOVED, "wal_removed"},
+ {RS_INVAL_HORIZON, "rows_removed"},
+ {RS_INVAL_WAL_LEVEL, "wal_level_insufficient"},
+ {RS_INVAL_IDLE_TIMEOUT, "idle_timeout"},
 };

 /* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES (lengthof(InvalidationCauses)-1)

-StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
+/*
+ * Ensure that the lookup table is up-to-date with the enums defined in
+ * ReplicationSlotInvalidationCause. Shifting 1 left by
+ * (RS_INVAL_MAX_CAUSES - 1) should give the highest defined value in
+ * the enum.
+ */
+StaticAssertDecl(RS_INVAL_IDLE_TIMEOUT == (1 << (RS_INVAL_MAX_CAUSES - 1)),
  "array length mismatch");

Where:
1.  RS_INVAL_MAX_CAUSES is based on the length of lookup table so it is 4
2.  the StaticAssert then confirms that the enum RS_INVAL_IDLE_TIMEOUT
 is the 4th enum entry

AFAICT that is not useful. The purpose of the static assert is (like
your comment says) to "Ensure that the lookup table is up-to-date with
the enums". Imagine if I added another (5th cause) enum called
RS_INVAL_BANANA but accidentally overlook updating the lookup table.
The code above isn't going to detect that -- the lookup table length
is still 4 (instead of 5) but RS_INVAL_IDLE_TIMEOUT is still the 4th
enum so the assert is happy. Hence my original suggestion to define
RS_INVAL_MAX_CAUSES adjacent to the enum in slot.h.

======
Kind Regards,
Peter Smith.
Fujitsu Australia