Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add a test for commit ac0e33136a using the injection point.

  2. Invalidate inactive replication slots.

  3. Fix incorrect slot type in BuildTupleHashTableExt

  4. Allow synced slots to have their inactive_since.

  5. Change last_inactive_time to inactive_since in pg_replication_slots.

  6. Track last_inactive_time in pg_replication_slots.

  7. Track invalidation_reason in pg_replication_slots.

  8. Add option force_initdb to PostgreSQL::Test::Cluster:init()

  9. Add a failover option to subscriptions.

  10. Allow setting failover property in the replication command.

  11. Allow to enable failover property for replication slots via SQL API.

  12. Track conflict_reason in pg_replication_slots.

  13. Log messages for replication slot acquisition and release.

  14. Remove vacuum_defer_cleanup_age

  15. Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids

  16. meson: Add initial version of meson based build system

  1. Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-01-11T05:18:13Z

    Hi,

Replication slots in postgres will prevent removal of required
resources when there is no connection using them (inactive). This
consumes storage because neither required WAL nor required rows from
the user tables/system catalogs can be removed by VACUUM as long as
they are required by a replication slot. In extreme cases this could
cause the transaction ID wraparound.

Currently postgres has the ability to invalidate inactive replication
slots based on the amount of WAL (set via max_slot_wal_keep_size GUC)
that will be needed for the slots in case they become active. However,
the wraparound issue isn't effectively covered by
max_slot_wal_keep_size - one can't tell postgres to invalidate a
replication slot if it is blocking VACUUM. Also, it is often tricky to
choose a default value for max_slot_wal_keep_size, because the amount
of WAL that gets generated and allocated storage for the database can
vary.

Therefore, it is often easy for developers to do the following:
a) set an XID age (age of slot's xmin or catalog_xmin) of say 1 or 1.5
billion, after which the slots get invalidated.
b) set a timeout of say 1 or 2 or 3 days, after which the inactive
slots get invalidated.

To implement (a), postgres needs a new GUC called max_slot_xid_age.
The checkpointer then invalidates all the slots whose xmin (the oldest
transaction that this slot needs the database to retain) or
catalog_xmin (the oldest transaction affecting the system catalogs
that this slot needs the database to retain) has reached the age
specified by this setting.

To implement (b), first postgres needs to track the replication slot
metrics like the time at which the slot became inactive (inactive_at
timestamptz) and the total number of times the slot became inactive in
its lifetime (inactive_count numeric) in ReplicationSlotPersistentData
structure. And, then it needs a new timeout GUC called
inactive_replication_slot_timeout. Whenever a slot becomes inactive,
the current timestamp and inactive count are stored in
ReplicationSlotPersistentData structure and persisted to disk. The
checkpointer then invalidates all the slots that are lying inactive
for about inactive_replication_slot_timeout duration starting from
inactive_at.

In addition to implementing (b), these two new metrics enable
developers to improve their monitoring tools as the metrics are
exposed via pg_replication_slots system view. For instance, one can
build a monitoring tool that signals when replication slots are lying
inactive for a day or so using inactive_at metric, and/or when a
replication slot is becoming inactive too frequently using inactive_at
metric.

I’m attaching the v1 patch set as described below:
0001 - Tracks invalidation_reason in pg_replication_slots. This is
needed because slots now have multiple reasons for slot invalidation.
0002 - Tracks inactive replication slot information inactive_at and
inactive_timeout.
0003 - Adds inactive_timeout based replication slot invalidation.
0004 - Adds XID based replication slot invalidation.

Thoughts?

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  2. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-01-26T19:48:00Z

    On Thu, Jan 11, 2024 at 10:48 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> Replication slots in postgres will prevent removal of required
> resources when there is no connection using them (inactive). This
> consumes storage because neither required WAL nor required rows from
> the user tables/system catalogs can be removed by VACUUM as long as
> they are required by a replication slot. In extreme cases this could
> cause the transaction ID wraparound.
>
> Currently postgres has the ability to invalidate inactive replication
> slots based on the amount of WAL (set via max_slot_wal_keep_size GUC)
> that will be needed for the slots in case they become active. However,
> the wraparound issue isn't effectively covered by
> max_slot_wal_keep_size - one can't tell postgres to invalidate a
> replication slot if it is blocking VACUUM. Also, it is often tricky to
> choose a default value for max_slot_wal_keep_size, because the amount
> of WAL that gets generated and allocated storage for the database can
> vary.
>
> Therefore, it is often easy for developers to do the following:
> a) set an XID age (age of slot's xmin or catalog_xmin) of say 1 or 1.5
> billion, after which the slots get invalidated.
> b) set a timeout of say 1 or 2 or 3 days, after which the inactive
> slots get invalidated.
>
> To implement (a), postgres needs a new GUC called max_slot_xid_age.
> The checkpointer then invalidates all the slots whose xmin (the oldest
> transaction that this slot needs the database to retain) or
> catalog_xmin (the oldest transaction affecting the system catalogs
> that this slot needs the database to retain) has reached the age
> specified by this setting.
>
> To implement (b), first postgres needs to track the replication slot
> metrics like the time at which the slot became inactive (inactive_at
> timestamptz) and the total number of times the slot became inactive in
> its lifetime (inactive_count numeric) in ReplicationSlotPersistentData
> structure. And, then it needs a new timeout GUC called
> inactive_replication_slot_timeout. Whenever a slot becomes inactive,
> the current timestamp and inactive count are stored in
> ReplicationSlotPersistentData structure and persisted to disk. The
> checkpointer then invalidates all the slots that are lying inactive
> for about inactive_replication_slot_timeout duration starting from
> inactive_at.
>
> In addition to implementing (b), these two new metrics enable
> developers to improve their monitoring tools as the metrics are
> exposed via pg_replication_slots system view. For instance, one can
> build a monitoring tool that signals when replication slots are lying
> inactive for a day or so using inactive_at metric, and/or when a
> replication slot is becoming inactive too frequently using inactive_at
> metric.
>
> I’m attaching the v1 patch set as described below:
> 0001 - Tracks invalidation_reason in pg_replication_slots. This is
> needed because slots now have multiple reasons for slot invalidation.
> 0002 - Tracks inactive replication slot information inactive_at and
> inactive_timeout.
> 0003 - Adds inactive_timeout based replication slot invalidation.
> 0004 - Adds XID based replication slot invalidation.
>
> Thoughts?

Needed a rebase due to c393308b. Please find the attached v2 patch set.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  3. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-01-31T13:05:00Z

    On Sat, Jan 27, 2024 at 1:18 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Thu, Jan 11, 2024 at 10:48 AM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Hi,
> >
> > Replication slots in postgres will prevent removal of required
> > resources when there is no connection using them (inactive). This
> > consumes storage because neither required WAL nor required rows from
> > the user tables/system catalogs can be removed by VACUUM as long as
> > they are required by a replication slot. In extreme cases this could
> > cause the transaction ID wraparound.
> >
> > Currently postgres has the ability to invalidate inactive replication
> > slots based on the amount of WAL (set via max_slot_wal_keep_size GUC)
> > that will be needed for the slots in case they become active. However,
> > the wraparound issue isn't effectively covered by
> > max_slot_wal_keep_size - one can't tell postgres to invalidate a
> > replication slot if it is blocking VACUUM. Also, it is often tricky to
> > choose a default value for max_slot_wal_keep_size, because the amount
> > of WAL that gets generated and allocated storage for the database can
> > vary.
> >
> > Therefore, it is often easy for developers to do the following:
> > a) set an XID age (age of slot's xmin or catalog_xmin) of say 1 or 1.5
> > billion, after which the slots get invalidated.
> > b) set a timeout of say 1 or 2 or 3 days, after which the inactive
> > slots get invalidated.
> >
> > To implement (a), postgres needs a new GUC called max_slot_xid_age.
> > The checkpointer then invalidates all the slots whose xmin (the oldest
> > transaction that this slot needs the database to retain) or
> > catalog_xmin (the oldest transaction affecting the system catalogs
> > that this slot needs the database to retain) has reached the age
> > specified by this setting.
> >
> > To implement (b), first postgres needs to track the replication slot
> > metrics like the time at which the slot became inactive (inactive_at
> > timestamptz) and the total number of times the slot became inactive in
> > its lifetime (inactive_count numeric) in ReplicationSlotPersistentData
> > structure. And, then it needs a new timeout GUC called
> > inactive_replication_slot_timeout. Whenever a slot becomes inactive,
> > the current timestamp and inactive count are stored in
> > ReplicationSlotPersistentData structure and persisted to disk. The
> > checkpointer then invalidates all the slots that are lying inactive
> > for about inactive_replication_slot_timeout duration starting from
> > inactive_at.
> >
> > In addition to implementing (b), these two new metrics enable
> > developers to improve their monitoring tools as the metrics are
> > exposed via pg_replication_slots system view. For instance, one can
> > build a monitoring tool that signals when replication slots are lying
> > inactive for a day or so using inactive_at metric, and/or when a
> > replication slot is becoming inactive too frequently using inactive_at
> > metric.
> >
> > I’m attaching the v1 patch set as described below:
> > 0001 - Tracks invalidation_reason in pg_replication_slots. This is
> > needed because slots now have multiple reasons for slot invalidation.
> > 0002 - Tracks inactive replication slot information inactive_at and
> > inactive_timeout.
> > 0003 - Adds inactive_timeout based replication slot invalidation.
> > 0004 - Adds XID based replication slot invalidation.
> >
> > Thoughts?
>
> Needed a rebase due to c393308b. Please find the attached v2 patch set.

Needed a rebase due to commit 776621a (conflict in
src/test/recovery/meson.build for new TAP test file added). Please
find the attached v3 patch set.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  4. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-02-05T09:45:50Z

    Hi,

On Thu, Jan 11, 2024 at 10:48:13AM +0530, Bharath Rupireddy wrote:
> Hi,
> 
> Therefore, it is often easy for developers to do the following:
> a) set an XID age (age of slot's xmin or catalog_xmin) of say 1 or 1.5
> billion, after which the slots get invalidated.
> b) set a timeout of say 1 or 2 or 3 days, after which the inactive
> slots get invalidated.
> 
> To implement (a), postgres needs a new GUC called max_slot_xid_age.
> The checkpointer then invalidates all the slots whose xmin (the oldest
> transaction that this slot needs the database to retain) or
> catalog_xmin (the oldest transaction affecting the system catalogs
> that this slot needs the database to retain) has reached the age
> specified by this setting.
> 
> To implement (b), first postgres needs to track the replication slot
> metrics like the time at which the slot became inactive (inactive_at
> timestamptz) and the total number of times the slot became inactive in
> its lifetime (inactive_count numeric) in ReplicationSlotPersistentData
> structure. And, then it needs a new timeout GUC called
> inactive_replication_slot_timeout. Whenever a slot becomes inactive,
> the current timestamp and inactive count are stored in
> ReplicationSlotPersistentData structure and persisted to disk. The
> checkpointer then invalidates all the slots that are lying inactive
> for about inactive_replication_slot_timeout duration starting from
> inactive_at.
> 
> In addition to implementing (b), these two new metrics enable
> developers to improve their monitoring tools as the metrics are
> exposed via pg_replication_slots system view. For instance, one can
> build a monitoring tool that signals when replication slots are lying
> inactive for a day or so using inactive_at metric, and/or when a
> replication slot is becoming inactive too frequently using inactive_at
> metric.

Thanks for the patch and +1 for the idea, I think adding those new
"invalidation reasons" make sense.

> 
> I’m attaching the v1 patch set as described below:
> 0001 - Tracks invalidation_reason in pg_replication_slots. This is
> needed because slots now have multiple reasons for slot invalidation.
> 0002 - Tracks inactive replication slot information inactive_at and
> inactive_timeout.
> 0003 - Adds inactive_timeout based replication slot invalidation.
> 0004 - Adds XID based replication slot invalidation.
>

I think it's better to have the XID one being discussed/implemented before the
inactive_timeout one: what about changing the 0002, 0003 and 0004 ordering?

0004 -> 0002
0002 -> 0003
0003 -> 0004

As far 0001:

"
This commit renames conflict_reason to
invalidation_reason, and adds the support to show invalidation
reasons for both physical and logical slots.
"

I'm not sure I like the fact that "invalidations" and "conflicts" are merged
into a single field. I'd vote to keep conflict_reason as it is and add a new
invalidation_reason (and put "conflict" as value when it is the case). The reason
is that I think they are 2 different concepts (could be linked though) and that
it would be easier to check for conflicts (means conflict_reason is not NULL).

Regards,
 
-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  5. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Dilip Kumar <dilipbalaut@gmail.com> — 2024-02-06T08:46:19Z

    On Thu, Jan 11, 2024 at 10:48 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> Replication slots in postgres will prevent removal of required
> resources when there is no connection using them (inactive). This
> consumes storage because neither required WAL nor required rows from
> the user tables/system catalogs can be removed by VACUUM as long as
> they are required by a replication slot. In extreme cases this could
> cause the transaction ID wraparound.
>
> Currently postgres has the ability to invalidate inactive replication
> slots based on the amount of WAL (set via max_slot_wal_keep_size GUC)
> that will be needed for the slots in case they become active. However,
> the wraparound issue isn't effectively covered by
> max_slot_wal_keep_size - one can't tell postgres to invalidate a
> replication slot if it is blocking VACUUM. Also, it is often tricky to
> choose a default value for max_slot_wal_keep_size, because the amount
> of WAL that gets generated and allocated storage for the database can
> vary.
>
> Therefore, it is often easy for developers to do the following:
> a) set an XID age (age of slot's xmin or catalog_xmin) of say 1 or 1.5
> billion, after which the slots get invalidated.
> b) set a timeout of say 1 or 2 or 3 days, after which the inactive
> slots get invalidated.
>
> To implement (a), postgres needs a new GUC called max_slot_xid_age.
> The checkpointer then invalidates all the slots whose xmin (the oldest
> transaction that this slot needs the database to retain) or
> catalog_xmin (the oldest transaction affecting the system catalogs
> that this slot needs the database to retain) has reached the age
> specified by this setting.
>
> To implement (b), first postgres needs to track the replication slot
> metrics like the time at which the slot became inactive (inactive_at
> timestamptz) and the total number of times the slot became inactive in
> its lifetime (inactive_count numeric) in ReplicationSlotPersistentData
> structure. And, then it needs a new timeout GUC called
> inactive_replication_slot_timeout. Whenever a slot becomes inactive,
> the current timestamp and inactive count are stored in
> ReplicationSlotPersistentData structure and persisted to disk. The
> checkpointer then invalidates all the slots that are lying inactive
> for about inactive_replication_slot_timeout duration starting from
> inactive_at.
>
> In addition to implementing (b), these two new metrics enable
> developers to improve their monitoring tools as the metrics are
> exposed via pg_replication_slots system view. For instance, one can
> build a monitoring tool that signals when replication slots are lying
> inactive for a day or so using inactive_at metric, and/or when a
> replication slot is becoming inactive too frequently using inactive_at
> metric.
>
> I’m attaching the v1 patch set as described below:
> 0001 - Tracks invalidation_reason in pg_replication_slots. This is
> needed because slots now have multiple reasons for slot invalidation.
> 0002 - Tracks inactive replication slot information inactive_at and
> inactive_timeout.
> 0003 - Adds inactive_timeout based replication slot invalidation.
> 0004 - Adds XID based replication slot invalidation.
>
> Thoughts?
>
+1 for the idea,  here are some comments on 0002, I will review other
patches soon and respond.

1.
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>inactive_at</structfield> <type>timestamptz</type>
+      </para>
+      <para>
+        The time at which the slot became inactive.
+        <literal>NULL</literal> if the slot is currently actively being
+        used.
+      </para></entry>
+     </row>

Maybe we can change the field name to 'last_inactive_at'? or maybe the
comment can explain timestampt at which slot was last inactivated.
I think since we are already maintaining the inactive_count so better
to explicitly say this is the last invaliding time.

2.
+ /*
+ * XXX: Can inactive_count of type uint64 ever overflow? It takes
+ * about a half-billion years for inactive_count to overflow even
+ * if slot becomes inactive for every 1 millisecond. So, using
+ * pg_add_u64_overflow might be an overkill.
+ */

Correct we don't need to use pg_add_u64_overflow for this counter.

3.

+
+ /* Convert to numeric. */
+ snprintf(buf, sizeof buf, UINT64_FORMAT, slot_contents.data.inactive_count);
+ values[i++] = DirectFunctionCall3(numeric_in,
+   CStringGetDatum(buf),
+   ObjectIdGetDatum(0),
+   Int32GetDatum(-1));

What is the purpose of doing this? I mean inactive_count is 8 byte
integer and you can define function outparameter as 'int8' which is 8
byte integer.  Then you don't need to convert int to string and then
to numeric?


-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  6. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-02-06T17:32:33Z

    On Tue, Feb 6, 2024 at 2:16 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> > Thoughts?
> >
> +1 for the idea,  here are some comments on 0002, I will review other
> patches soon and respond.

Thanks for looking at it.

> +       <structfield>inactive_at</structfield> <type>timestamptz</type>
>
> Maybe we can change the field name to 'last_inactive_at'? or maybe the
> comment can explain timestampt at which slot was last inactivated.
> I think since we are already maintaining the inactive_count so better
> to explicitly say this is the last invaliding time.

last_inactive_at looks better, so will use that in the next version of
the patch.

> 2.
> + /*
> + * XXX: Can inactive_count of type uint64 ever overflow? It takes
> + * about a half-billion years for inactive_count to overflow even
> + * if slot becomes inactive for every 1 millisecond. So, using
> + * pg_add_u64_overflow might be an overkill.
> + */
>
> Correct we don't need to use pg_add_u64_overflow for this counter.

Will remove this comment in the next version of the patch.

> + /* Convert to numeric. */
> + snprintf(buf, sizeof buf, UINT64_FORMAT, slot_contents.data.inactive_count);
> + values[i++] = DirectFunctionCall3(numeric_in,
> +   CStringGetDatum(buf),
> +   ObjectIdGetDatum(0),
> +   Int32GetDatum(-1));
>
> What is the purpose of doing this? I mean inactive_count is 8 byte
> integer and you can define function outparameter as 'int8' which is 8
> byte integer.  Then you don't need to convert int to string and then
> to numeric?

Nope, it's of type uint64, so reporting it as numeric is a way
typically used elsewhere - see code around /* Convert to numeric. */.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  7. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-02-06T18:52:07Z

    On Mon, Feb 5, 2024 at 3:15 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Thanks for the patch and +1 for the idea, I think adding those new
> "invalidation reasons" make sense.

Thanks for looking at it.

> I think it's better to have the XID one being discussed/implemented before the
> inactive_timeout one: what about changing the 0002, 0003 and 0004 ordering?
>
> 0004 -> 0002
> 0002 -> 0003
> 0003 -> 0004

Done that way.

> As far 0001:
>
> "
> This commit renames conflict_reason to
> invalidation_reason, and adds the support to show invalidation
> reasons for both physical and logical slots.
> "
>
> I'm not sure I like the fact that "invalidations" and "conflicts" are merged
> into a single field. I'd vote to keep conflict_reason as it is and add a new
> invalidation_reason (and put "conflict" as value when it is the case). The reason
> is that I think they are 2 different concepts (could be linked though) and that
> it would be easier to check for conflicts (means conflict_reason is not NULL).

So, do you want conflict_reason for only logical slots, and a separate
column for invalidation_reason for both logical and physical slots? Is
there any strong reason to have two properties "conflict" and
"invalidated" for slots? They both are the same internally, so why
confuse the users?

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  8. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-02-09T07:42:53Z

    Hi,

On Wed, Feb 07, 2024 at 12:22:07AM +0530, Bharath Rupireddy wrote:
> On Mon, Feb 5, 2024 at 3:15 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > I'm not sure I like the fact that "invalidations" and "conflicts" are merged
> > into a single field. I'd vote to keep conflict_reason as it is and add a new
> > invalidation_reason (and put "conflict" as value when it is the case). The reason
> > is that I think they are 2 different concepts (could be linked though) and that
> > it would be easier to check for conflicts (means conflict_reason is not NULL).
> 
> So, do you want conflict_reason for only logical slots, and a separate
> column for invalidation_reason for both logical and physical slots?

Yes, with "conflict" as value in case of conflicts (and one would need to refer
to the conflict_reason reason to see the reason).

> Is there any strong reason to have two properties "conflict" and
> "invalidated" for slots?

I think "conflict" is an important topic and does contain several reasons. The
slot "first" conflict and then leads to slot "invalidation". 

> They both are the same internally, so why
> confuse the users?

I don't think that would confuse the users, I do think that would be easier to
check for conflicting slots.

I did not look closely at the code, just played a bit with the patch and was able
to produce something like:

postgres=# select slot_name,slot_type,active,active_pid,wal_status,invalidation_reason from pg_replication_slots;
  slot_name  | slot_type | active | active_pid | wal_status | invalidation_reason
-------------+-----------+--------+------------+------------+---------------------
 rep1        | physical  | f      |            | reserved   |
 master_slot | physical  | t      |    1482441 | unreserved | wal_removed
(2 rows)

does that make sense to have an "active/working" slot "ivalidated"?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  9. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-02-20T06:35:00Z

    On Fri, Feb 9, 2024 at 1:12 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> I think "conflict" is an important topic and does contain several reasons. The
> slot "first" conflict and then leads to slot "invalidation".
>
> > They both are the same internally, so why
> > confuse the users?
>
> I don't think that would confuse the users, I do think that would be easier to
> check for conflicting slots.

I've added a separate column for invalidation reasons for now. I'll
see how others think on this as the time goes by.

> I did not look closely at the code, just played a bit with the patch and was able
> to produce something like:
>
> postgres=# select slot_name,slot_type,active,active_pid,wal_status,invalidation_reason from pg_replication_slots;
>   slot_name  | slot_type | active | active_pid | wal_status | invalidation_reason
> -------------+-----------+--------+------------+------------+---------------------
>  rep1        | physical  | f      |            | reserved   |
>  master_slot | physical  | t      |    1482441 | unreserved | wal_removed
> (2 rows)
>
> does that make sense to have an "active/working" slot "ivalidated"?

Thanks. Can you please provide the steps to generate this error? Are
you setting max_slot_wal_keep_size on primary to generate
"wal_removed"?

Attached v5 patch set after rebasing and addressing review comments.
Please review it further.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  10. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-02-21T05:25:00Z

    On Tue, Feb 20, 2024 at 12:05 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
>> [...] and was able to produce something like:
> >
> > postgres=# select slot_name,slot_type,active,active_pid,wal_status,invalidation_reason from pg_replication_slots;
> >   slot_name  | slot_type | active | active_pid | wal_status | invalidation_reason
> > -------------+-----------+--------+------------+------------+---------------------
> >  rep1        | physical  | f      |            | reserved   |
> >  master_slot | physical  | t      |    1482441 | unreserved | wal_removed
> > (2 rows)
> >
> > does that make sense to have an "active/working" slot "ivalidated"?
>
> Thanks. Can you please provide the steps to generate this error? Are
> you setting max_slot_wal_keep_size on primary to generate
> "wal_removed"?

I'm able to reproduce [1] the state [2] where the slot got invalidated
first, then its wal_status became unreserved, but still the slot is
serving after the standby comes up online after it catches up with the
primary getting the WAL files from the archive. There's a good reason
for this state -
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/replication/slotfuncs.c;h=d2fa5e669a32f19989b0d987d3c7329851a1272e;hb=ff9e1e764fcce9a34467d614611a34d4d2a91b50#l351.
This intermittent state can only happen for physical slots, not for
logical slots because logical subscribers can't get the missing
changes from the WAL stored in the archive.

And, the fact looks to be that an invalidated slot can never become
normal but still can serve a standby if the standby is able to catch
up by fetching required WAL (this is the WAL the slot couldn't keep
for the standby) from elsewhere (archive via restore_command).

As far as the 0001 patch is concerned, it reports the
invalidation_reason as long as slot_contents.data.invalidated !=
RS_INVAL_NONE. I think this is okay.

Thoughts?

[1]
./initdb -D db17
echo "max_wal_size = 128MB
max_slot_wal_keep_size = 64MB
archive_mode = on
archive_command='cp %p
/home/ubuntu/postgres/pg17/bin/archived_wal/%f'" | tee -a
db17/postgresql.conf

./pg_ctl -D db17 -l logfile17 start

./psql -d postgres -p 5432 -c "SELECT
pg_create_physical_replication_slot('sb_repl_slot', true, false);"

rm -rf sbdata logfilesbdata
./pg_basebackup -D sbdata

echo "port=5433
primary_conninfo='host=localhost port=5432 dbname=postgres user=ubuntu'
primary_slot_name='sb_repl_slot'
restore_command='cp /home/ubuntu/postgres/pg17/bin/archived_wal/%f
%p'" | tee -a sbdata/postgresql.conf

touch sbdata/standby.signal

./pg_ctl -D sbdata -l logfilesbdata start
./psql -d postgres -p 5433 -c "SELECT pg_is_in_recovery();"

./pg_ctl -D sbdata -l logfilesbdata stop

./psql -d postgres -p 5432 -c "SELECT pg_logical_emit_message(true,
'mymessage', repeat('aaaa', 10000000));"
./psql -d postgres -p 5432 -c "CHECKPOINT;"
./pg_ctl -D sbdata -l logfilesbdata start
./psql -d postgres -p 5432 -xc "SELECT * FROM pg_replication_slots;"

[2]
postgres=# SELECT * FROM pg_replication_slots;
-[ RECORD 1 ]-------+-------------
slot_name           | sb_repl_slot
plugin              |
slot_type           | physical
datoid              |
database            |
temporary           | f
active              | t
active_pid          | 710667
xmin                |
catalog_xmin        |
restart_lsn         | 0/115D21A0
confirmed_flush_lsn |
wal_status          | unreserved
safe_wal_size       | 77782624
two_phase           | f
conflict_reason     |
failover            | f
synced              | f
invalidation_reason | wal_removed
last_inactive_at    |
inactive_count      | 1

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  11. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-02-21T12:25:25Z

    Hi,

On Wed, Feb 21, 2024 at 10:55:00AM +0530, Bharath Rupireddy wrote:
> On Tue, Feb 20, 2024 at 12:05 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> >> [...] and was able to produce something like:
> > >
> > > postgres=# select slot_name,slot_type,active,active_pid,wal_status,invalidation_reason from pg_replication_slots;
> > >   slot_name  | slot_type | active | active_pid | wal_status | invalidation_reason
> > > -------------+-----------+--------+------------+------------+---------------------
> > >  rep1        | physical  | f      |            | reserved   |
> > >  master_slot | physical  | t      |    1482441 | unreserved | wal_removed
> > > (2 rows)
> > >
> > > does that make sense to have an "active/working" slot "ivalidated"?
> >
> > Thanks. Can you please provide the steps to generate this error? Are
> > you setting max_slot_wal_keep_size on primary to generate
> > "wal_removed"?
> 
> I'm able to reproduce [1] the state [2] where the slot got invalidated
> first, then its wal_status became unreserved, but still the slot is
> serving after the standby comes up online after it catches up with the
> primary getting the WAL files from the archive. There's a good reason
> for this state -
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/replication/slotfuncs.c;h=d2fa5e669a32f19989b0d987d3c7329851a1272e;hb=ff9e1e764fcce9a34467d614611a34d4d2a91b50#l351.
> This intermittent state can only happen for physical slots, not for
> logical slots because logical subscribers can't get the missing
> changes from the WAL stored in the archive.
> 
> And, the fact looks to be that an invalidated slot can never become
> normal but still can serve a standby if the standby is able to catch
> up by fetching required WAL (this is the WAL the slot couldn't keep
> for the standby) from elsewhere (archive via restore_command).
> 
> As far as the 0001 patch is concerned, it reports the
> invalidation_reason as long as slot_contents.data.invalidated !=
> RS_INVAL_NONE. I think this is okay.
> 
> Thoughts?

Yeah, looking at the code I agree that looks ok. OTOH, that looks confusing,
maybe we should add a few words about it in the doc?

Looking at v5-0001:

+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>invalidation_reason</structfield> <type>text</type>
+      </para>
+      <para>

My initial thought was to put "conflict" value in this new field in case of
conflict (not to mention the conflict reason in it). With the current proposal
invalidation_reason could report the same as conflict_reason, which sounds weird
to me.

Does that make sense to you to use "conflict" as value in "invalidation_reason"
when the slot has "conflict_reason" not NULL?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  12. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-02-21T14:40:00Z

    On Wed, Feb 21, 2024 at 5:55 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > As far as the 0001 patch is concerned, it reports the
> > invalidation_reason as long as slot_contents.data.invalidated !=
> > RS_INVAL_NONE. I think this is okay.
> >
> > Thoughts?
>
> Yeah, looking at the code I agree that looks ok. OTOH, that looks confusing,
> maybe we should add a few words about it in the doc?

I'll think about it.

> Looking at v5-0001:
>
> +      <entry role="catalog_table_entry"><para role="column_definition">
> +       <structfield>invalidation_reason</structfield> <type>text</type>
> +      </para>
> +      <para>
>
> My initial thought was to put "conflict" value in this new field in case of
> conflict (not to mention the conflict reason in it). With the current proposal
> invalidation_reason could report the same as conflict_reason, which sounds weird
> to me.
>
> Does that make sense to you to use "conflict" as value in "invalidation_reason"
> when the slot has "conflict_reason" not NULL?

I'm thinking the other way around - how about we revert
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=007693f2a3ac2ac19affcb03ad43cdb36ccff5b5,
that is, put in place "conflict" as a boolean and introduce
invalidation_reason the text form. So, for logical slots, whenever the
"conflict" column is true, the reason is found in invaldiation_reason
column? How does it sound? Again the debate might be "conflict" vs
"invalidation", but that looks clean IMHO.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  13. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-02-22T08:14:57Z

    Hi,

On Wed, Feb 21, 2024 at 08:10:00PM +0530, Bharath Rupireddy wrote:
> On Wed, Feb 21, 2024 at 5:55 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> > My initial thought was to put "conflict" value in this new field in case of
> > conflict (not to mention the conflict reason in it). With the current proposal
> > invalidation_reason could report the same as conflict_reason, which sounds weird
> > to me.
> >
> > Does that make sense to you to use "conflict" as value in "invalidation_reason"
> > when the slot has "conflict_reason" not NULL?
> 
> I'm thinking the other way around - how about we revert
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=007693f2a3ac2ac19affcb03ad43cdb36ccff5b5,
> that is, put in place "conflict" as a boolean and introduce
> invalidation_reason the text form. So, for logical slots, whenever the
> "conflict" column is true, the reason is found in invaldiation_reason
> column? How does it sound?

Yeah, I think that looks fine too. We would need more change (like take care of
ddd5f4f54a for example).

CC'ing Amit, Hou-San and Shveta to get their point of view (as the ones behind
007693f2a3 and ddd5f4f54a).

Regarding,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  14. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-01T14:32:00Z

    On Thu, Feb 22, 2024 at 1:44 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > > Does that make sense to you to use "conflict" as value in "invalidation_reason"
> > > when the slot has "conflict_reason" not NULL?
> >
> > I'm thinking the other way around - how about we revert
> > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=007693f2a3ac2ac19affcb03ad43cdb36ccff5b5,
> > that is, put in place "conflict" as a boolean and introduce
> > invalidation_reason the text form. So, for logical slots, whenever the
> > "conflict" column is true, the reason is found in invaldiation_reason
> > column? How does it sound?
>
> Yeah, I think that looks fine too. We would need more change (like take care of
> ddd5f4f54a for example).
>
> CC'ing Amit, Hou-San and Shveta to get their point of view (as the ones behind
> 007693f2a3 and ddd5f4f54a).

Yeah, let's wait for what others think about it.

FWIW, I've had to rebase the patches due to 943f7ae1c. Please see the
attached v6 patch set.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  15. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-01T22:11:08Z

    On Wed, Feb 21, 2024 at 08:10:00PM +0530, Bharath Rupireddy wrote:
> I'm thinking the other way around - how about we revert
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=007693f2a3ac2ac19affcb03ad43cdb36ccff5b5,
> that is, put in place "conflict" as a boolean and introduce
> invalidation_reason the text form. So, for logical slots, whenever the
> "conflict" column is true, the reason is found in invaldiation_reason
> column? How does it sound? Again the debate might be "conflict" vs
> "invalidation", but that looks clean IMHO.

Would you ever see "conflict" as false and "invalidation_reason" as
non-null for a logical slot?

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  16. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-03T18:10:00Z

    On Sat, Mar 2, 2024 at 3:41 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> > [....] how about we revert
> > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=007693f2a3ac2ac19affcb03ad43cdb36ccff5b5,
>
> Would you ever see "conflict" as false and "invalidation_reason" as
> non-null for a logical slot?

No. Because both conflict and invalidation_reason are decided based on
the invalidation reason i.e. value of slot_contents.data.invalidated.
IOW, a logical slot that reports conflict as true must have been
invalidated.

Do you have any thoughts on reverting 007693f and introducing
invalidation_reason?

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  17. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-03T21:44:34Z

    On Sun, Mar 03, 2024 at 11:40:00PM +0530, Bharath Rupireddy wrote:
> On Sat, Mar 2, 2024 at 3:41 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>> Would you ever see "conflict" as false and "invalidation_reason" as
>> non-null for a logical slot?
> 
> No. Because both conflict and invalidation_reason are decided based on
> the invalidation reason i.e. value of slot_contents.data.invalidated.
> IOW, a logical slot that reports conflict as true must have been
> invalidated.
> 
> Do you have any thoughts on reverting 007693f and introducing
> invalidation_reason?

Unless I am misinterpreting some details, ISTM we could rename this column
to invalidation_reason and use it for both logical and physical slots.  I'm
not seeing a strong need for another column.  Perhaps I am missing
something...

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  18. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Michael Paquier <michael@paquier.xyz> — 2024-03-04T01:32:00Z

    On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote:
> On Sun, Mar 03, 2024 at 11:40:00PM +0530, Bharath Rupireddy wrote:
>> Do you have any thoughts on reverting 007693f and introducing
>> invalidation_reason?
> 
> Unless I am misinterpreting some details, ISTM we could rename this column
> to invalidation_reason and use it for both logical and physical slots.  I'm
> not seeing a strong need for another column.  Perhaps I am missing
> something...

And also, please don't be hasty in taking a decision that would
involve a revert of 007693f without informing the committer of this 
commit about that.  I am adding Amit Kapila in CC of this thread for
awareness.
--
Michael

  19. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-04T08:41:01Z

    Hi,

On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote:
> On Sun, Mar 03, 2024 at 11:40:00PM +0530, Bharath Rupireddy wrote:
> > On Sat, Mar 2, 2024 at 3:41 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
> >> Would you ever see "conflict" as false and "invalidation_reason" as
> >> non-null for a logical slot?
> > 
> > No. Because both conflict and invalidation_reason are decided based on
> > the invalidation reason i.e. value of slot_contents.data.invalidated.
> > IOW, a logical slot that reports conflict as true must have been
> > invalidated.
> > 
> > Do you have any thoughts on reverting 007693f and introducing
> > invalidation_reason?
> 
> Unless I am misinterpreting some details, ISTM we could rename this column
> to invalidation_reason and use it for both logical and physical slots.  I'm
> not seeing a strong need for another column.

Yeah having two columns was more for convenience purpose. Without the "conflict"
one, a slot conflicting with recovery would be "a logical slot having a non NULL
invalidation_reason".

I'm also fine with one column if most of you prefer that way.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  20. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-05T19:20:38Z

    On Mon, Mar 4, 2024 at 2:11 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote:
> > On Sun, Mar 03, 2024 at 11:40:00PM +0530, Bharath Rupireddy wrote:
> > > On Sat, Mar 2, 2024 at 3:41 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
> > >> Would you ever see "conflict" as false and "invalidation_reason" as
> > >> non-null for a logical slot?
> > >
> > > No. Because both conflict and invalidation_reason are decided based on
> > > the invalidation reason i.e. value of slot_contents.data.invalidated.
> > > IOW, a logical slot that reports conflict as true must have been
> > > invalidated.
> > >
> > > Do you have any thoughts on reverting 007693f and introducing
> > > invalidation_reason?
> >
> > Unless I am misinterpreting some details, ISTM we could rename this column
> > to invalidation_reason and use it for both logical and physical slots.  I'm
> > not seeing a strong need for another column.
>
> Yeah having two columns was more for convenience purpose. Without the "conflict"
> one, a slot conflicting with recovery would be "a logical slot having a non NULL
> invalidation_reason".
>
> I'm also fine with one column if most of you prefer that way.

While we debate on the above, please find the attached v7 patch set
after rebasing.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  21. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-05T19:44:43Z

    On Wed, Mar 06, 2024 at 12:50:38AM +0530, Bharath Rupireddy wrote:
> On Mon, Mar 4, 2024 at 2:11 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
>> On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote:
>> > Unless I am misinterpreting some details, ISTM we could rename this column
>> > to invalidation_reason and use it for both logical and physical slots.  I'm
>> > not seeing a strong need for another column.
>>
>> Yeah having two columns was more for convenience purpose. Without the "conflict"
>> one, a slot conflicting with recovery would be "a logical slot having a non NULL
>> invalidation_reason".
>>
>> I'm also fine with one column if most of you prefer that way.
> 
> While we debate on the above, please find the attached v7 patch set
> after rebasing.

It looks like Bertrand is okay with reusing the same column for both
logical and physical slots, which IIUC is what you initially proposed in v1
of the patch set.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  22. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-06T09:12:15Z

    Hi,

On Tue, Mar 05, 2024 at 01:44:43PM -0600, Nathan Bossart wrote:
> On Wed, Mar 06, 2024 at 12:50:38AM +0530, Bharath Rupireddy wrote:
> > On Mon, Mar 4, 2024 at 2:11 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> >> On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote:
> >> > Unless I am misinterpreting some details, ISTM we could rename this column
> >> > to invalidation_reason and use it for both logical and physical slots.  I'm
> >> > not seeing a strong need for another column.
> >>
> >> Yeah having two columns was more for convenience purpose. Without the "conflict"
> >> one, a slot conflicting with recovery would be "a logical slot having a non NULL
> >> invalidation_reason".
> >>
> >> I'm also fine with one column if most of you prefer that way.
> > 
> > While we debate on the above, please find the attached v7 patch set
> > after rebasing.
> 
> It looks like Bertrand is okay with reusing the same column for both
> logical and physical slots

Yeah, I'm okay with one column.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  23. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-06T09:16:57Z

    On Wed, Mar 6, 2024 at 2:42 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Tue, Mar 05, 2024 at 01:44:43PM -0600, Nathan Bossart wrote:
> > On Wed, Mar 06, 2024 at 12:50:38AM +0530, Bharath Rupireddy wrote:
> > > On Mon, Mar 4, 2024 at 2:11 PM Bertrand Drouvot
> > > <bertranddrouvot.pg@gmail.com> wrote:
> > >> On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote:
> > >> > Unless I am misinterpreting some details, ISTM we could rename this column
> > >> > to invalidation_reason and use it for both logical and physical slots.  I'm
> > >> > not seeing a strong need for another column.
> > >>
> > >> Yeah having two columns was more for convenience purpose. Without the "conflict"
> > >> one, a slot conflicting with recovery would be "a logical slot having a non NULL
> > >> invalidation_reason".
> > >>
> > >> I'm also fine with one column if most of you prefer that way.
> > >
> > > While we debate on the above, please find the attached v7 patch set
> > > after rebasing.
> >
> > It looks like Bertrand is okay with reusing the same column for both
> > logical and physical slots
>
> Yeah, I'm okay with one column.

Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  24. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-06T10:26:32Z

    Hi,

On Wed, Mar 06, 2024 at 02:46:57PM +0530, Bharath Rupireddy wrote:
> On Wed, Mar 6, 2024 at 2:42 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> > Yeah, I'm okay with one column.
> 
> Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.

Thanks!

A few comments:

1 ===

+       The reason for the slot's invalidation. <literal>NULL</literal> if the
+       slot is currently actively being used.

s/currently actively being used/not invalidated/ ? (I mean it could be valid
and not being used).

2 ===

+       the slot is marked as invalidated. In case of logical slots, it
+       represents the reason for the logical slot's conflict with recovery.

s/the reason for the logical slot's conflict with recovery./the recovery conflict reason./ ?

3 ===

@@ -667,13 +667,13 @@ get_old_cluster_logical_slot_infos(DbInfo *dbinfo, bool live_check)
         * removed.
         */
        res = executeQueryOrDie(conn, "SELECT slot_name, plugin, two_phase, failover, "
-                                                       "%s as caught_up, conflict_reason IS NOT NULL as invalid "
+                                                       "%s as caught_up, invalidation_reason IS NOT NULL as invalid "
                                                        "FROM pg_catalog.pg_replication_slots "
                                                        "WHERE slot_type = 'logical' AND "
                                                        "database = current_database() AND "
                                                        "temporary IS FALSE;",
                                                        live_check ? "FALSE" :
-                                                       "(CASE WHEN conflict_reason IS NOT NULL THEN FALSE "
+                                                       "(CASE WHEN invalidation_reason IS NOT NULL THEN FALSE "

Yeah that's fine because there is logical slot filtering here.

4 ===

-GetSlotInvalidationCause(const char *conflict_reason)
+GetSlotInvalidationCause(const char *invalidation_reason)

Should we change the comment "Maps a conflict reason" above this function?

5 ===

-# Check conflict_reason is NULL for physical slot
+# Check invalidation_reason is NULL for physical slot
 $res = $node_primary->safe_psql(
        'postgres', qq[
-                SELECT conflict_reason is null FROM pg_replication_slots where slot_name = '$primary_slotname';]
+                SELECT invalidation_reason is null FROM pg_replication_slots where slot_name = '$primary_slotname';]
 );


I don't think this test is needed anymore: it does not make that much sense since
it's done after the primary database initialization and startup.

6 ===

@@ -680,7 +680,7 @@ ok( $node_standby->poll_query_until(
 is( $node_standby->safe_psql(
                'postgres',
                q[select bool_or(conflicting) from
-                 (select conflict_reason is not NULL as conflicting
+                 (select invalidation_reason is not NULL as conflicting
                   from pg_replication_slots WHERE slot_type = 'logical')]),
        'f',
        'Logical slots are reported as non conflicting');

What about?

"
# Verify slots are reported as valid in pg_replication_slots
is( $node_standby->safe_psql(
        'postgres',
        q[select bool_or(invalidated) from
          (select invalidation_reason is not NULL as invalidated
           from pg_replication_slots WHERE slot_type = 'logical')]),
    'f',
    'Logical slots are reported as valid');
"

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  25. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-06T10:58:27Z

    On Mon, Mar 4, 2024 at 3:14 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Sun, Mar 03, 2024 at 11:40:00PM +0530, Bharath Rupireddy wrote:
> > On Sat, Mar 2, 2024 at 3:41 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
> >> Would you ever see "conflict" as false and "invalidation_reason" as
> >> non-null for a logical slot?
> >
> > No. Because both conflict and invalidation_reason are decided based on
> > the invalidation reason i.e. value of slot_contents.data.invalidated.
> > IOW, a logical slot that reports conflict as true must have been
> > invalidated.
> >
> > Do you have any thoughts on reverting 007693f and introducing
> > invalidation_reason?
>
> Unless I am misinterpreting some details, ISTM we could rename this column
> to invalidation_reason and use it for both logical and physical slots.  I'm
> not seeing a strong need for another column.  Perhaps I am missing
> something...
>

IIUC, the current conflict_reason is primarily used to determine
logical slots on standby that got invalidated due to recovery time
conflict. On the primary, it will also show logical slots that got
invalidated due to the corresponding WAL got removed. Is that
understanding correct? If so, we are already sort of overloading this
column. However, now adding more invalidation reasons that won't
happen during recovery conflict handling will change entirely the
purpose (as per the name we use) of this variable. I think
invalidation_reason could depict this column correctly but OTOH I
guess it would lose its original meaning/purpose.

-- 
With Regards,
Amit Kapila.

  26. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-06T11:19:04Z

    On Wed, Mar 6, 2024 at 2:47 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
>
> Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.
>

@@ -1629,6 +1634,20 @@
InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
  }
  }
  break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ if (s->data.last_inactive_at > 0)
+ {
+ TimestampTz now;
+
+ Assert(s->data.persistency == RS_PERSISTENT);
+ Assert(s->active_pid == 0);
+
+ now = GetCurrentTimestamp();
+ if (TimestampDifferenceExceeds(s->data.last_inactive_at, now,
+    inactive_replication_slot_timeout * 1000))

You might want to consider its interaction with sync slots on standby.
Say, there is no activity on slots in terms of processing the changes
for slots. Now, we won't perform sync of such slots on standby showing
them inactive as per your new criteria where as same slots could still
be valid on primary as the walsender is still active. This may be more
of a theoretical point as in running system there will probably be
some activity but I think this needs some thougths.


-- 
With Regards,
Amit Kapila.

  27. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-08T14:38:19Z

    On Wed, Mar 6, 2024 at 4:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> IIUC, the current conflict_reason is primarily used to determine
> logical slots on standby that got invalidated due to recovery time
> conflict. On the primary, it will also show logical slots that got
> invalidated due to the corresponding WAL got removed. Is that
> understanding correct?

That's right.

> If so, we are already sort of overloading this
> column. However, now adding more invalidation reasons that won't
> happen during recovery conflict handling will change entirely the
> purpose (as per the name we use) of this variable. I think
> invalidation_reason could depict this column correctly but OTOH I
> guess it would lose its original meaning/purpose.

Hm. I get the concern. Are you okay with having inavlidation_reason
separately for both logical and physical slots? In such a case,
logical slots that got invalidated on the standby will have duplicate
info in conflict_reason and invalidation_reason, is this fine?

Another idea is to make 'conflict_reason text' as a 'conflicting
boolean' again (revert 007693f2a3), and have 'invalidation_reason
text' for both logical and physical slots. So, whenever 'conflicting'
is true, one can look at invalidation_reason for the reason for
conflict. How does this sound?

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  28. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-08T17:12:20Z

    On Wed, Mar 6, 2024 at 4:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> You might want to consider its interaction with sync slots on standby.
> Say, there is no activity on slots in terms of processing the changes
> for slots. Now, we won't perform sync of such slots on standby showing
> them inactive as per your new criteria where as same slots could still
> be valid on primary as the walsender is still active. This may be more
> of a theoretical point as in running system there will probably be
> some activity but I think this needs some thougths.

I believe the xmin and catalog_xmin of the sync slots on the standby
keep advancing depending on the slots on the primary, no? If yes, the
XID age based invalidation shouldn't be a problem.

I believe there are no walsenders started for the sync slots on the
standbys, right? If yes, the inactive timeout based invalidation also
shouldn't be a problem. Because, the inactive timeouts for a slot are
tracked only for walsenders because they are the ones that typically
hold replication slots for longer durations and for real replication
use. We did a similar thing in a recent commit [1].

Is my understanding right? Do you still see any problems with it?

[1]
commit 7c3fb505b14e86581b6a052075a294c78c91b123
Author: Amit Kapila <akapila@postgresql.org>
Date:   Tue Nov 21 07:59:53 2023 +0530

    Log messages for replication slot acquisition and release.
.........
    Note that these messages are emitted only for walsenders but not for
    backends. This is because walsenders are the ones that typically hold
    replication slots for longer durations, unlike backends which hold them
    for executing replication related functions.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  29. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-11T05:55:57Z

    On Fri, Mar 8, 2024 at 8:08 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 6, 2024 at 4:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > IIUC, the current conflict_reason is primarily used to determine
> > logical slots on standby that got invalidated due to recovery time
> > conflict. On the primary, it will also show logical slots that got
> > invalidated due to the corresponding WAL got removed. Is that
> > understanding correct?
>
> That's right.
>
> > If so, we are already sort of overloading this
> > column. However, now adding more invalidation reasons that won't
> > happen during recovery conflict handling will change entirely the
> > purpose (as per the name we use) of this variable. I think
> > invalidation_reason could depict this column correctly but OTOH I
> > guess it would lose its original meaning/purpose.
>
> Hm. I get the concern. Are you okay with having inavlidation_reason
> separately for both logical and physical slots? In such a case,
> logical slots that got invalidated on the standby will have duplicate
> info in conflict_reason and invalidation_reason, is this fine?
>

If we have duplicate information in two columns that could be
confusing for users. BTW, isn't the recovery conflict occur only
because of rows_removed and wal_level_insufficient reasons? The
wal_removed or the new reasons you are proposing can't happen because
of recovery conflict. Am, I missing something here?

> Another idea is to make 'conflict_reason text' as a 'conflicting
> boolean' again (revert 007693f2a3), and have 'invalidation_reason
> text' for both logical and physical slots. So, whenever 'conflicting'
> is true, one can look at invalidation_reason for the reason for
> conflict. How does this sound?
>

So, does this mean that conflicting will only be true for some of the
reasons (say wal_level_insufficient, rows_removed, wal_removed) and
logical slots but not for others? I think that will also not eliminate
the duplicate information as user could have deduced that from single
column

-- 
With Regards,
Amit Kapila.

  30. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-11T10:14:46Z

    On Fri, Mar 8, 2024 at 10:42 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 6, 2024 at 4:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > You might want to consider its interaction with sync slots on standby.
> > Say, there is no activity on slots in terms of processing the changes
> > for slots. Now, we won't perform sync of such slots on standby showing
> > them inactive as per your new criteria where as same slots could still
> > be valid on primary as the walsender is still active. This may be more
> > of a theoretical point as in running system there will probably be
> > some activity but I think this needs some thougths.
>
> I believe the xmin and catalog_xmin of the sync slots on the standby
> keep advancing depending on the slots on the primary, no? If yes, the
> XID age based invalidation shouldn't be a problem.
>
> I believe there are no walsenders started for the sync slots on the
> standbys, right? If yes, the inactive timeout based invalidation also
> shouldn't be a problem. Because, the inactive timeouts for a slot are
> tracked only for walsenders because they are the ones that typically
> hold replication slots for longer durations and for real replication
> use. We did a similar thing in a recent commit [1].
>
> Is my understanding right?
>

Yes, your understanding is correct. I wanted us to consider having new
parameters like 'inactive_replication_slot_timeout' to be at
slot-level instead of GUC. I think this new parameter doesn't seem to
be the similar as 'max_slot_wal_keep_size' which leads to truncation
of WAL at global and then invalidates the appropriate slots. OTOH, the
'inactive_replication_slot_timeout' doesn't appear to have a similar
global effect. The other thing we should consider is what if the
checkpoint happens at a timeout greater than
'inactive_replication_slot_timeout'? Shall, we consider doing it via
some other background process or do we think checkpointer is the best
we can have?

>
 Do you still see any problems with it?
>

Sorry, I haven't done any detailed review yet so can't say with
confidence whether there is any problem or not w.r.t sync slots.

-- 
With Regards,
Amit Kapila.

  31. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-11T10:39:27Z

    On Wed, Mar 6, 2024 at 2:47 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.
>

Commit message says: "Currently postgres has the ability to invalidate
inactive replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in case
they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a customer
generates, and their allocated storage will vary greatly in
production, making it difficult to pin down a one-size-fits-all value.
It is often easy for developers to set an XID age (age of slot's xmin
or catalog_xmin) of say 1 or 1.5 billion, after which the slots get
invalidated."

I don't see how it will be easier for the user to choose the default
value of 'max_slot_xid_age' compared to 'max_slot_wal_keep_size'. But,
I agree similar to 'max_slot_wal_keep_size', 'max_slot_xid_age' can be
another parameter to allow vacuum to proceed removing the rows which
otherwise it wouldn't have been as those would be required by some
slot. Now, if this understanding is correct, we should probably make
this invalidation happen by (auto)vacuum after computing the age based
on this new parameter.

-- 
With Regards,
Amit Kapila.

  32. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-11T14:13:57Z

    On Mon, Mar 11, 2024 at 04:09:27PM +0530, Amit Kapila wrote:
> I don't see how it will be easier for the user to choose the default
> value of 'max_slot_xid_age' compared to 'max_slot_wal_keep_size'. But,
> I agree similar to 'max_slot_wal_keep_size', 'max_slot_xid_age' can be
> another parameter to allow vacuum to proceed removing the rows which
> otherwise it wouldn't have been as those would be required by some
> slot.

Yeah, the idea is to help prevent transaction ID wraparound, so I would
expect max_slot_xid_age to ordinarily be set relatively high, i.e., 1.5B+.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  33. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-12T07:54:16Z

    Hi,

On Fri, Mar 08, 2024 at 10:42:20PM +0530, Bharath Rupireddy wrote:
> On Wed, Mar 6, 2024 at 4:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > You might want to consider its interaction with sync slots on standby.
> > Say, there is no activity on slots in terms of processing the changes
> > for slots. Now, we won't perform sync of such slots on standby showing
> > them inactive as per your new criteria where as same slots could still
> > be valid on primary as the walsender is still active. This may be more
> > of a theoretical point as in running system there will probably be
> > some activity but I think this needs some thougths.
> 
> I believe the xmin and catalog_xmin of the sync slots on the standby
> keep advancing depending on the slots on the primary, no? If yes, the
> XID age based invalidation shouldn't be a problem.
> 
> I believe there are no walsenders started for the sync slots on the
> standbys, right? If yes, the inactive timeout based invalidation also
> shouldn't be a problem. Because, the inactive timeouts for a slot are
> tracked only for walsenders because they are the ones that typically
> hold replication slots for longer durations and for real replication
> use. We did a similar thing in a recent commit [1].
> 
> Is my understanding right? Do you still see any problems with it?

Would that make sense to "simply" discard/prevent those kind of invalidations
for "synced" slot on standby? I mean, do they make sense given the fact that
those slots are not usable until the standby is promoted?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  34. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-12T12:21:43Z

    On Tue, Mar 12, 2024 at 1:24 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 08, 2024 at 10:42:20PM +0530, Bharath Rupireddy wrote:
> > On Wed, Mar 6, 2024 at 4:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > You might want to consider its interaction with sync slots on standby.
> > > Say, there is no activity on slots in terms of processing the changes
> > > for slots. Now, we won't perform sync of such slots on standby showing
> > > them inactive as per your new criteria where as same slots could still
> > > be valid on primary as the walsender is still active. This may be more
> > > of a theoretical point as in running system there will probably be
> > > some activity but I think this needs some thougths.
> >
> > I believe the xmin and catalog_xmin of the sync slots on the standby
> > keep advancing depending on the slots on the primary, no? If yes, the
> > XID age based invalidation shouldn't be a problem.
> >
> > I believe there are no walsenders started for the sync slots on the
> > standbys, right? If yes, the inactive timeout based invalidation also
> > shouldn't be a problem. Because, the inactive timeouts for a slot are
> > tracked only for walsenders because they are the ones that typically
> > hold replication slots for longer durations and for real replication
> > use. We did a similar thing in a recent commit [1].
> >
> > Is my understanding right? Do you still see any problems with it?
>
> Would that make sense to "simply" discard/prevent those kind of invalidations
> for "synced" slot on standby? I mean, do they make sense given the fact that
> those slots are not usable until the standby is promoted?
>

AFAIR, we don't prevent similar invalidations due to
'max_slot_wal_keep_size' for sync slots, so why to prevent it for
these new parameters? This will unnecessarily create inconsistency in
the invalidation behavior.

-- 
With Regards,
Amit Kapila.

  35. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-12T15:25:37Z

    On Mon, Mar 11, 2024 at 11:26 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > Hm. I get the concern. Are you okay with having inavlidation_reason
> > separately for both logical and physical slots? In such a case,
> > logical slots that got invalidated on the standby will have duplicate
> > info in conflict_reason and invalidation_reason, is this fine?
> >
>
> If we have duplicate information in two columns that could be
> confusing for users. BTW, isn't the recovery conflict occur only
> because of rows_removed and wal_level_insufficient reasons? The
> wal_removed or the new reasons you are proposing can't happen because
> of recovery conflict. Am, I missing something here?

My understanding aligns with yours that the rows_removed and
wal_level_insufficient invalidations can occur only upon recovery
conflict.

FWIW, a test named 'synchronized slot has been invalidated' in
040_standby_failover_slots_sync.pl inappropriately uses
conflict_reason = 'wal_removed' logical slot on standby. As per the
above understanding, it's inappropriate to use conflict_reason here
because wal_removed invalidation doesn't conflict with recovery.

> > Another idea is to make 'conflict_reason text' as a 'conflicting
> > boolean' again (revert 007693f2a3), and have 'invalidation_reason
> > text' for both logical and physical slots. So, whenever 'conflicting'
> > is true, one can look at invalidation_reason for the reason for
> > conflict. How does this sound?
> >
>
> So, does this mean that conflicting will only be true for some of the
> reasons (say wal_level_insufficient, rows_removed, wal_removed) and
> logical slots but not for others? I think that will also not eliminate
> the duplicate information as user could have deduced that from single
> column.

So, how about we turn conflict_reason to only report the reasons that
actually cause conflict with recovery for logical slots, something
like below, and then have invalidation_cause as a generic column for
all sorts of invalidation reasons for both logical and physical slots?

ReplicationSlotInvalidationCause cause = slot_contents.data.invalidated;

if (slot_contents.data.database == InvalidOid ||
    cause == RS_INVAL_NONE ||
    cause != RS_INVAL_HORIZON ||
    cause != RS_INVAL_WAL_LEVEL)
{
    nulls[i++] = true;
}
else
{
    Assert(cause == RS_INVAL_HORIZON || cause == RS_INVAL_WAL_LEVEL);

    values[i++] = CStringGetTextDatum(SlotInvalidationCauses[cause]);
}

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  36. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-12T15:40:59Z

    Hi,

On Tue, Mar 12, 2024 at 05:51:43PM +0530, Amit Kapila wrote:
> On Tue, Mar 12, 2024 at 1:24 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Fri, Mar 08, 2024 at 10:42:20PM +0530, Bharath Rupireddy wrote:
> > > On Wed, Mar 6, 2024 at 4:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > You might want to consider its interaction with sync slots on standby.
> > > > Say, there is no activity on slots in terms of processing the changes
> > > > for slots. Now, we won't perform sync of such slots on standby showing
> > > > them inactive as per your new criteria where as same slots could still
> > > > be valid on primary as the walsender is still active. This may be more
> > > > of a theoretical point as in running system there will probably be
> > > > some activity but I think this needs some thougths.
> > >
> > > I believe the xmin and catalog_xmin of the sync slots on the standby
> > > keep advancing depending on the slots on the primary, no? If yes, the
> > > XID age based invalidation shouldn't be a problem.
> > >
> > > I believe there are no walsenders started for the sync slots on the
> > > standbys, right? If yes, the inactive timeout based invalidation also
> > > shouldn't be a problem. Because, the inactive timeouts for a slot are
> > > tracked only for walsenders because they are the ones that typically
> > > hold replication slots for longer durations and for real replication
> > > use. We did a similar thing in a recent commit [1].
> > >
> > > Is my understanding right? Do you still see any problems with it?
> >
> > Would that make sense to "simply" discard/prevent those kind of invalidations
> > for "synced" slot on standby? I mean, do they make sense given the fact that
> > those slots are not usable until the standby is promoted?
> >
> 
> AFAIR, we don't prevent similar invalidations due to
> 'max_slot_wal_keep_size' for sync slots,

Right, we'd invalidate them on the standby should the standby sync slot restart_lsn
exceeds the limit.

> so why to prevent it for
> these new parameters? This will unnecessarily create inconsistency in
> the invalidation behavior.

Yeah, but I think wal removal has a direct impact on the slot usuability which
is probably not the case with the new XID and Timeout ones. That's why I thought
about handling them differently (but I'm also fine if that's not the case).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  37. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-12T15:44:40Z

    On Tue, Mar 12, 2024 at 5:51 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > Would that make sense to "simply" discard/prevent those kind of invalidations
> > for "synced" slot on standby? I mean, do they make sense given the fact that
> > those slots are not usable until the standby is promoted?
>
> AFAIR, we don't prevent similar invalidations due to
> 'max_slot_wal_keep_size' for sync slots, so why to prevent it for
> these new parameters? This will unnecessarily create inconsistency in
> the invalidation behavior.

Right. +1 to keep the behaviour consistent for all invalidations.
However, an assertion that inactive_timeout isn't set for synced slots
on the standby isn't a bad idea because we rely on the fact that
walsenders aren't started for synced slots. Again, I think it misses
the consistency in the invalidation behaviour.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  38. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-12T15:49:35Z

    On Tue, Mar 12, 2024 at 9:11 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > AFAIR, we don't prevent similar invalidations due to
> > 'max_slot_wal_keep_size' for sync slots,
>
> Right, we'd invalidate them on the standby should the standby sync slot restart_lsn
> exceeds the limit.

Right. Help me understand this a bit - is the wal_removed invalidation
going to conflict with recovery on the standby?

Per the discussion upthread, I'm trying to understand what
invalidation reasons will exactly cause conflict with recovery? Is it
just rows_removed and wal_level_insufficient invalidations? My
understanding on the conflict with recovery and invalidation reason
has been a bit off track. Perhaps, we need to clarify these two things
in the docs for the end users as well?

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  39. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-12T16:39:48Z

    On Mon, Mar 11, 2024 at 3:44 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Yes, your understanding is correct. I wanted us to consider having new
> parameters like 'inactive_replication_slot_timeout' to be at
> slot-level instead of GUC. I think this new parameter doesn't seem to
> be the similar as 'max_slot_wal_keep_size' which leads to truncation
> of WAL at global and then invalidates the appropriate slots. OTOH, the
> 'inactive_replication_slot_timeout' doesn't appear to have a similar
> global effect.

last_inactive_at is tracked for each slot using which slots get
invalidated based on inactive_replication_slot_timeout. It's like
max_slot_wal_keep_size invalidating slots based on restart_lsn. In a
way, both are similar, right?

> The other thing we should consider is what if the
> checkpoint happens at a timeout greater than
> 'inactive_replication_slot_timeout'?

In such a case, the slots get invalidated upon the next checkpoint as
the (current_checkpointer_timeout - last_inactive_at) will then be
greater than inactive_replication_slot_timeout.

> Shall, we consider doing it via
> some other background process or do we think checkpointer is the best
> we can have?

The same problem exists if we do it with some other background
process. I think the checkpointer is best because it already
invalidates slots for wal_removed cause, and flushes all replication
slots to disk. Moving this new invalidation functionality into some
other background process such as autovacuum will not only burden that
process' work but also mix up the unique functionality of that
background process.

Having said above, I'm open to ideas from others as I'm not so sure if
there's any issue with checkpointer invalidating the slots for new
reasons.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  40. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-12T17:21:49Z

    On Mon, Mar 11, 2024 at 4:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> I don't see how it will be easier for the user to choose the default
> value of 'max_slot_xid_age' compared to 'max_slot_wal_keep_size'. But,
> I agree similar to 'max_slot_wal_keep_size', 'max_slot_xid_age' can be
> another parameter to allow vacuum to proceed removing the rows which
> otherwise it wouldn't have been as those would be required by some
> slot. Now, if this understanding is correct, we should probably make
> this invalidation happen by (auto)vacuum after computing the age based
> on this new parameter.

Currently, the patch computes the XID age in the checkpointer using
the next XID (gets from ReadNextFullTransactionId()) and slot's xmin
and catalog_xmin. I think the checkpointer is best because it already
invalidates slots for wal_removed cause, and flushes all replication
slots to disk. Moving this new invalidation functionality into some
other background process such as autovacuum will not only burden that
process' work but also mix up the unique functionality of that
background process.

Having said above, I'm open to ideas from others as I'm not so sure if
there's any issue with checkpointer invalidating the slots for new
reasons.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  41. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-13T03:51:32Z

    On Tue, Mar 12, 2024 at 8:55 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Mon, Mar 11, 2024 at 11:26 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > Hm. I get the concern. Are you okay with having inavlidation_reason
> > > separately for both logical and physical slots? In such a case,
> > > logical slots that got invalidated on the standby will have duplicate
> > > info in conflict_reason and invalidation_reason, is this fine?
> > >
> >
> > If we have duplicate information in two columns that could be
> > confusing for users. BTW, isn't the recovery conflict occur only
> > because of rows_removed and wal_level_insufficient reasons? The
> > wal_removed or the new reasons you are proposing can't happen because
> > of recovery conflict. Am, I missing something here?
>
> My understanding aligns with yours that the rows_removed and
> wal_level_insufficient invalidations can occur only upon recovery
> conflict.
>
> FWIW, a test named 'synchronized slot has been invalidated' in
> 040_standby_failover_slots_sync.pl inappropriately uses
> conflict_reason = 'wal_removed' logical slot on standby. As per the
> above understanding, it's inappropriate to use conflict_reason here
> because wal_removed invalidation doesn't conflict with recovery.
>
> > > Another idea is to make 'conflict_reason text' as a 'conflicting
> > > boolean' again (revert 007693f2a3), and have 'invalidation_reason
> > > text' for both logical and physical slots. So, whenever 'conflicting'
> > > is true, one can look at invalidation_reason for the reason for
> > > conflict. How does this sound?
> > >
> >
> > So, does this mean that conflicting will only be true for some of the
> > reasons (say wal_level_insufficient, rows_removed, wal_removed) and
> > logical slots but not for others? I think that will also not eliminate
> > the duplicate information as user could have deduced that from single
> > column.
>
> So, how about we turn conflict_reason to only report the reasons that
> actually cause conflict with recovery for logical slots, something
> like below, and then have invalidation_cause as a generic column for
> all sorts of invalidation reasons for both logical and physical slots?
>

If our above understanding is correct then coflict_reason will be a
subset of invalidation_reason. If so, whatever way we arrange this
information, there will be some sort of duplicity unless we just have
one column 'invalidation_reason' and update the docs to interpret it
correctly for conflicts.

-- 
With Regards,
Amit Kapila.

  42. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-13T04:08:18Z

    On Tue, Mar 12, 2024 at 9:11 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Tue, Mar 12, 2024 at 05:51:43PM +0530, Amit Kapila wrote:
> > On Tue, Mar 12, 2024 at 1:24 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
>
> > so why to prevent it for
> > these new parameters? This will unnecessarily create inconsistency in
> > the invalidation behavior.
>
> Yeah, but I think wal removal has a direct impact on the slot usuability which
> is probably not the case with the new XID and Timeout ones.
>

BTW, is XID the based parameter 'max_slot_xid_age' not have similarity
with 'max_slot_wal_keep_size'? I think it will impact the rows we
removed based on xid horizons. Don't we need to consider it while
vacuum computing the xid horizons in ComputeXidHorizons() similar to
what we do for WAL w.r.t 'max_slot_wal_keep_size'?

-- 
With Regards,
Amit Kapila.

  43. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-13T04:24:15Z

    On Tue, Mar 12, 2024 at 10:10 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Mon, Mar 11, 2024 at 3:44 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Yes, your understanding is correct. I wanted us to consider having new
> > parameters like 'inactive_replication_slot_timeout' to be at
> > slot-level instead of GUC. I think this new parameter doesn't seem to
> > be the similar as 'max_slot_wal_keep_size' which leads to truncation
> > of WAL at global and then invalidates the appropriate slots. OTOH, the
> > 'inactive_replication_slot_timeout' doesn't appear to have a similar
> > global effect.
>
> last_inactive_at is tracked for each slot using which slots get
> invalidated based on inactive_replication_slot_timeout. It's like
> max_slot_wal_keep_size invalidating slots based on restart_lsn. In a
> way, both are similar, right?
>

There is some similarity but 'max_slot_wal_keep_size' leads to
truncation of WAL which in turn leads to invalidation of slots. Here,
I am also trying to be cautious in adding a GUC unless it is required
or having a slot-level parameter doesn't serve the need. Having said
that, I see that there is an argument that we should follow the path
of 'max_slot_wal_keep_size' GUC and there is some value to it but
still I think avoiding a new GUC for inactivity in the slot would
outweigh.

-- 
With Regards,
Amit Kapila.

  44. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-13T05:43:18Z

    On Wed, Mar 6, 2024 at 2:47 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 6, 2024 at 2:42 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Hi,
> >
> > On Tue, Mar 05, 2024 at 01:44:43PM -0600, Nathan Bossart wrote:
> > > On Wed, Mar 06, 2024 at 12:50:38AM +0530, Bharath Rupireddy wrote:
> > > > On Mon, Mar 4, 2024 at 2:11 PM Bertrand Drouvot
> > > > <bertranddrouvot.pg@gmail.com> wrote:
> > > >> On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote:
> > > >> > Unless I am misinterpreting some details, ISTM we could rename this column
> > > >> > to invalidation_reason and use it for both logical and physical slots.  I'm
> > > >> > not seeing a strong need for another column.
> > > >>
> > > >> Yeah having two columns was more for convenience purpose. Without the "conflict"
> > > >> one, a slot conflicting with recovery would be "a logical slot having a non NULL
> > > >> invalidation_reason".
> > > >>
> > > >> I'm also fine with one column if most of you prefer that way.
> > > >
> > > > While we debate on the above, please find the attached v7 patch set
> > > > after rebasing.
> > >
> > > It looks like Bertrand is okay with reusing the same column for both
> > > logical and physical slots
> >
> > Yeah, I'm okay with one column.
>
> Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.

JFYI, the patch does not apply to the head. There is a conflict in
multiple files.

thanks
Shveta

  45. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-13T07:15:06Z

    > JFYI, the patch does not apply to the head. There is a conflict in
> multiple files.

For review purposes, I applied v8 to the March 6 code-base. I have yet
to review in detail, please find my initial thoughts:

1)
I found that 'inactive_replication_slot_timeout' works only if there
was any walsender ever started for that slot . The logic is under
'am_walsender' check.  Is this intentional?
If I create a slot and use only pg_logical_slot_get_changes or
pg_replication_slot_advance on it, it never gets invalidated due to
timeout.  While, when I set 'max_slot_xid_age' or say
'max_slot_wal_keep_size' to a lower value, the said slot is
invalidated correctly with 'xid_aged' and 'wal_removed' reasons
respectively.

Example:
With inactive_replication_slot_timeout=1min, test1_3  is the slot for
which there is no walsender and only advance and get_changes SQL
functions were called; test1_4 is the one for which pg_recvlogical was
run for a second.

 test1_3     |   785 | | reserved   |                           | t
    |                                  |
 test1_4     |   798 | | lost       | inactive_timeout    | t        |
2024-03-13 11:52:41.58446+05:30  |

And when inactive_replication_slot_timeout=0  and max_slot_xid_age=10

 test1_3     |   785 | | lost       |          xid_aged       | t
  |                                  |
 test1_4     |   798 | | lost       | inactive_timeout    | t        |
2024-03-13 11:52:41.58446+05:30  |


2)
The msg for patch 3 says:
--------------
a) when replication slots is lying inactive for a day or so using
last_inactive_at metric,
b) when a replication slot is becoming inactive too frequently using
last_inactive_at metric.
--------------
 I think in b, you want to refer to inactive_count instead of last_inactive_at?

3)
I do not see invalidation_reason updated for 2 new reasons in system-views.sgml


thanks
Shveta

  46. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-13T07:21:18Z

    Hi,

On Tue, Mar 12, 2024 at 09:19:35PM +0530, Bharath Rupireddy wrote:
> On Tue, Mar 12, 2024 at 9:11 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > AFAIR, we don't prevent similar invalidations due to
> > > 'max_slot_wal_keep_size' for sync slots,
> >
> > Right, we'd invalidate them on the standby should the standby sync slot restart_lsn
> > exceeds the limit.
> 
> Right. Help me understand this a bit - is the wal_removed invalidation
> going to conflict with recovery on the standby?

I don't think so, as it's not directly related to recovery. The slot will
be invalided on the standby though.

> Per the discussion upthread, I'm trying to understand what
> invalidation reasons will exactly cause conflict with recovery? Is it
> just rows_removed and wal_level_insufficient invalidations? 

Yes, that's the ones added in be87200efd.

See the error messages on a standby:

== wal removal

postgres=#  SELECT * FROM pg_logical_slot_get_changes('lsub4_slot', NULL, NULL, 'include-xids', '0');
ERROR:  can no longer get changes from replication slot "lsub4_slot"
DETAIL:  This slot has been invalidated because it exceeded the maximum reserved size.

== wal level

postgres=# select conflict_reason from pg_replication_slots where slot_name = 'lsub5_slot';;
    conflict_reason
------------------------
 wal_level_insufficient
(1 row)

postgres=#  SELECT * FROM pg_logical_slot_get_changes('lsub5_slot', NULL, NULL, 'include-xids', '0');
ERROR:  can no longer get changes from replication slot "lsub5_slot"
DETAIL:  This slot has been invalidated because it was conflicting with recovery.

== rows removal

postgres=# select conflict_reason from pg_replication_slots where slot_name = 'lsub6_slot';;
 conflict_reason
-----------------
 rows_removed
(1 row)

postgres=#  SELECT * FROM pg_logical_slot_get_changes('lsub6_slot', NULL, NULL, 'include-xids', '0');
ERROR:  can no longer get changes from replication slot "lsub6_slot"
DETAIL:  This slot has been invalidated because it was conflicting with recovery.

As you can see, only wal level and rows removal are mentioning conflict with
recovery.

So, are we already "wrong" mentioning "wal_removed" in conflict_reason?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  47. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-13T09:15:14Z

    On Fri, Mar 8, 2024 at 10:42 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 6, 2024 at 4:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > You might want to consider its interaction with sync slots on standby.
> > Say, there is no activity on slots in terms of processing the changes
> > for slots. Now, we won't perform sync of such slots on standby showing
> > them inactive as per your new criteria where as same slots could still
> > be valid on primary as the walsender is still active. This may be more
> > of a theoretical point as in running system there will probably be
> > some activity but I think this needs some thougths.
>
> I believe the xmin and catalog_xmin of the sync slots on the standby
> keep advancing depending on the slots on the primary, no? If yes, the
> XID age based invalidation shouldn't be a problem.

If the user has not enabled slot-sync worker and is relying on the SQL
function pg_sync_replication_slots(), then the xmin and catalog_xmin
of synced slots may not keep on advancing. These will be advanced only
on next run of function. But meanwhile the synced slots may be
invalidated due to 'xid_aged'.  Then the next time, when user runs
pg_sync_replication_slots() again, the invalidated slots will be
dropped and will be recreated by this SQL function (provided they are
valid on primary and are invalidated on standby alone).  I am not
stating that it is a problem, but we need to think if this is what we
want. Secondly, the behaviour is not same with 'inactive_timeout'
invalidation. Synced slots are immune to 'inactive_timeout'
invalidation as this invalidation happens only in walsender, while
these are not immune to 'xid_aged' invalidation. So again, needs some
thoughts here.

> I believe there are no walsenders started for the sync slots on the
> standbys, right? If yes, the inactive timeout based invalidation also
> shouldn't be a problem. Because, the inactive timeouts for a slot are
> tracked only for walsenders because they are the ones that typically
> hold replication slots for longer durations and for real replication
> use. We did a similar thing in a recent commit [1].
>
> Is my understanding right? Do you still see any problems with it?

I have explained the situation above for us to think over it better.

thanks
Shveta

  48. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-13T15:54:17Z

    On Wed, Mar 13, 2024 at 9:21 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > So, how about we turn conflict_reason to only report the reasons that
> > actually cause conflict with recovery for logical slots, something
> > like below, and then have invalidation_cause as a generic column for
> > all sorts of invalidation reasons for both logical and physical slots?
>
> If our above understanding is correct then coflict_reason will be a
> subset of invalidation_reason. If so, whatever way we arrange this
> information, there will be some sort of duplicity unless we just have
> one column 'invalidation_reason' and update the docs to interpret it
> correctly for conflicts.

Yes, there will be some sort of duplicity if we emit conflict_reason
as a text field. However, I still think the better way is to turn
conflict_reason text to conflict boolean and set it to true only on
rows_removed and wal_level_insufficient invalidations. When conflict
boolean is true, one (including all the tests that we've added
recently) can look for invalidation_reason text field for the reason.
This sounds reasonable to me as opposed to we just mentioning in the
docs that "if invalidation_reason is rows_removed or
wal_level_insufficient it's the reason for conflict with recovery".

Thoughts?

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  49. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-13T16:36:27Z

    On Wed, Mar 13, 2024 at 12:51 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> See the error messages on a standby:
>
> == wal removal
>
> postgres=#  SELECT * FROM pg_logical_slot_get_changes('lsub4_slot', NULL, NULL, 'include-xids', '0');
> ERROR:  can no longer get changes from replication slot "lsub4_slot"
> DETAIL:  This slot has been invalidated because it exceeded the maximum reserved size.
>
> == wal level
>
> postgres=# select conflict_reason from pg_replication_slots where slot_name = 'lsub5_slot';;
>     conflict_reason
> ------------------------
>  wal_level_insufficient
> (1 row)
>
> postgres=#  SELECT * FROM pg_logical_slot_get_changes('lsub5_slot', NULL, NULL, 'include-xids', '0');
> ERROR:  can no longer get changes from replication slot "lsub5_slot"
> DETAIL:  This slot has been invalidated because it was conflicting with recovery.
>
> == rows removal
>
> postgres=# select conflict_reason from pg_replication_slots where slot_name = 'lsub6_slot';;
>  conflict_reason
> -----------------
>  rows_removed
> (1 row)
>
> postgres=#  SELECT * FROM pg_logical_slot_get_changes('lsub6_slot', NULL, NULL, 'include-xids', '0');
> ERROR:  can no longer get changes from replication slot "lsub6_slot"
> DETAIL:  This slot has been invalidated because it was conflicting with recovery.
>
> As you can see, only wal level and rows removal are mentioning conflict with
> recovery.
>
> So, are we already "wrong" mentioning "wal_removed" in conflict_reason?

It looks like yes. So, how about we fix it the way proposed here -
https://www.postgresql.org/message-id/CALj2ACVd_dizYQiZwwUfsb%2BhG-fhGYo_kEDq0wn_vNwQvOrZHg%40mail.gmail.com?

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  50. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-13T16:46:13Z

    On Wed, Mar 13, 2024 at 11:13 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> > Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.
>
> JFYI, the patch does not apply to the head. There is a conflict in
> multiple files.

Thanks for looking into this. I noticed that the v8 patches needed
rebase. Before I go do anything with the patches, I'm trying to gain
consensus on the design. Following is the summary of design choices
we've discussed so far:
1) conflict_reason vs invalidation_reason.
2) When to compute the XID age?
3) Where to do the invalidations? Is it in the checkpointer or
autovacuum or some other process?
4) Interaction of these new invalidations with sync slots on the standby.

I hope to get on to these one after the other.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  51. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-14T06:54:00Z

    On Wed, Mar 13, 2024 at 9:24 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 13, 2024 at 9:21 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > So, how about we turn conflict_reason to only report the reasons that
> > > actually cause conflict with recovery for logical slots, something
> > > like below, and then have invalidation_cause as a generic column for
> > > all sorts of invalidation reasons for both logical and physical slots?
> >
> > If our above understanding is correct then coflict_reason will be a
> > subset of invalidation_reason. If so, whatever way we arrange this
> > information, there will be some sort of duplicity unless we just have
> > one column 'invalidation_reason' and update the docs to interpret it
> > correctly for conflicts.
>
> Yes, there will be some sort of duplicity if we emit conflict_reason
> as a text field. However, I still think the better way is to turn
> conflict_reason text to conflict boolean and set it to true only on
> rows_removed and wal_level_insufficient invalidations. When conflict
> boolean is true, one (including all the tests that we've added
> recently) can look for invalidation_reason text field for the reason.
> This sounds reasonable to me as opposed to we just mentioning in the
> docs that "if invalidation_reason is rows_removed or
> wal_level_insufficient it's the reason for conflict with recovery".
>

Fair point. I think we can go either way. Bertrand, Nathan, and
others, do you have an opinion on this matter?

-- 
With Regards,
Amit Kapila.

  52. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-14T06:57:26Z

    On Wed, Mar 13, 2024 at 10:16 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 13, 2024 at 11:13 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.
> >
> > JFYI, the patch does not apply to the head. There is a conflict in
> > multiple files.
>
> Thanks for looking into this. I noticed that the v8 patches needed
> rebase. Before I go do anything with the patches, I'm trying to gain
> consensus on the design. Following is the summary of design choices
> we've discussed so far:
> 1) conflict_reason vs invalidation_reason.
> 2) When to compute the XID age?
>

I feel we should focus on two things (a) one is to introduce a new
column invalidation_reason, and (b) let's try to first complete
invalidation due to timeout. We can look into XID stuff if time
permits, remember, we don't have ample time left.

With Regards,
Amit Kapila.

  53. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-14T14:27:46Z

    On Thu, Mar 14, 2024 at 12:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Mar 13, 2024 at 9:24 PM Bharath Rupireddy
> >
> > Yes, there will be some sort of duplicity if we emit conflict_reason
> > as a text field. However, I still think the better way is to turn
> > conflict_reason text to conflict boolean and set it to true only on
> > rows_removed and wal_level_insufficient invalidations. When conflict
> > boolean is true, one (including all the tests that we've added
> > recently) can look for invalidation_reason text field for the reason.
> > This sounds reasonable to me as opposed to we just mentioning in the
> > docs that "if invalidation_reason is rows_removed or
> > wal_level_insufficient it's the reason for conflict with recovery".
> >
> Fair point. I think we can go either way. Bertrand, Nathan, and
> others, do you have an opinion on this matter?

While we wait to hear from others on this, I'm attaching the v9 patch
set implementing the above idea (check 0001 patch). Please have a
look. I'll come back to the other review comments soon.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  54. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-15T04:44:49Z

    On Thu, Mar 14, 2024 at 7:58 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Thu, Mar 14, 2024 at 12:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Mar 13, 2024 at 9:24 PM Bharath Rupireddy
> > >
> > > Yes, there will be some sort of duplicity if we emit conflict_reason
> > > as a text field. However, I still think the better way is to turn
> > > conflict_reason text to conflict boolean and set it to true only on
> > > rows_removed and wal_level_insufficient invalidations. When conflict
> > > boolean is true, one (including all the tests that we've added
> > > recently) can look for invalidation_reason text field for the reason.
> > > This sounds reasonable to me as opposed to we just mentioning in the
> > > docs that "if invalidation_reason is rows_removed or
> > > wal_level_insufficient it's the reason for conflict with recovery".

+1 on maintaining both conflicting and invalidation_reason

> > Fair point. I think we can go either way. Bertrand, Nathan, and
> > others, do you have an opinion on this matter?
>
> While we wait to hear from others on this, I'm attaching the v9 patch
> set implementing the above idea (check 0001 patch). Please have a
> look. I'll come back to the other review comments soon.

Thanks for the patch. JFYI, patch09 does not apply to HEAD, some
recent commit caused the conflict.

Some trivial comments on patch001 (yet to review other patches)

1)
info.c:

- "%s as caught_up, conflict_reason IS NOT NULL as invalid "
+ "%s as caught_up, invalidation_reason IS NOT NULL as invalid "

Can we revert back to 'conflicting as invalid' since it is a query for
logical slots only.

2)
040_standby_failover_slots_sync.pl:

- q{SELECT conflict_reason IS NULL AND synced AND NOT temporary FROM
pg_replication_slots WHERE slot_name = 'lsub1_slot';}
+ q{SELECT invalidation_reason IS NULL AND synced AND NOT temporary
FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';}

Here too, can we have 'NOT conflicting' instead of '
invalidation_reason IS NULL' as it is a logical slot test.

thanks
Shveta

  55. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-15T05:14:55Z

    On Wed, Mar 13, 2024 at 9:38 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> BTW, is XID the based parameter 'max_slot_xid_age' not have similarity
> with 'max_slot_wal_keep_size'? I think it will impact the rows we
> removed based on xid horizons. Don't we need to consider it while
> vacuum computing the xid horizons in ComputeXidHorizons() similar to
> what we do for WAL w.r.t 'max_slot_wal_keep_size'?

I'm having a hard time understanding why we'd need something up there
in ComputeXidHorizons(). Can you elaborate it a bit please?

What's proposed with max_slot_xid_age is that during checkpoint we
look at slot's xmin and catalog_xmin, and the current system txn id.
Then, if the XID age of (xmin, catalog_xmin) and current_xid crosses
max_slot_xid_age, we invalidate the slot.  Let me illustrate how all
this works:

1. Setup a primary and standby with hot_standby_feedback set to on on
standby. For instance, check my scripts at [1].

2. Stop the standby to make the slot inactive on the primary. Check
the slot is holding xmin of 738.
./pg_ctl -D sbdata -l logfilesbdata stop

postgres=# SELECT * FROM pg_replication_slots;
-[ RECORD 1 ]-------+-------------
slot_name           | sb_repl_slot
plugin              |
slot_type           | physical
datoid              |
database            |
temporary           | f
active              | f
active_pid          |
xmin                | 738
catalog_xmin        |
restart_lsn         | 0/3000000
confirmed_flush_lsn |
wal_status          | reserved
safe_wal_size       |
two_phase           | f
conflict_reason     |
failover            | f
synced              | f

3. Start consuming the XIDs on the primary with the following script
for instance
./psql -d postgres -p 5432
DROP TABLE tab_int;
CREATE TABLE tab_int (a int);

do $$
begin
  for i in 1..268435 loop
    -- use an exception block so that each iteration eats an XID
    begin
      insert into tab_int values (i);
    exception
      when division_by_zero then null;
    end;
  end loop;
end$$;

4. Make some dead rows in the table.
update tab_int set a = a+1;
delete from tab_int where a%4=0;

postgres=# SELECT n_dead_tup, n_tup_ins, n_tup_upd, n_tup_del FROM
pg_stat_user_tables WHERE relname = 'tab_int';
-[ RECORD 1 ]------
n_dead_tup | 335544
n_tup_ins  | 268435
n_tup_upd  | 268435
n_tup_del  | 67109

5. Try vacuuming to delete the dead rows, observe 'tuples: 0 removed,
536870 remain, 335544 are dead but not yet removable'. The dead rows
can't be removed because the inactive slot is holding an xmin, see
'removable cutoff: 738, which was 268441 XIDs old when operation
ended'.

postgres=# vacuum verbose tab_int;
INFO:  vacuuming "postgres.public.tab_int"
INFO:  finished vacuuming "postgres.public.tab_int": index scans: 0
pages: 0 removed, 2376 remain, 2376 scanned (100.00% of total)
tuples: 0 removed, 536870 remain, 335544 are dead but not yet removable
removable cutoff: 738, which was 268441 XIDs old when operation ended
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen
index scan not needed: 0 pages from table (0.00% of total) had 0 dead
item identifiers removed
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s
buffer usage: 4759 hits, 0 misses, 0 dirtied
WAL usage: 0 records, 0 full page images, 0 bytes
system usage: CPU: user: 0.07 s, system: 0.00 s, elapsed: 0.07 s
VACUUM

6. Now, repeat the above steps but with setting max_slot_xid_age =
200000 on the primary.

7. Do a checkpoint to invalidate the slot.
postgres=# checkpoint;
CHECKPOINT
postgres=# SELECT * FROM pg_replication_slots;
-[ RECORD 1 ]-------+-------------
slot_name           | sb_repl_slot
plugin              |
slot_type           | physical
datoid              |
database            |
temporary           | f
active              | f
active_pid          |
xmin                | 738
catalog_xmin        |
restart_lsn         | 0/3000000
confirmed_flush_lsn |
wal_status          | lost
safe_wal_size       |
two_phase           | f
conflicting         |
failover            | f
synced              | f
invalidation_reason | xid_aged

8. And, then vacuum the table, observe 'tuples: 335544 removed, 201326
remain, 0 are dead but not yet removable'.

postgres=# vacuum verbose tab_int;
INFO:  vacuuming "postgres.public.tab_int"
INFO:  finished vacuuming "postgres.public.tab_int": index scans: 0
pages: 0 removed, 2376 remain, 2376 scanned (100.00% of total)
tuples: 335544 removed, 201326 remain, 0 are dead but not yet removable
removable cutoff: 269179, which was 0 XIDs old when operation ended
new relfrozenxid: 269179, which is 268441 XIDs ahead of previous value
frozen: 1189 pages from table (50.04% of total) had 201326 tuples frozen
index scan not needed: 0 pages from table (0.00% of total) had 0 dead
item identifiers removed
avg read rate: 0.000 MB/s, avg write rate: 193.100 MB/s
buffer usage: 4760 hits, 0 misses, 2381 dirtied
WAL usage: 5942 records, 2378 full page images, 8343275 bytes
system usage: CPU: user: 0.09 s, system: 0.00 s, elapsed: 0.09 s
VACUUM

[1]
cd /home/ubuntu/postgres/pg17/bin
./pg_ctl -D db17 -l logfile17 stop
rm -rf db17 logfile17
rm -rf /home/ubuntu/postgres/pg17/bin/archived_wal
mkdir /home/ubuntu/postgres/pg17/bin/archived_wal

./initdb -D db17
echo "archive_mode = on
archive_command='cp %p
/home/ubuntu/postgres/pg17/bin/archived_wal/%f'" | tee -a
db17/postgresql.conf

./pg_ctl -D db17 -l logfile17 start
./psql -d postgres -p 5432 -c "SELECT
pg_create_physical_replication_slot('sb_repl_slot', true, false);"

rm -rf sbdata logfilesbdata
./pg_basebackup -D sbdata
echo "port=5433
primary_conninfo='host=localhost port=5432 dbname=postgres user=ubuntu'
primary_slot_name='sb_repl_slot'
restore_command='cp /home/ubuntu/postgres/pg17/bin/archived_wal/%f %p'
hot_standby_feedback = on" | tee -a sbdata/postgresql.conf

touch sbdata/standby.signal

./pg_ctl -D sbdata -l logfilesbdata start
./psql -d postgres -p 5433 -c "SELECT pg_is_in_recovery();"

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  56. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-15T07:19:07Z

    On Thu, Mar 14, 2024 at 7:58 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> While we wait to hear from others on this, I'm attaching the v9 patch
> set implementing the above idea (check 0001 patch). Please have a
> look. I'll come back to the other review comments soon.
>

patch002:

1)
I would like to understand the purpose of 'inactive_count'? Is it only
for users for monitoring purposes? We are not using it anywhere
internally.

I shutdown the instance 5 times and found that 'inactive_count' became
5 for all the slots created on that instance. Is this intentional? I
mean we can not really use them if the instance is down.  I felt it
should increment the inactive_count only if during the span of
instance, they were actually inactive i.e. no streaming or replication
happening through them.


2)
slot.c:
+ case RS_INVAL_XID_AGE:
+ {
+ if (TransactionIdIsNormal(s->data.xmin))
+ {
+                          ..........
+ }
+ if (TransactionIdIsNormal(s->data.catalog_xmin))
+ {
+                          ..........
+ }
+ }

Can we optimize this code? It has duplicate code for processing
s->data.catalog_xmin and s->data.xmin. Can we create a sub-function
for this purpose and call it twice here?

thanks
Shveta

  57. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-15T12:05:27Z

    On Fri, Mar 15, 2024 at 10:15 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> > > > wal_level_insufficient it's the reason for conflict with recovery".
>
> +1 on maintaining both conflicting and invalidation_reason

Thanks.

> Thanks for the patch. JFYI, patch09 does not apply to HEAD, some
> recent commit caused the conflict.

Yep, the conflict is in src/test/recovery/meson.build and is because
of e6927270cd18d535b77cbe79c55c6584351524be.

> Some trivial comments on patch001 (yet to review other patches)

Thanks for looking into this.

> 1)
> info.c:
>
> - "%s as caught_up, conflict_reason IS NOT NULL as invalid "
> + "%s as caught_up, invalidation_reason IS NOT NULL as invalid "
>
> Can we revert back to 'conflicting as invalid' since it is a query for
> logical slots only.

I guess, no. There the intention is to check for invalid logical slots
not just for the conflicting ones. The logical slots can get
invalidated due to other reasons as well.

> 2)
> 040_standby_failover_slots_sync.pl:
>
> - q{SELECT conflict_reason IS NULL AND synced AND NOT temporary FROM
> pg_replication_slots WHERE slot_name = 'lsub1_slot';}
> + q{SELECT invalidation_reason IS NULL AND synced AND NOT temporary
> FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';}
>
> Here too, can we have 'NOT conflicting' instead of '
> invalidation_reason IS NULL' as it is a logical slot test.

I guess no. The tests are ensuring the slot on the standby isn't invalidated.

In general, one needs to use the 'conflicting' column from
pg_replication_slots when the intention is to look for reasons for
conflicts, otherwise use the 'invalidation_reason' column for
invalidations.

Please see the attached v10 patch set after resolving the merge
conflict and fixing an indentation warning in the TAP test file.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  58. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-15T14:28:31Z

    On Thu, Mar 14, 2024 at 12:24:00PM +0530, Amit Kapila wrote:
> On Wed, Mar 13, 2024 at 9:24 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
>> On Wed, Mar 13, 2024 at 9:21 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>> > > So, how about we turn conflict_reason to only report the reasons that
>> > > actually cause conflict with recovery for logical slots, something
>> > > like below, and then have invalidation_cause as a generic column for
>> > > all sorts of invalidation reasons for both logical and physical slots?
>> >
>> > If our above understanding is correct then coflict_reason will be a
>> > subset of invalidation_reason. If so, whatever way we arrange this
>> > information, there will be some sort of duplicity unless we just have
>> > one column 'invalidation_reason' and update the docs to interpret it
>> > correctly for conflicts.
>>
>> Yes, there will be some sort of duplicity if we emit conflict_reason
>> as a text field. However, I still think the better way is to turn
>> conflict_reason text to conflict boolean and set it to true only on
>> rows_removed and wal_level_insufficient invalidations. When conflict
>> boolean is true, one (including all the tests that we've added
>> recently) can look for invalidation_reason text field for the reason.
>> This sounds reasonable to me as opposed to we just mentioning in the
>> docs that "if invalidation_reason is rows_removed or
>> wal_level_insufficient it's the reason for conflict with recovery".
> 
> Fair point. I think we can go either way. Bertrand, Nathan, and
> others, do you have an opinion on this matter?

WFM

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  59. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-15T16:45:18Z

    Hi,

On Thu, Mar 14, 2024 at 12:24:00PM +0530, Amit Kapila wrote:
> On Wed, Mar 13, 2024 at 9:24 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Wed, Mar 13, 2024 at 9:21 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > > So, how about we turn conflict_reason to only report the reasons that
> > > > actually cause conflict with recovery for logical slots, something
> > > > like below, and then have invalidation_cause as a generic column for
> > > > all sorts of invalidation reasons for both logical and physical slots?
> > >
> > > If our above understanding is correct then coflict_reason will be a
> > > subset of invalidation_reason. If so, whatever way we arrange this
> > > information, there will be some sort of duplicity unless we just have
> > > one column 'invalidation_reason' and update the docs to interpret it
> > > correctly for conflicts.
> >
> > Yes, there will be some sort of duplicity if we emit conflict_reason
> > as a text field. However, I still think the better way is to turn
> > conflict_reason text to conflict boolean and set it to true only on
> > rows_removed and wal_level_insufficient invalidations. When conflict
> > boolean is true, one (including all the tests that we've added
> > recently) can look for invalidation_reason text field for the reason.
> > This sounds reasonable to me as opposed to we just mentioning in the
> > docs that "if invalidation_reason is rows_removed or
> > wal_level_insufficient it's the reason for conflict with recovery".
> >
> 
> Fair point. I think we can go either way. Bertrand, Nathan, and
> others, do you have an opinion on this matter?

Sounds like a good approach to me and one will be able to quickly identify
if a conflict occured.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  60. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-16T03:59:01Z

    On Fri, Mar 15, 2024 at 12:49 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> patch002:
>
> 1)
> I would like to understand the purpose of 'inactive_count'? Is it only
> for users for monitoring purposes? We are not using it anywhere
> internally.

inactive_count metric helps detect unstable replication slots
connections that have a lot of disconnections. It's not used for the
inactive_timeout based slot invalidation mechanism.

> I shutdown the instance 5 times and found that 'inactive_count' became
> 5 for all the slots created on that instance. Is this intentional?

Yes, it's incremented on shutdown (and for that matter upon every slot
release) for all the slots that are tied to walsenders.

> I mean we can not really use them if the instance is down.  I felt it
> should increment the inactive_count only if during the span of
> instance, they were actually inactive i.e. no streaming or replication
> happening through them.

inactive_count is persisted to disk- upon clean shutdown, so, once the
slots become active again, one gets to see the metric and deduce some
info on disconnections.

Having said that, I'm okay to hear from others on the inactive_count
metric being added.

> 2)
> slot.c:
> + case RS_INVAL_XID_AGE:
>
> Can we optimize this code? It has duplicate code for processing
> s->data.catalog_xmin and s->data.xmin. Can we create a sub-function
> for this purpose and call it twice here?

Good idea. Done that way.

> 2)
> The msg for patch 3 says:
> --------------
> a) when replication slots is lying inactive for a day or so using
> last_inactive_at metric,
> b) when a replication slot is becoming inactive too frequently using
> last_inactive_at metric.
> --------------
>  I think in b, you want to refer to inactive_count instead of last_inactive_at?

Right. Changed.

> 3)
> I do not see invalidation_reason updated for 2 new reasons in system-views.sgml

Nice catch. Added them now.

I've also responded to Bertrand's comments here.

On Wed, Mar 6, 2024 at 3:56 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> A few comments:
>
> 1 ===
>
> +       The reason for the slot's invalidation. <literal>NULL</literal> if the
> +       slot is currently actively being used.
>
> s/currently actively being used/not invalidated/ ? (I mean it could be valid
> and not being used).

Changed.

> 3 ===
>
>         res = executeQueryOrDie(conn, "SELECT slot_name, plugin, two_phase, failover, "
> -                                                       "%s as caught_up, conflict_reason IS NOT NULL as invalid "
> +                                                       "%s as caught_up, invalidation_reason IS NOT NULL as invalid "
>                                                         "FROM pg_catalog.pg_replication_slots "
> -                                                       "(CASE WHEN conflict_reason IS NOT NULL THEN FALSE "
> +                                                       "(CASE WHEN invalidation_reason IS NOT NULL THEN FALSE "
>
> Yeah that's fine because there is logical slot filtering here.

Right. And, we really are looking for invalid slots there, so use of
invalidation_reason is much more correct than conflicting.

> 4 ===
>
> -GetSlotInvalidationCause(const char *conflict_reason)
> +GetSlotInvalidationCause(const char *invalidation_reason)
>
> Should we change the comment "Maps a conflict reason" above this function?

Changed.

> 5 ===
>
> -# Check conflict_reason is NULL for physical slot
> +# Check invalidation_reason is NULL for physical slot
>  $res = $node_primary->safe_psql(
>         'postgres', qq[
> -                SELECT conflict_reason is null FROM pg_replication_slots where slot_name = '$primary_slotname';]
> +                SELECT invalidation_reason is null FROM pg_replication_slots where slot_name = '$primary_slotname';]
>  );
>
>
> I don't think this test is needed anymore: it does not make that much sense since
> it's done after the primary database initialization and startup.

It is now turned into a test verifying 'conflicting boolean' is null
for the physical slot. Isn't that okay?

> 6 ===
>
>         'Logical slots are reported as non conflicting');
>
> What about?
>
> "
> # Verify slots are reported as valid in pg_replication_slots
>     'Logical slots are reported as valid');
> "

Changed.

Please see the attached v11 patch set with all the above review
comments addressed.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  61. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-16T10:24:46Z

    On Fri, Mar 15, 2024 at 10:45 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 13, 2024 at 9:38 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > BTW, is XID the based parameter 'max_slot_xid_age' not have similarity
> > with 'max_slot_wal_keep_size'? I think it will impact the rows we
> > removed based on xid horizons. Don't we need to consider it while
> > vacuum computing the xid horizons in ComputeXidHorizons() similar to
> > what we do for WAL w.r.t 'max_slot_wal_keep_size'?
>
> I'm having a hard time understanding why we'd need something up there
> in ComputeXidHorizons(). Can you elaborate it a bit please?
>
> What's proposed with max_slot_xid_age is that during checkpoint we
> look at slot's xmin and catalog_xmin, and the current system txn id.
> Then, if the XID age of (xmin, catalog_xmin) and current_xid crosses
> max_slot_xid_age, we invalidate the slot.
>

I can see that in your patch (in function
InvalidatePossiblyObsoleteSlot()). As per my understanding, we need
something similar for slot xids in ComputeXidHorizons() as we are
doing WAL in KeepLogSeg(). In KeepLogSeg(), we compute the minimum LSN
location required by slots and then adjust it for
'max_slot_wal_keep_size'. On similar lines, currently in
ComputeXidHorizons(), we compute the minimum xid required by slots
(procArray->replication_slot_xmin and
procArray->replication_slot_catalog_xmin) but then don't adjust it for
'max_slot_xid_age'. I could be missing something in this but it is
better to keep discussing this and try to move with another parameter
'inactive_replication_slot_timeout' which according to me can be kept
at slot level instead of a GUC but OTOH we need to see the arguments
on both side and then decide which makes more sense.

-- 
With Regards,
Amit Kapila.

  62. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-17T08:33:10Z

    On Sat, Mar 16, 2024 at 3:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> procArray->replication_slot_catalog_xmin) but then don't adjust it for
> 'max_slot_xid_age'. I could be missing something in this but it is
> better to keep discussing this and try to move with another parameter
> 'inactive_replication_slot_timeout' which according to me can be kept
> at slot level instead of a GUC but OTOH we need to see the arguments
> on both side and then decide which makes more sense.

Hm. Are you suggesting inactive_timeout to be a slot level parameter
similar to 'failover' property added recently by
c393308b69d229b664391ac583b9e07418d411b6 and
73292404370c9900a96e2bebdc7144f7010339cf? With this approach, one can
set inactive_timeout while creating the slot either via
pg_create_physical_replication_slot() or
pg_create_logical_replication_slot() or CREATE_REPLICATION_SLOT or
ALTER_REPLICATION_SLOT command, and postgres tracks the
last_inactive_at for every slot based on which the slot gets
invalidated. If this understanding is right, I can go ahead and work
towards it.

Alternatively, we can go the route of making GUC a list of key-value
pairs of {slot_name, inactive_timeout}, but this kind of GUC for
setting slot level parameters is going to be the first of its kind, so
I'd prefer the above approach.

Thoughts?

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  63. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-18T03:20:56Z

    On Sun, Mar 17, 2024 at 2:03 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Sat, Mar 16, 2024 at 3:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > procArray->replication_slot_catalog_xmin) but then don't adjust it for
> > 'max_slot_xid_age'. I could be missing something in this but it is
> > better to keep discussing this and try to move with another parameter
> > 'inactive_replication_slot_timeout' which according to me can be kept
> > at slot level instead of a GUC but OTOH we need to see the arguments
> > on both side and then decide which makes more sense.
>
> Hm. Are you suggesting inactive_timeout to be a slot level parameter
> similar to 'failover' property added recently by
> c393308b69d229b664391ac583b9e07418d411b6 and
> 73292404370c9900a96e2bebdc7144f7010339cf? With this approach, one can
> set inactive_timeout while creating the slot either via
> pg_create_physical_replication_slot() or
> pg_create_logical_replication_slot() or CREATE_REPLICATION_SLOT or
> ALTER_REPLICATION_SLOT command, and postgres tracks the
> last_inactive_at for every slot based on which the slot gets
> invalidated. If this understanding is right, I can go ahead and work
> towards it.
>

Yeah, I have something like that in mind. You can prepare the patch
but it would be good if others involved in this thread can also share
their opinion.

> Alternatively, we can go the route of making GUC a list of key-value
> pairs of {slot_name, inactive_timeout}, but this kind of GUC for
> setting slot level parameters is going to be the first of its kind, so
> I'd prefer the above approach.
>

I would prefer a slot-level parameter in this case rather than a GUC.

-- 
With Regards,
Amit Kapila.

  64. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-18T04:28:42Z

    On Sat, Mar 16, 2024 at 3:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > What's proposed with max_slot_xid_age is that during checkpoint we
> > look at slot's xmin and catalog_xmin, and the current system txn id.
> > Then, if the XID age of (xmin, catalog_xmin) and current_xid crosses
> > max_slot_xid_age, we invalidate the slot.
> >
>
> I can see that in your patch (in function
> InvalidatePossiblyObsoleteSlot()). As per my understanding, we need
> something similar for slot xids in ComputeXidHorizons() as we are
> doing WAL in KeepLogSeg(). In KeepLogSeg(), we compute the minimum LSN
> location required by slots and then adjust it for
> 'max_slot_wal_keep_size'. On similar lines, currently in
> ComputeXidHorizons(), we compute the minimum xid required by slots
> (procArray->replication_slot_xmin and
> procArray->replication_slot_catalog_xmin) but then don't adjust it for
> 'max_slot_xid_age'. I could be missing something in this but it is
> better to keep discussing this

After invalidating slots because of max_slot_xid_age, the
procArray->replication_slot_xmin and
procArray->replication_slot_catalog_xmin are recomputed immediately in
InvalidateObsoleteReplicationSlots->ReplicationSlotsComputeRequiredXmin->ProcArraySetReplicationSlotXmin.
And, later the XID horizons in ComputeXidHorizons are computed before
the vacuum on each table via GetOldestNonRemovableTransactionId.
Aren't these enough? Do you want the XID horizons recomputed
immediately, something like the below?

/* Invalidate replication slots based on xmin or catalog_xmin age */
if (max_slot_xid_age > 0)
{
    if (InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE,
                                           0, InvalidOid,
                                           InvalidTransactionId))
    {
        ComputeXidHorizonsResult horizons;

        /*
         * Some slots have been invalidated; update the XID horizons
         * as a side-effect.
         */
        ComputeXidHorizons(&horizons);
    }
}

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  65. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-18T04:33:53Z

    On Mon, Mar 18, 2024 at 9:58 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Sat, Mar 16, 2024 at 3:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > What's proposed with max_slot_xid_age is that during checkpoint we
> > > look at slot's xmin and catalog_xmin, and the current system txn id.
> > > Then, if the XID age of (xmin, catalog_xmin) and current_xid crosses
> > > max_slot_xid_age, we invalidate the slot.
> > >
> >
> > I can see that in your patch (in function
> > InvalidatePossiblyObsoleteSlot()). As per my understanding, we need
> > something similar for slot xids in ComputeXidHorizons() as we are
> > doing WAL in KeepLogSeg(). In KeepLogSeg(), we compute the minimum LSN
> > location required by slots and then adjust it for
> > 'max_slot_wal_keep_size'. On similar lines, currently in
> > ComputeXidHorizons(), we compute the minimum xid required by slots
> > (procArray->replication_slot_xmin and
> > procArray->replication_slot_catalog_xmin) but then don't adjust it for
> > 'max_slot_xid_age'. I could be missing something in this but it is
> > better to keep discussing this
>
> After invalidating slots because of max_slot_xid_age, the
> procArray->replication_slot_xmin and
> procArray->replication_slot_catalog_xmin are recomputed immediately in
> InvalidateObsoleteReplicationSlots->ReplicationSlotsComputeRequiredXmin->ProcArraySetReplicationSlotXmin.
> And, later the XID horizons in ComputeXidHorizons are computed before
> the vacuum on each table via GetOldestNonRemovableTransactionId.
> Aren't these enough?
>

IIUC, this will be delayed by one cycle in the vacuum rather than
doing it when the slot's xmin age is crossed and it can be
invalidated.

 Do you want the XID horizons recomputed
> immediately, something like the below?
>

I haven't thought of the exact logic but we can try to mimic the
handling similar to WAL.

-- 
With Regards,
Amit Kapila.

  66. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-18T09:18:37Z

    Hi,

On Sat, Mar 16, 2024 at 09:29:01AM +0530, Bharath Rupireddy wrote:
> I've also responded to Bertrand's comments here.

Thanks!

> 
> On Wed, Mar 6, 2024 at 3:56 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > 5 ===
> >
> > -# Check conflict_reason is NULL for physical slot
> > +# Check invalidation_reason is NULL for physical slot
> >  $res = $node_primary->safe_psql(
> >         'postgres', qq[
> > -                SELECT conflict_reason is null FROM pg_replication_slots where slot_name = '$primary_slotname';]
> > +                SELECT invalidation_reason is null FROM pg_replication_slots where slot_name = '$primary_slotname';]
> >  );
> >
> >
> > I don't think this test is needed anymore: it does not make that much sense since
> > it's done after the primary database initialization and startup.
> 
> It is now turned into a test verifying 'conflicting boolean' is null
> for the physical slot. Isn't that okay?

Yeah makes more sense now, thanks!

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  67. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-18T09:32:40Z

    Hi,

On Mon, Mar 18, 2024 at 08:50:56AM +0530, Amit Kapila wrote:
> On Sun, Mar 17, 2024 at 2:03 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Sat, Mar 16, 2024 at 3:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > procArray->replication_slot_catalog_xmin) but then don't adjust it for
> > > 'max_slot_xid_age'. I could be missing something in this but it is
> > > better to keep discussing this and try to move with another parameter
> > > 'inactive_replication_slot_timeout' which according to me can be kept
> > > at slot level instead of a GUC but OTOH we need to see the arguments
> > > on both side and then decide which makes more sense.
> >
> > Hm. Are you suggesting inactive_timeout to be a slot level parameter
> > similar to 'failover' property added recently by
> > c393308b69d229b664391ac583b9e07418d411b6 and
> > 73292404370c9900a96e2bebdc7144f7010339cf? With this approach, one can
> > set inactive_timeout while creating the slot either via
> > pg_create_physical_replication_slot() or
> > pg_create_logical_replication_slot() or CREATE_REPLICATION_SLOT or
> > ALTER_REPLICATION_SLOT command, and postgres tracks the
> > last_inactive_at for every slot based on which the slot gets
> > invalidated. If this understanding is right, I can go ahead and work
> > towards it.
> >
> 
> Yeah, I have something like that in mind. You can prepare the patch
> but it would be good if others involved in this thread can also share
> their opinion.

I think it makes sense to put the inactive_timeout granularity at the slot
level (as the activity could vary a lot say between one slot linked to a 
subcription and one linked to some plugins). As far max_slot_xid_age I've the
feeling that a new GUC is good enough.

> > Alternatively, we can go the route of making GUC a list of key-value
> > pairs of {slot_name, inactive_timeout}, but this kind of GUC for
> > setting slot level parameters is going to be the first of its kind, so
> > I'd prefer the above approach.
> >
> 
> I would prefer a slot-level parameter in this case rather than a GUC.

Yeah, same here.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  68. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-18T10:12:15Z

    Hi,

On Sat, Mar 16, 2024 at 09:29:01AM +0530, Bharath Rupireddy wrote:
> Please see the attached v11 patch set with all the above review
> comments addressed.

Thanks!

Looking at 0001:

1 ===

+       True if this logical slot conflicted with recovery (and so is now
+       invalidated). When this column is true, check

Worth to add back the physical slot mention "Always NULL for physical slots."?

2 ===

@@ -1023,9 +1023,10 @@ CREATE VIEW pg_replication_slots AS
             L.wal_status,
             L.safe_wal_size,
             L.two_phase,
-            L.conflict_reason,
+            L.conflicting,
             L.failover,
-            L.synced
+            L.synced,
+            L.invalidation_reason

What about making invalidation_reason close to conflict_reason?

3 ===

- * Maps a conflict reason for a replication slot to
+ * Maps a invalidation reason for a replication slot to

s/a invalidation/an invalidation/?

4 ===

While at it, shouldn't we also rename "conflict" to say "invalidation_cause" in
InvalidatePossiblyObsoleteSlot()?

5 ===

+ * rows_removed and wal_level_insufficient are only two reasons

s/are only two/are the only two/?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  69. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-18T14:49:51Z

    Hi,

On Thu, Mar 14, 2024 at 12:27:26PM +0530, Amit Kapila wrote:
> On Wed, Mar 13, 2024 at 10:16 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Wed, Mar 13, 2024 at 11:13 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > > Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.
> > >
> > > JFYI, the patch does not apply to the head. There is a conflict in
> > > multiple files.
> >
> > Thanks for looking into this. I noticed that the v8 patches needed
> > rebase. Before I go do anything with the patches, I'm trying to gain
> > consensus on the design. Following is the summary of design choices
> > we've discussed so far:
> > 1) conflict_reason vs invalidation_reason.
> > 2) When to compute the XID age?
> >
> 
> I feel we should focus on two things (a) one is to introduce a new
> column invalidation_reason, and (b) let's try to first complete
> invalidation due to timeout. We can look into XID stuff if time
> permits, remember, we don't have ample time left.

Agree. While it makes sense to invalidate slots for wal removal in
CreateCheckPoint() (because this is the place where wal is removed), I 'm not
sure this is the right place for the 2 new cases.

Let's focus on the timeout one as proposed above (as probably the simplest one):
as this one is purely related to time and activity what about to invalidate them
when?:

- their usage resume
- in pg_get_replication_slots()

The idea is to invalidate the slot when one resumes activity on it or wants to
get information about it (and among other things wants to know if the slot is
valid or not).

Thoughts?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  70. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-19T05:26:25Z

    On Mon, Mar 18, 2024 at 8:19 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Thu, Mar 14, 2024 at 12:27:26PM +0530, Amit Kapila wrote:
> > On Wed, Mar 13, 2024 at 10:16 PM Bharath Rupireddy
> > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > >
> > > On Wed, Mar 13, 2024 at 11:13 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > > Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change.
> > > >
> > > > JFYI, the patch does not apply to the head. There is a conflict in
> > > > multiple files.
> > >
> > > Thanks for looking into this. I noticed that the v8 patches needed
> > > rebase. Before I go do anything with the patches, I'm trying to gain
> > > consensus on the design. Following is the summary of design choices
> > > we've discussed so far:
> > > 1) conflict_reason vs invalidation_reason.
> > > 2) When to compute the XID age?
> > >
> >
> > I feel we should focus on two things (a) one is to introduce a new
> > column invalidation_reason, and (b) let's try to first complete
> > invalidation due to timeout. We can look into XID stuff if time
> > permits, remember, we don't have ample time left.
>
> Agree. While it makes sense to invalidate slots for wal removal in
> CreateCheckPoint() (because this is the place where wal is removed), I 'm not
> sure this is the right place for the 2 new cases.
>
> Let's focus on the timeout one as proposed above (as probably the simplest one):
> as this one is purely related to time and activity what about to invalidate them
> when?:
>
> - their usage resume
> - in pg_get_replication_slots()
>
> The idea is to invalidate the slot when one resumes activity on it or wants to
> get information about it (and among other things wants to know if the slot is
> valid or not).
>

Trying to invalidate at those two places makes sense to me but we
still need to cover the cases where it takes very long to resume the
slot activity and the dangling slot cases where the activity is never
resumed. How about apart from the above two places, trying to
invalidate in CheckPointReplicationSlots() where we are traversing all
the slots? This could prevent invalid slots from being marked as
dirty.

BTW, how will the user use 'inactive_count' to know whether a
replication slot is becoming inactive too frequently? The patch just
keeps incrementing this counter, one will never know in the last 'n'
minutes, how many times the slot became inactive unless there is some
monitoring tool that keeps capturing this counter from time to time
and calculates the frequency in some way. Even, if this is useful, it
is not clear to me whether we need to store 'inactive_count' in the
slot's persistent data. I understand it could be a metric required by
the user but wouldn't it be better to track this via
pg_stat_replication_slots such that we don't need to store this in
slot's persist data? If this understanding is correct, I would say
let's remove 'inactive_count' as well from the main patch and discuss
it separately.

-- 
With Regards,
Amit Kapila.

  71. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-19T09:41:10Z

    Hi,

On Tue, Mar 19, 2024 at 10:56:25AM +0530, Amit Kapila wrote:
> On Mon, Mar 18, 2024 at 8:19 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> > Agree. While it makes sense to invalidate slots for wal removal in
> > CreateCheckPoint() (because this is the place where wal is removed), I 'm not
> > sure this is the right place for the 2 new cases.
> >
> > Let's focus on the timeout one as proposed above (as probably the simplest one):
> > as this one is purely related to time and activity what about to invalidate them
> > when?:
> >
> > - their usage resume
> > - in pg_get_replication_slots()
> >
> > The idea is to invalidate the slot when one resumes activity on it or wants to
> > get information about it (and among other things wants to know if the slot is
> > valid or not).
> >
> 
> Trying to invalidate at those two places makes sense to me but we
> still need to cover the cases where it takes very long to resume the
> slot activity and the dangling slot cases where the activity is never
> resumed.

I understand it's better to have the slot reflecting its real status internally
but it is a real issue if that's not the case until the activity on it is resumed?
(just asking, not saying we should not)

> How about apart from the above two places, trying to
> invalidate in CheckPointReplicationSlots() where we are traversing all
> the slots?

I think that's a good place but there is still a window of time (that could also
be "large" depending of the activity and the checkpoint frequency) during which
the slot is not known as invalid internally. But yeah, at leat we know that we'll
mark it as invalid at some point...

BTW:

        if (am_walsender)
        {
+               if (slot->data.persistency == RS_PERSISTENT)
+               {
+                       SpinLockAcquire(&slot->mutex);
+                       slot->data.last_inactive_at = GetCurrentTimestamp();
+                       slot->data.inactive_count++;
+                       SpinLockRelease(&slot->mutex);

I'm also feeling the same concern as Shveta mentioned in [1]: that a "normal"
backend using pg_logical_slot_get_changes() or friends would not set the
last_inactive_at.

[1]: https://www.postgresql.org/message-id/CAJpy0uD64X%3D2ENmbHaRiWTKeQawr-rbGoy_GdhQQLVXzUSKTMg%40mail.gmail.com

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  72. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-19T10:50:35Z

    On Tue, Mar 19, 2024 at 3:11 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Tue, Mar 19, 2024 at 10:56:25AM +0530, Amit Kapila wrote:
> > On Mon, Mar 18, 2024 at 8:19 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > > Agree. While it makes sense to invalidate slots for wal removal in
> > > CreateCheckPoint() (because this is the place where wal is removed), I 'm not
> > > sure this is the right place for the 2 new cases.
> > >
> > > Let's focus on the timeout one as proposed above (as probably the simplest one):
> > > as this one is purely related to time and activity what about to invalidate them
> > > when?:
> > >
> > > - their usage resume
> > > - in pg_get_replication_slots()
> > >
> > > The idea is to invalidate the slot when one resumes activity on it or wants to
> > > get information about it (and among other things wants to know if the slot is
> > > valid or not).
> > >
> >
> > Trying to invalidate at those two places makes sense to me but we
> > still need to cover the cases where it takes very long to resume the
> > slot activity and the dangling slot cases where the activity is never
> > resumed.
>
> I understand it's better to have the slot reflecting its real status internally
> but it is a real issue if that's not the case until the activity on it is resumed?
> (just asking, not saying we should not)
>

Sorry, I didn't understand your point. Can you try to explain by example?

-- 
With Regards,
Amit Kapila.

  73. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-19T12:42:21Z

    Hi,

On Tue, Mar 19, 2024 at 04:20:35PM +0530, Amit Kapila wrote:
> On Tue, Mar 19, 2024 at 3:11 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Tue, Mar 19, 2024 at 10:56:25AM +0530, Amit Kapila wrote:
> > > On Mon, Mar 18, 2024 at 8:19 PM Bertrand Drouvot
> > > <bertranddrouvot.pg@gmail.com> wrote:
> > > > Agree. While it makes sense to invalidate slots for wal removal in
> > > > CreateCheckPoint() (because this is the place where wal is removed), I 'm not
> > > > sure this is the right place for the 2 new cases.
> > > >
> > > > Let's focus on the timeout one as proposed above (as probably the simplest one):
> > > > as this one is purely related to time and activity what about to invalidate them
> > > > when?:
> > > >
> > > > - their usage resume
> > > > - in pg_get_replication_slots()
> > > >
> > > > The idea is to invalidate the slot when one resumes activity on it or wants to
> > > > get information about it (and among other things wants to know if the slot is
> > > > valid or not).
> > > >
> > >
> > > Trying to invalidate at those two places makes sense to me but we
> > > still need to cover the cases where it takes very long to resume the
> > > slot activity and the dangling slot cases where the activity is never
> > > resumed.
> >
> > I understand it's better to have the slot reflecting its real status internally
> > but it is a real issue if that's not the case until the activity on it is resumed?
> > (just asking, not saying we should not)
> >
> 
> Sorry, I didn't understand your point. Can you try to explain by example?

Sorry if that was not clear, let me try to rephrase it first: what issue to you
see if the invalidation of such a slot occurs only when its usage resume or
when pg_get_replication_slots() is triggered? I understand that this could lead
to the slot not being invalidated (maybe forever) but is that an issue for an
inactive slot?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  74. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-19T19:18:55Z

    On Mon, Mar 18, 2024 at 3:02 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > > Hm. Are you suggesting inactive_timeout to be a slot level parameter
> > > similar to 'failover' property added recently by
> > > c393308b69d229b664391ac583b9e07418d411b6 and
> > > 73292404370c9900a96e2bebdc7144f7010339cf?
> >
> > Yeah, I have something like that in mind. You can prepare the patch
> > but it would be good if others involved in this thread can also share
> > their opinion.
>
> I think it makes sense to put the inactive_timeout granularity at the slot
> level (as the activity could vary a lot say between one slot linked to a
> subcription and one linked to some plugins). As far max_slot_xid_age I've the
> feeling that a new GUC is good enough.

Well, here I'm implementing the above idea. The attached v12 patches
majorly have the following changes:

1. inactive_timeout is now slot-level, that is, one can set it while
creating the slot either via SQL functions or via replication commands
or via subscription.
2. last_inactive_at and inactive_timeout are now tracked in on-disk
replication slot data structure.
3. last_inactive_at is now set even for non-walsenders whenever the
slot is released as opposed to initial versions of the patches setting
it only for walsenders.
4. slot's inactive_timeout parameter is now migrated to the new
cluster with pg_upgrade.
5. slot's inactive_timeout parameter is now synced to the standby when
failover is enabled for the slot.
6. Test cases are added to cover most of the above cases including new
invalidation mechanisms.

Following are some open points:

1. Where to do inactive_timeout invalidation exactly if not the checkpointer.
2. Where to do XID age invalidation exactly if not the checkpointer.
3. How to go about recomputing XID horizons based on max_slot_xid_age.
Does the slot's horizon's need to be adjusted in ComputeXidHorizons()?
4. New invalidation mechanisms interaction with slot sync feature.
5. Review comments on 0001 from Bertrand.

Please see the attached v12 patches.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  75. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-20T02:28:20Z

    On Tue, Mar 19, 2024 at 6:12 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Tue, Mar 19, 2024 at 04:20:35PM +0530, Amit Kapila wrote:
> > On Tue, Mar 19, 2024 at 3:11 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > On Tue, Mar 19, 2024 at 10:56:25AM +0530, Amit Kapila wrote:
> > > > On Mon, Mar 18, 2024 at 8:19 PM Bertrand Drouvot
> > > > <bertranddrouvot.pg@gmail.com> wrote:
> > > > > Agree. While it makes sense to invalidate slots for wal removal in
> > > > > CreateCheckPoint() (because this is the place where wal is removed), I 'm not
> > > > > sure this is the right place for the 2 new cases.
> > > > >
> > > > > Let's focus on the timeout one as proposed above (as probably the simplest one):
> > > > > as this one is purely related to time and activity what about to invalidate them
> > > > > when?:
> > > > >
> > > > > - their usage resume
> > > > > - in pg_get_replication_slots()
> > > > >
> > > > > The idea is to invalidate the slot when one resumes activity on it or wants to
> > > > > get information about it (and among other things wants to know if the slot is
> > > > > valid or not).
> > > > >
> > > >
> > > > Trying to invalidate at those two places makes sense to me but we
> > > > still need to cover the cases where it takes very long to resume the
> > > > slot activity and the dangling slot cases where the activity is never
> > > > resumed.
> > >
> > > I understand it's better to have the slot reflecting its real status internally
> > > but it is a real issue if that's not the case until the activity on it is resumed?
> > > (just asking, not saying we should not)
> > >
> >
> > Sorry, I didn't understand your point. Can you try to explain by example?
>
> Sorry if that was not clear, let me try to rephrase it first: what issue to you
> see if the invalidation of such a slot occurs only when its usage resume or
> when pg_get_replication_slots() is triggered? I understand that this could lead
> to the slot not being invalidated (maybe forever) but is that an issue for an
> inactive slot?
>

It has the risk of preventing WAL and row removal. I think this is the
primary reason we are at the first place planning to have such a
parameter. So, we should have some way to invalidate it even when the
walsender/backend process doesn't use it again.

-- 
With Regards,
Amit Kapila.

  76. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-20T03:28:05Z

    On Wed, Mar 20, 2024 at 12:49 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
>
> Following are some open points:
>
> 1. Where to do inactive_timeout invalidation exactly if not the checkpointer.
>

I have suggested to do it at the time of CheckpointReplicationSlots()
and Bertrand suggested to do it whenever we resume using the slot. I
think we should follow both the suggestions.

> 2. Where to do XID age invalidation exactly if not the checkpointer.
> 3. How to go about recomputing XID horizons based on max_slot_xid_age.
> Does the slot's horizon's need to be adjusted in ComputeXidHorizons()?
>

I suggest postponing the patch for xid based invalidation for a later
discussion.

> 4. New invalidation mechanisms interaction with slot sync feature.
>

Yeah, this is important. My initial thoughts are that synced slots
shouldn't be invalidated on the standby due to timeout.

> 5. Review comments on 0001 from Bertrand.
>
> Please see the attached v12 patches.
>

Thanks for quickly updating the patches.

-- 
With Regards,
Amit Kapila.

  77. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-20T05:47:47Z

    On Mon, Mar 18, 2024 at 3:42 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> Looking at 0001:

Thanks for reviewing.

> 1 ===
>
> +       True if this logical slot conflicted with recovery (and so is now
> +       invalidated). When this column is true, check
>
> Worth to add back the physical slot mention "Always NULL for physical slots."?

Will change.

> 2 ===
>
> @@ -1023,9 +1023,10 @@ CREATE VIEW pg_replication_slots AS
>              L.wal_status,
>              L.safe_wal_size,
>              L.two_phase,
> -            L.conflict_reason,
> +            L.conflicting,
>              L.failover,
> -            L.synced
> +            L.synced,
> +            L.invalidation_reason
>
> What about making invalidation_reason close to conflict_reason?

Not required I think. One can pick the required columns in the SELECT
clause anyways.

> 3 ===
>
> - * Maps a conflict reason for a replication slot to
> + * Maps a invalidation reason for a replication slot to
>
> s/a invalidation/an invalidation/?

Will change.

> 4 ===
>
> While at it, shouldn't we also rename "conflict" to say "invalidation_cause" in
> InvalidatePossiblyObsoleteSlot()?

That's inline with our understanding about conflict vs invalidation,
and keeps the function generic. Will change.

> 5 ===
>
> + * rows_removed and wal_level_insufficient are only two reasons
>
> s/are only two/are the only two/?

Will change..

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  78. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-20T07:34:04Z

    Hi,

On Wed, Mar 20, 2024 at 08:58:05AM +0530, Amit Kapila wrote:
> On Wed, Mar 20, 2024 at 12:49 AM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> >
> > Following are some open points:
> >
> > 1. Where to do inactive_timeout invalidation exactly if not the checkpointer.
> >
> 
> I have suggested to do it at the time of CheckpointReplicationSlots()
> and Bertrand suggested to do it whenever we resume using the slot. I
> think we should follow both the suggestions.

Agree. I also think that pg_get_replication_slots() would be a good place, so
that queries would return the right invalidation status.

> > 4. New invalidation mechanisms interaction with slot sync feature.
> >
> 
> Yeah, this is important. My initial thoughts are that synced slots
> shouldn't be invalidated on the standby due to timeout.

+1

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  79. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-20T08:21:54Z

    Hi,

On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> On Mon, Mar 18, 2024 at 3:02 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > > Hm. Are you suggesting inactive_timeout to be a slot level parameter
> > > > similar to 'failover' property added recently by
> > > > c393308b69d229b664391ac583b9e07418d411b6 and
> > > > 73292404370c9900a96e2bebdc7144f7010339cf?
> > >
> > > Yeah, I have something like that in mind. You can prepare the patch
> > > but it would be good if others involved in this thread can also share
> > > their opinion.
> >
> > I think it makes sense to put the inactive_timeout granularity at the slot
> > level (as the activity could vary a lot say between one slot linked to a
> > subcription and one linked to some plugins). As far max_slot_xid_age I've the
> > feeling that a new GUC is good enough.
> 
> Well, here I'm implementing the above idea.

Thanks!

> The attached v12 patches
> majorly have the following changes:
> 
> 2. last_inactive_at and inactive_timeout are now tracked in on-disk
> replication slot data structure.

Should last_inactive_at be tracked on disk? Say the engine is down for a period
of time > inactive_timeout then the slot will be invalidated after the engine
re-start (if no activity before we invalidate the slot). Should the time the
engine is down be counted as "inactive" time? I've the feeling it should not, and
that we should only take into account inactive time while the engine is up.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  80. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-20T13:38:18Z

    Hi,

On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> On Mon, Mar 18, 2024 at 3:02 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > > Hm. Are you suggesting inactive_timeout to be a slot level parameter
> > > > similar to 'failover' property added recently by
> > > > c393308b69d229b664391ac583b9e07418d411b6 and
> > > > 73292404370c9900a96e2bebdc7144f7010339cf?
> > >
> > > Yeah, I have something like that in mind. You can prepare the patch
> > > but it would be good if others involved in this thread can also share
> > > their opinion.
> >
> > I think it makes sense to put the inactive_timeout granularity at the slot
> > level (as the activity could vary a lot say between one slot linked to a
> > subcription and one linked to some plugins). As far max_slot_xid_age I've the
> > feeling that a new GUC is good enough.
> 
> Well, here I'm implementing the above idea. The attached v12 patches
> majorly have the following changes:
> 

Regarding v12-0004: "Allow setting inactive_timeout in the replication command",
shouldn't we also add an new SQL API say: pg_alter_replication_slot() that would
allow to change the timeout property? 

That would allow users to alter this property without the need to make a
replication connection. 

But the issue is that it would make it inconsistent with the new inactivetimeout
in the subscription that is added in "v12-0005". But do we need to display
subinactivetimeout in pg_subscription (and even allow it at subscription creation
/ alter) after all? (I've the feeling there is less such a need as compare to
subfailover, subtwophasestate for example).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  81. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-20T23:35:46Z

    On Wed, Mar 20, 2024 at 1:04 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Wed, Mar 20, 2024 at 08:58:05AM +0530, Amit Kapila wrote:
> > On Wed, Mar 20, 2024 at 12:49 AM Bharath Rupireddy
> > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > >
> > > Following are some open points:
> > >
> > > 1. Where to do inactive_timeout invalidation exactly if not the checkpointer.
> > >
> > I have suggested to do it at the time of CheckpointReplicationSlots()
> > and Bertrand suggested to do it whenever we resume using the slot. I
> > think we should follow both the suggestions.
>
> Agree. I also think that pg_get_replication_slots() would be a good place, so
> that queries would return the right invalidation status.

I've addressed review comments and attaching the v13 patches with the
following changes:

1. Invalidate replication slot due to inactive_timeout:
1.1 In CheckpointReplicationSlots() to help with automatic invalidation.
1.2 In pg_get_replication_slots to help readers see the latest slot information.
1.3 In ReplicationSlotAcquire for walsenders as typically walsenders
are the ones that use slots for longer durations for streaming
standbys and logical subscribers.
1.4 In ReplicationSlotAcquire when called from
pg_logical_slot_get_changes_guts to help with logical decoding clients
to disallow decoding from invalidated slots.
1.5 In ReplicationSlotAcquire when called from
pg_replication_slot_advance to help with disallowing advancing
invalidated slots.
2. Have a new input parameter bool check_for_invalidation for
ReplicationSlotAcquire(). When true, check for the inactive_timeout
invalidation, if invalidated, error out.
3. Have a new function to just do inactive_timeout invalidation.
4. Do not update last_inactive_at for failover slots on standby to not
invalidate failover slots on the standby.
5. In ReplicationSlotAcquire(), invalidate the slot before making it active.
6. Make last_inactive_at a shared-memory parameter as opposed to an
on-disk parameter to help not count the server downtime for inactive
time.
7. Let the failover slot on standby and pg_upgraded slots get
inactive_timeout parameter from the primary and old cluster
respectively.

Please see the attached v13 patches.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  82. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-20T23:49:05Z

    On Wed, Mar 20, 2024 at 7:08 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Regarding v12-0004: "Allow setting inactive_timeout in the replication command",
> shouldn't we also add an new SQL API say: pg_alter_replication_slot() that would
> allow to change the timeout property?
>
> That would allow users to alter this property without the need to make a
> replication connection.

+1 to add a new SQL function pg_alter_replication_slot(). It helps
first create the slots and then later decide the appropriate
inactive_timeout. It might grow into altering other slot parameters
such as failover (I'm not sure if altering failover property on the
primary after a while makes it the right candidate for syncing on the
standby). Perhaps, we can add it for altering just inactive_timeout
for now and be done with it.

FWIW, ALTER_REPLICATION_SLOT was added keeping in mind just the
failover property for logical slots, that's why it emits an error
"cannot use ALTER_REPLICATION_SLOT with a physical replication slot"

> But the issue is that it would make it inconsistent with the new inactivetimeout
> in the subscription that is added in "v12-0005".

Can you please elaborate what the inconsistency it causes with inactivetimeout?

> But do we need to display
> subinactivetimeout in pg_subscription (and even allow it at subscription creation
> / alter) after all? (I've the feeling there is less such a need as compare to
> subfailover, subtwophasestate for example).

Maybe we don't need to. One can always trace down to the replication
slot associated with the subscription on the publisher, and get to
know what the slot's inactive_timeout setting is. However, it looks to
me that it avoids one going to the publisher to know the
inactive_timeout value for a subscription. Moreover, we are allowing
the inactive_timeout to be set via CREATE/ALTER SUBSCRIPTION command,
I believe there's nothing wrong if it's also part of the
pg_subscription catalog.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  83. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-21T03:17:18Z

    On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> >
> > 2. last_inactive_at and inactive_timeout are now tracked in on-disk
> > replication slot data structure.
>
> Should last_inactive_at be tracked on disk? Say the engine is down for a period
> of time > inactive_timeout then the slot will be invalidated after the engine
> re-start (if no activity before we invalidate the slot). Should the time the
> engine is down be counted as "inactive" time? I've the feeling it should not, and
> that we should only take into account inactive time while the engine is up.
>

Good point. The question is how do we achieve this without persisting
the 'last_inactive_at'? Say, 'last_inactive_at' for a particular slot
had some valid value before we shut down but it still didn't cross the
configured 'inactive_timeout' value, so, we won't be able to
invalidate it. Now, after the restart, as we don't know the
last_inactive_at's value before the shutdown, we will initialize it
with 0 (this is what Bharath seems to have done in the latest
v13-0002* patch). After this, even if walsender or backend never
acquires the slot, we won't invalidate it. OTOH, if we track
'last_inactive_at' on the disk, after, restart, we could initialize it
to the current time if the value is non-zero. Do you have any better
ideas?

-- 
With Regards,
Amit Kapila.

  84. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-21T03:37:18Z

    On Thu, Mar 21, 2024 at 5:19 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 20, 2024 at 7:08 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Regarding v12-0004: "Allow setting inactive_timeout in the replication command",
> > shouldn't we also add an new SQL API say: pg_alter_replication_slot() that would
> > allow to change the timeout property?
> >
> > That would allow users to alter this property without the need to make a
> > replication connection.
>
> +1 to add a new SQL function pg_alter_replication_slot().
>

I also don't see any obvious problem with such an API. However, this
is not a good time to invent new APIs. Let's keep the feature simple
and then we can extend it in the next version after more discussion
and probably by that time we will get some feedback from the field as
well.

>
> It helps
> first create the slots and then later decide the appropriate
> inactive_timeout. It might grow into altering other slot parameters
> such as failover (I'm not sure if altering failover property on the
> primary after a while makes it the right candidate for syncing on the
> standby). Perhaps, we can add it for altering just inactive_timeout
> for now and be done with it.
>
> FWIW, ALTER_REPLICATION_SLOT was added keeping in mind just the
> failover property for logical slots, that's why it emits an error
> "cannot use ALTER_REPLICATION_SLOT with a physical replication slot"
>
> > But the issue is that it would make it inconsistent with the new inactivetimeout
> > in the subscription that is added in "v12-0005".
>
> Can you please elaborate what the inconsistency it causes with inactivetimeout?
>

I think the inconsistency can arise from the fact that on publisher
one can change the inactive_timeout for the slot corresponding to a
subscription but the subscriber won't know, so it will still show the
old value. If we want we can document this as a limitation and let
users be aware of it. However, I feel at this stage, let's not even
expose this from the subscription or maybe we can discuss it once/if
we are done with other patches. Anyway, if one wants to use this
feature with a subscription, she can create a slot first on the
publisher with inactive_timeout value and then associate such a slot
with a required subscription.

-- 
With Regards,
Amit Kapila.

  85. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-21T05:23:54Z

    On Thu, Mar 21, 2024 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> I also don't see any obvious problem with such an API. However, this
> is not a good time to invent new APIs. Let's keep the feature simple
> and then we can extend it in the next version after more discussion
> and probably by that time we will get some feedback from the field as
> well.

I couldn't agree more.

> > > But the issue is that it would make it inconsistent with the new inactivetimeout
> > > in the subscription that is added in "v12-0005".
> >
> > Can you please elaborate what the inconsistency it causes with inactivetimeout?
> >
> I think the inconsistency can arise from the fact that on publisher
> one can change the inactive_timeout for the slot corresponding to a
> subscription but the subscriber won't know, so it will still show the
> old value.

Understood.

> If we want we can document this as a limitation and let
> users be aware of it. However, I feel at this stage, let's not even
> expose this from the subscription or maybe we can discuss it once/if
> we are done with other patches. Anyway, if one wants to use this
> feature with a subscription, she can create a slot first on the
> publisher with inactive_timeout value and then associate such a slot
> with a required subscription.

If we are not exposing it via subscription (meaning, we don't consider
v13-0004 and v13-0005 patches), I feel we can have a new SQL API
pg_alter_replication_slot(int inactive_timeout) for now just altering
the inactive_timeout of a given slot.

With this approach, one can do either of the following:
1) Create a slot with SQL API with inactive_timeout set, and use it
for subscriptions or for streaming standbys.
2) Create a slot with SQL API without inactive_timeout set, use it for
subscriptions or for streaming standbys, and set inactive_timeout
later via pg_alter_replication_slot() depending on how the slot is
consumed
3) Create a subscription with create_slot=true, and set
inactive_timeout via pg_alter_replication_slot() depending on how the
slot is consumed.

This approach seems consistent and minimal to start with.

If we agree on this, I'll drop both 0004 and 0005 that are allowing
inactive_timeout to be set via replication commands and via
create/alter subscription respectively, and implement
pg_alter_replication_slot().

FWIW, adding the new SQL API pg_alter_replication_slot() isn't that hard.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  86. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-21T05:25:31Z

    On Thu, Mar 21, 2024 at 8:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> > >
> > > 2. last_inactive_at and inactive_timeout are now tracked in on-disk
> > > replication slot data structure.
> >
> > Should last_inactive_at be tracked on disk? Say the engine is down for a period
> > of time > inactive_timeout then the slot will be invalidated after the engine
> > re-start (if no activity before we invalidate the slot). Should the time the
> > engine is down be counted as "inactive" time? I've the feeling it should not, and
> > that we should only take into account inactive time while the engine is up.
> >
>
> Good point. The question is how do we achieve this without persisting
> the 'last_inactive_at'? Say, 'last_inactive_at' for a particular slot
> had some valid value before we shut down but it still didn't cross the
> configured 'inactive_timeout' value, so, we won't be able to
> invalidate it. Now, after the restart, as we don't know the
> last_inactive_at's value before the shutdown, we will initialize it
> with 0 (this is what Bharath seems to have done in the latest
> v13-0002* patch). After this, even if walsender or backend never
> acquires the slot, we won't invalidate it. OTOH, if we track
> 'last_inactive_at' on the disk, after, restart, we could initialize it
> to the current time if the value is non-zero. Do you have any better
> ideas?

This sounds reasonable to me at least.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  87. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-21T05:53:48Z

    Hi,

On Thu, Mar 21, 2024 at 08:47:18AM +0530, Amit Kapila wrote:
> On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> > >
> > > 2. last_inactive_at and inactive_timeout are now tracked in on-disk
> > > replication slot data structure.
> >
> > Should last_inactive_at be tracked on disk? Say the engine is down for a period
> > of time > inactive_timeout then the slot will be invalidated after the engine
> > re-start (if no activity before we invalidate the slot). Should the time the
> > engine is down be counted as "inactive" time? I've the feeling it should not, and
> > that we should only take into account inactive time while the engine is up.
> >
> 
> Good point. The question is how do we achieve this without persisting
> the 'last_inactive_at'? Say, 'last_inactive_at' for a particular slot
> had some valid value before we shut down but it still didn't cross the
> configured 'inactive_timeout' value, so, we won't be able to
> invalidate it. Now, after the restart, as we don't know the
> last_inactive_at's value before the shutdown, we will initialize it
> with 0 (this is what Bharath seems to have done in the latest
> v13-0002* patch). After this, even if walsender or backend never
> acquires the slot, we won't invalidate it. OTOH, if we track
> 'last_inactive_at' on the disk, after, restart, we could initialize it
> to the current time if the value is non-zero. Do you have any better
> ideas?
> 

I think that setting last_inactive_at when we restart makes sense if the slot
has been active previously. I think the idea is because it's holding xmin/catalog_xmin
and that we don't want to prevent rows removal longer that the timeout.

So what about relying on xmin/catalog_xmin instead that way?

- For physical slots if xmin is set then set last_inactive_at to the current
time at restart (else zero).

- For logical slot, it's not the same as the catalog_xmin is set at the slot
creation time. So what about setting last_inactive_at at the current time at 
restart but also at creation time for logical slot? (Setting it to zero at
creation time (as we do in v13) does not look right, given the fact that it's
"already" holding a catalog_xmin).

That way, we'd ensure that we are not holding rows for longer that the timeout
and we don't need to persist last_inactive_at.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  88. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-21T06:07:24Z

    Hi,

On Thu, Mar 21, 2024 at 10:53:54AM +0530, Bharath Rupireddy wrote:
> On Thu, Mar 21, 2024 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > But the issue is that it would make it inconsistent with the new inactivetimeout
> > > > in the subscription that is added in "v12-0005".
> > >
> > > Can you please elaborate what the inconsistency it causes with inactivetimeout?
> > >
> > I think the inconsistency can arise from the fact that on publisher
> > one can change the inactive_timeout for the slot corresponding to a
> > subscription but the subscriber won't know, so it will still show the
> > old value.

Yeah, that was what I had in mind.

> > If we want we can document this as a limitation and let
> > users be aware of it. However, I feel at this stage, let's not even
> > expose this from the subscription or maybe we can discuss it once/if
> > we are done with other patches.

I agree, it's important to expose it for things like "failover" but I think we
can get rid of it for the timeout one.

>>  Anyway, if one wants to use this
> > feature with a subscription, she can create a slot first on the
> > publisher with inactive_timeout value and then associate such a slot
> > with a required subscription.

Right.

> 
> If we are not exposing it via subscription (meaning, we don't consider
> v13-0004 and v13-0005 patches), I feel we can have a new SQL API
> pg_alter_replication_slot(int inactive_timeout) for now just altering
> the inactive_timeout of a given slot.

Agree, that seems more "natural" that going through a replication connection.

> With this approach, one can do either of the following:
> 1) Create a slot with SQL API with inactive_timeout set, and use it
> for subscriptions or for streaming standbys.

Yes.

> 2) Create a slot with SQL API without inactive_timeout set, use it for
> subscriptions or for streaming standbys, and set inactive_timeout
> later via pg_alter_replication_slot() depending on how the slot is
> consumed

Yes.

> 3) Create a subscription with create_slot=true, and set
> inactive_timeout via pg_alter_replication_slot() depending on how the
> slot is consumed.

Yes.

We could also do the above 3 and altering the timeout with a replication
connection but the SQL API seems more natural to me.

> 
> This approach seems consistent and minimal to start with.
> 
> If we agree on this, I'll drop both 0004 and 0005 that are allowing
> inactive_timeout to be set via replication commands and via
> create/alter subscription respectively, and implement
> pg_alter_replication_slot().

+1 on this.

> FWIW, adding the new SQL API pg_alter_replication_slot() isn't that hard.

Also I think we should ensure that one could "only" alter the timeout property
for the time being (if not that could lead to the subscription inconsistency 
mentioned above).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  89. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-21T06:13:54Z

    On Thu, Mar 21, 2024 at 11:23 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Thu, Mar 21, 2024 at 08:47:18AM +0530, Amit Kapila wrote:
> > On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> > > >
> > > > 2. last_inactive_at and inactive_timeout are now tracked in on-disk
> > > > replication slot data structure.
> > >
> > > Should last_inactive_at be tracked on disk? Say the engine is down for a period
> > > of time > inactive_timeout then the slot will be invalidated after the engine
> > > re-start (if no activity before we invalidate the slot). Should the time the
> > > engine is down be counted as "inactive" time? I've the feeling it should not, and
> > > that we should only take into account inactive time while the engine is up.
> > >
> >
> > Good point. The question is how do we achieve this without persisting
> > the 'last_inactive_at'? Say, 'last_inactive_at' for a particular slot
> > had some valid value before we shut down but it still didn't cross the
> > configured 'inactive_timeout' value, so, we won't be able to
> > invalidate it. Now, after the restart, as we don't know the
> > last_inactive_at's value before the shutdown, we will initialize it
> > with 0 (this is what Bharath seems to have done in the latest
> > v13-0002* patch). After this, even if walsender or backend never
> > acquires the slot, we won't invalidate it. OTOH, if we track
> > 'last_inactive_at' on the disk, after, restart, we could initialize it
> > to the current time if the value is non-zero. Do you have any better
> > ideas?
> >
>
> I think that setting last_inactive_at when we restart makes sense if the slot
> has been active previously. I think the idea is because it's holding xmin/catalog_xmin
> and that we don't want to prevent rows removal longer that the timeout.
>
> So what about relying on xmin/catalog_xmin instead that way?
>

That doesn't sound like a great idea because xmin/catalog_xmin values
won't tell us before restart whether it was active or not. It could
have been inactive for long time before restart but the xmin values
could still be valid. What about we always set 'last_inactive_at' at
restart (if the slot's inactive_timeout has non-zero value) and reset
it as soon as someone acquires that slot? Now, if the slot doesn't get
acquired till 'inactive_timeout', checkpointer will invalidate the
slot.

-- 
With Regards,
Amit Kapila.

  90. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-21T06:23:32Z

    On Thu, Mar 21, 2024 at 11:37 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Thu, Mar 21, 2024 at 10:53:54AM +0530, Bharath Rupireddy wrote:
> > On Thu, Mar 21, 2024 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > But the issue is that it would make it inconsistent with the new inactivetimeout
> > > > > in the subscription that is added in "v12-0005".
> > > >
> > > > Can you please elaborate what the inconsistency it causes with inactivetimeout?
> > > >
> > > I think the inconsistency can arise from the fact that on publisher
> > > one can change the inactive_timeout for the slot corresponding to a
> > > subscription but the subscriber won't know, so it will still show the
> > > old value.
>
> Yeah, that was what I had in mind.
>
> > > If we want we can document this as a limitation and let
> > > users be aware of it. However, I feel at this stage, let's not even
> > > expose this from the subscription or maybe we can discuss it once/if
> > > we are done with other patches.
>
> I agree, it's important to expose it for things like "failover" but I think we
> can get rid of it for the timeout one.
>
> >>  Anyway, if one wants to use this
> > > feature with a subscription, she can create a slot first on the
> > > publisher with inactive_timeout value and then associate such a slot
> > > with a required subscription.
>
> Right.
>
> >
> > If we are not exposing it via subscription (meaning, we don't consider
> > v13-0004 and v13-0005 patches), I feel we can have a new SQL API
> > pg_alter_replication_slot(int inactive_timeout) for now just altering
> > the inactive_timeout of a given slot.
>
> Agree, that seems more "natural" that going through a replication connection.
>
> > With this approach, one can do either of the following:
> > 1) Create a slot with SQL API with inactive_timeout set, and use it
> > for subscriptions or for streaming standbys.
>
> Yes.
>
> > 2) Create a slot with SQL API without inactive_timeout set, use it for
> > subscriptions or for streaming standbys, and set inactive_timeout
> > later via pg_alter_replication_slot() depending on how the slot is
> > consumed
>
> Yes.
>
> > 3) Create a subscription with create_slot=true, and set
> > inactive_timeout via pg_alter_replication_slot() depending on how the
> > slot is consumed.
>
> Yes.
>
> We could also do the above 3 and altering the timeout with a replication
> connection but the SQL API seems more natural to me.
>

If we want to go with this then I think we should at least ensure that
if one specified timeout via CREATE_REPLICATION_SLOT or
ALTER_REPLICATION_SLOT that should be honored.

-- 
With Regards,
Amit Kapila.

  91. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-21T06:45:28Z

    Hi,

On Thu, Mar 21, 2024 at 11:43:54AM +0530, Amit Kapila wrote:
> On Thu, Mar 21, 2024 at 11:23 AM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Thu, Mar 21, 2024 at 08:47:18AM +0530, Amit Kapila wrote:
> > > On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot
> > > <bertranddrouvot.pg@gmail.com> wrote:
> > > >
> > > > On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> > > > >
> > > > > 2. last_inactive_at and inactive_timeout are now tracked in on-disk
> > > > > replication slot data structure.
> > > >
> > > > Should last_inactive_at be tracked on disk? Say the engine is down for a period
> > > > of time > inactive_timeout then the slot will be invalidated after the engine
> > > > re-start (if no activity before we invalidate the slot). Should the time the
> > > > engine is down be counted as "inactive" time? I've the feeling it should not, and
> > > > that we should only take into account inactive time while the engine is up.
> > > >
> > >
> > > Good point. The question is how do we achieve this without persisting
> > > the 'last_inactive_at'? Say, 'last_inactive_at' for a particular slot
> > > had some valid value before we shut down but it still didn't cross the
> > > configured 'inactive_timeout' value, so, we won't be able to
> > > invalidate it. Now, after the restart, as we don't know the
> > > last_inactive_at's value before the shutdown, we will initialize it
> > > with 0 (this is what Bharath seems to have done in the latest
> > > v13-0002* patch). After this, even if walsender or backend never
> > > acquires the slot, we won't invalidate it. OTOH, if we track
> > > 'last_inactive_at' on the disk, after, restart, we could initialize it
> > > to the current time if the value is non-zero. Do you have any better
> > > ideas?
> > >
> >
> > I think that setting last_inactive_at when we restart makes sense if the slot
> > has been active previously. I think the idea is because it's holding xmin/catalog_xmin
> > and that we don't want to prevent rows removal longer that the timeout.
> >
> > So what about relying on xmin/catalog_xmin instead that way?
> >
> 
> That doesn't sound like a great idea because xmin/catalog_xmin values
> won't tell us before restart whether it was active or not. It could
> have been inactive for long time before restart but the xmin values
> could still be valid.

Right, the idea here was more like "don't hold xmin/catalog_xmin" for longer
than timeout.

My concern was that we set catalog_xmin at logical slot creation time. So if we
set last_inactive_at to zero at creation time and the slot is not used for a long
period of time > timeout, then I think it's not helping there.

> What about we always set 'last_inactive_at' at
> restart (if the slot's inactive_timeout has non-zero value) and reset
> it as soon as someone acquires that slot? Now, if the slot doesn't get
> acquired till 'inactive_timeout', checkpointer will invalidate the
> slot.

Yeah that sounds good to me, but I think we should set last_inactive_at at creation
time too, if not:

- physical slot could remain valid for long time after creation (which is fine)
but the behavior would change at restart.
- logical slot would have the "issue" reported above (holding catalog_xmin). 

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  92. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-21T06:50:12Z

    Hi,

On Thu, Mar 21, 2024 at 11:53:32AM +0530, Amit Kapila wrote:
> On Thu, Mar 21, 2024 at 11:37 AM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > We could also do the above 3 and altering the timeout with a replication
> > connection but the SQL API seems more natural to me.
> >
> 
> If we want to go with this then I think we should at least ensure that
> if one specified timeout via CREATE_REPLICATION_SLOT or
> ALTER_REPLICATION_SLOT that should be honored.

Yeah, agree.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  93. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-21T07:10:16Z

    Hi,

On Thu, Mar 21, 2024 at 05:05:46AM +0530, Bharath Rupireddy wrote:
> On Wed, Mar 20, 2024 at 1:04 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Wed, Mar 20, 2024 at 08:58:05AM +0530, Amit Kapila wrote:
> > > On Wed, Mar 20, 2024 at 12:49 AM Bharath Rupireddy
> > > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > > >
> > > > Following are some open points:
> > > >
> > > > 1. Where to do inactive_timeout invalidation exactly if not the checkpointer.
> > > >
> > > I have suggested to do it at the time of CheckpointReplicationSlots()
> > > and Bertrand suggested to do it whenever we resume using the slot. I
> > > think we should follow both the suggestions.
> >
> > Agree. I also think that pg_get_replication_slots() would be a good place, so
> > that queries would return the right invalidation status.
> 
> I've addressed review comments and attaching the v13 patches with the
> following changes:

Thanks!

v13-0001 looks good to me. The only Nit (that I've mentioned up-thread) is that
in the pg_replication_slots view, the invalidation_reason is "far away" from the
conflicting field. I understand that one could query the fields individually but
when describing the view or reading the doc, it seems more appropriate to see
them closer. Also as "failover" and "synced" are also new in version 17, there
is no risk to break order by "17,18" kind of queries (which are the failover
and sync positions).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  94. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-21T09:13:46Z

    On Thu, Mar 21, 2024 at 12:40 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> v13-0001 looks good to me. The only Nit (that I've mentioned up-thread) is that
> in the pg_replication_slots view, the invalidation_reason is "far away" from the
> conflicting field. I understand that one could query the fields individually but
> when describing the view or reading the doc, it seems more appropriate to see
> them closer. Also as "failover" and "synced" are also new in version 17, there
> is no risk to break order by "17,18" kind of queries (which are the failover
> and sync positions).

Hm, yeah, I can change that in the next version of the patches. Thanks.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  95. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-21T09:50:01Z

    On Thu, Mar 21, 2024 at 12:15 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Thu, Mar 21, 2024 at 11:43:54AM +0530, Amit Kapila wrote:
> > On Thu, Mar 21, 2024 at 11:23 AM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > On Thu, Mar 21, 2024 at 08:47:18AM +0530, Amit Kapila wrote:
> > > > On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot
> > > > <bertranddrouvot.pg@gmail.com> wrote:
> > > > >
> > > > > On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote:
> > > > > >
> > > > > > 2. last_inactive_at and inactive_timeout are now tracked in on-disk
> > > > > > replication slot data structure.
> > > > >
> > > > > Should last_inactive_at be tracked on disk? Say the engine is down for a period
> > > > > of time > inactive_timeout then the slot will be invalidated after the engine
> > > > > re-start (if no activity before we invalidate the slot). Should the time the
> > > > > engine is down be counted as "inactive" time? I've the feeling it should not, and
> > > > > that we should only take into account inactive time while the engine is up.
> > > > >
> > > >
> > > > Good point. The question is how do we achieve this without persisting
> > > > the 'last_inactive_at'? Say, 'last_inactive_at' for a particular slot
> > > > had some valid value before we shut down but it still didn't cross the
> > > > configured 'inactive_timeout' value, so, we won't be able to
> > > > invalidate it. Now, after the restart, as we don't know the
> > > > last_inactive_at's value before the shutdown, we will initialize it
> > > > with 0 (this is what Bharath seems to have done in the latest
> > > > v13-0002* patch). After this, even if walsender or backend never
> > > > acquires the slot, we won't invalidate it. OTOH, if we track
> > > > 'last_inactive_at' on the disk, after, restart, we could initialize it
> > > > to the current time if the value is non-zero. Do you have any better
> > > > ideas?
> > > >
> > >
> > > I think that setting last_inactive_at when we restart makes sense if the slot
> > > has been active previously. I think the idea is because it's holding xmin/catalog_xmin
> > > and that we don't want to prevent rows removal longer that the timeout.
> > >
> > > So what about relying on xmin/catalog_xmin instead that way?
> > >
> >
> > That doesn't sound like a great idea because xmin/catalog_xmin values
> > won't tell us before restart whether it was active or not. It could
> > have been inactive for long time before restart but the xmin values
> > could still be valid.
>
> Right, the idea here was more like "don't hold xmin/catalog_xmin" for longer
> than timeout.
>
> My concern was that we set catalog_xmin at logical slot creation time. So if we
> set last_inactive_at to zero at creation time and the slot is not used for a long
> period of time > timeout, then I think it's not helping there.
>

But, we do call ReplicationSlotRelease() after slot creation. For
example, see CreateReplicationSlot(). So wouldn't that take care of
the case you are worried about?

-- 
With Regards,
Amit Kapila.

  96. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-21T10:43:31Z

    On Thu, Mar 21, 2024 at 3:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > My concern was that we set catalog_xmin at logical slot creation time. So if we
> > set last_inactive_at to zero at creation time and the slot is not used for a long
> > period of time > timeout, then I think it's not helping there.
>
> But, we do call ReplicationSlotRelease() after slot creation. For
> example, see CreateReplicationSlot(). So wouldn't that take care of
> the case you are worried about?

Right. That's true even for pg_create_physical_replication_slot and
pg_create_logical_replication_slot. AFAICS, setting it to the current
timestamp in ReplicationSlotRelease suffices unless I'm missing
something.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  97. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-21T10:55:46Z

    On Thu, Mar 21, 2024 at 2:44 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Thu, Mar 21, 2024 at 12:40 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > v13-0001 looks good to me. The only Nit (that I've mentioned up-thread) is that
> > in the pg_replication_slots view, the invalidation_reason is "far away" from the
> > conflicting field. I understand that one could query the fields individually but
> > when describing the view or reading the doc, it seems more appropriate to see
> > them closer. Also as "failover" and "synced" are also new in version 17, there
> > is no risk to break order by "17,18" kind of queries (which are the failover
> > and sync positions).
>
> Hm, yeah, I can change that in the next version of the patches. Thanks.
>

This makes sense to me. Apart from this, few more comments on 0001.
1.
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -676,13 +676,13 @@ get_old_cluster_logical_slot_infos(DbInfo
*dbinfo, bool live_check)
  * removed.
  */
  res = executeQueryOrDie(conn, "SELECT slot_name, plugin, two_phase,
failover, "
- "%s as caught_up, conflict_reason IS NOT NULL as invalid "
+ "%s as caught_up, invalidation_reason IS NOT NULL as invalid "
  "FROM pg_catalog.pg_replication_slots "
  "WHERE slot_type = 'logical' AND "
  "database = current_database() AND "
  "temporary IS FALSE;",
  live_check ? "FALSE" :
- "(CASE WHEN conflict_reason IS NOT NULL THEN FALSE "
+ "(CASE WHEN conflicting THEN FALSE "

I think here at both places we need to change 'conflict_reason' to
'conflicting'.

2.
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>invalidation_reason</structfield> <type>text</type>
+      </para>
+      <para>
+       The reason for the slot's invalidation. It is set for both logical and
+       physical slots. <literal>NULL</literal> if the slot is not invalidated.
+       Possible values are:
+       <itemizedlist spacing="compact">
+        <listitem>
+         <para>
+          <literal>wal_removed</literal> means that the required WAL has been
+          removed.
+         </para>
+        </listitem>
+        <listitem>
+         <para>
+          <literal>rows_removed</literal> means that the required rows have
+          been removed.
+         </para>
+        </listitem>
+        <listitem>
+         <para>
+          <literal>wal_level_insufficient</literal> means that the
+          primary doesn't have a <xref linkend="guc-wal-level"/> sufficient to
+          perform logical decoding.
+         </para>

Can the reasons 'rows_removed' and 'wal_level_insufficient' appear for
physical slots? If not, then it is not clear from above text.

3.
-# Verify slots are reported as non conflicting in pg_replication_slots
+# Verify slots are reported as valid in pg_replication_slots
 is( $node_standby->safe_psql(
  'postgres',
  q[select bool_or(conflicting) from
-   (select conflict_reason is not NULL as conflicting
-    from pg_replication_slots WHERE slot_type = 'logical')]),
+   (select conflicting from pg_replication_slots
+ where slot_type = 'logical')]),
  'f',
- 'Logical slots are reported as non conflicting');
+ 'Logical slots are reported as valid');

I don't think we need to change the comment or success message in this test.

-- 
With Regards,
Amit Kapila.

  98. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-21T11:20:50Z

    Hi,

On Thu, Mar 21, 2024 at 04:13:31PM +0530, Bharath Rupireddy wrote:
> On Thu, Mar 21, 2024 at 3:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > My concern was that we set catalog_xmin at logical slot creation time. So if we
> > > set last_inactive_at to zero at creation time and the slot is not used for a long
> > > period of time > timeout, then I think it's not helping there.
> >
> > But, we do call ReplicationSlotRelease() after slot creation. For
> > example, see CreateReplicationSlot(). So wouldn't that take care of
> > the case you are worried about?
> 
> Right. That's true even for pg_create_physical_replication_slot and
> pg_create_logical_replication_slot. AFAICS, setting it to the current
> timestamp in ReplicationSlotRelease suffices unless I'm missing
> something.

Right, but we have:

"
    if (set_last_inactive_at &&
        slot->data.persistency == RS_PERSISTENT)
    {
        /*
         * There's no point in allowing failover slots to get invalidated
         * based on slot's inactive_timeout parameter on standby. The failover
         * slots simply get synced from the primary on the standby.
         */
        if (!(RecoveryInProgress() && slot->data.failover))
        {
            SpinLockAcquire(&slot->mutex);
            slot->last_inactive_at = GetCurrentTimestamp();
            SpinLockRelease(&slot->mutex);
        }
    }
"

while we set set_last_inactive_at to false at creation time so that last_inactive_at
is not set to GetCurrentTimestamp(). We should set set_last_inactive_at to true
if a timeout is provided during the slot creation.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  99. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-21T17:51:03Z

    On Thu, Mar 21, 2024 at 4:25 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> This makes sense to me. Apart from this, few more comments on 0001.

Thanks for looking into it.

> 1.
> - "%s as caught_up, conflict_reason IS NOT NULL as invalid "
> + "%s as caught_up, invalidation_reason IS NOT NULL as invalid "
>   live_check ? "FALSE" :
> - "(CASE WHEN conflict_reason IS NOT NULL THEN FALSE "
> + "(CASE WHEN conflicting THEN FALSE "
>
> I think here at both places we need to change 'conflict_reason' to
> 'conflicting'.

Basically, the idea there is to not live_check for invalidated logical
slots. It has nothing to do with conflicting. Up until now,
conflict_reason is also reporting wal_removed (although wrongly
including rows_removed, wal_level_insufficient, the two reasons for
conflicts). So, I think invalidation_reason is right for invalid
column. Also, I think we need to change conflicting to
invalidation_reason for live_check. So, I've changed that to use
invalidation_reason for both columns.

> 2.
>
> Can the reasons 'rows_removed' and 'wal_level_insufficient' appear for
> physical slots?

No. They can only occur for logical slots, check
InvalidatePossiblyObsoleteSlot, only the logical slots get
invalidated.

> If not, then it is not clear from above text.

I've stated that "It is set only for logical slots." for rows_removed
and wal_level_insufficient. Other reasons can occur for both slots.

> 3.
> -# Verify slots are reported as non conflicting in pg_replication_slots
> +# Verify slots are reported as valid in pg_replication_slots
>  is( $node_standby->safe_psql(
>   'postgres',
>   q[select bool_or(conflicting) from
> -   (select conflict_reason is not NULL as conflicting
> -    from pg_replication_slots WHERE slot_type = 'logical')]),
> +   (select conflicting from pg_replication_slots
> + where slot_type = 'logical')]),
>   'f',
> - 'Logical slots are reported as non conflicting');
> + 'Logical slots are reported as valid');
>
> I don't think we need to change the comment or success message in this test.

Yes. There the intention of the test case is to verify logical slots
are reported as non conflicting. So, I changed them.

Please find the v14-0001 patch for now. I'll post the other patches soon.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  100. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-22T05:19:17Z

    On Thu, Mar 21, 2024 at 11:21 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
>
> Please find the v14-0001 patch for now. I'll post the other patches soon.
>

LGTM. Let's wait for Bertrand to see if he has more comments on 0001
and then I'll push it.

-- 
With Regards,
Amit Kapila.

  101. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-22T07:09:14Z

    Hi,

On Fri, Mar 22, 2024 at 10:49:17AM +0530, Amit Kapila wrote:
> On Thu, Mar 21, 2024 at 11:21 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> >
> > Please find the v14-0001 patch for now.

Thanks!

> LGTM. Let's wait for Bertrand to see if he has more comments on 0001
> and then I'll push it.

LGTM too.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  102. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-22T08:15:01Z

    On Fri, Mar 22, 2024 at 12:39 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > > Please find the v14-0001 patch for now.
>
> Thanks!
>
> > LGTM. Let's wait for Bertrand to see if he has more comments on 0001
> > and then I'll push it.
>
> LGTM too.

Thanks. Here I'm implementing the following:

0001 Track invalidation_reason in pg_replication_slots
0002 Track last_inactive_at in pg_replication_slots
0003 Allow setting inactive_timeout for replication slots via SQL API
0004 Introduce new SQL funtion pg_alter_replication_slot
0005 Allow setting inactive_timeout in the replication command
0006 Add inactive_timeout based replication slot invalidation

1. Keep it last_inactive_at as a shared memory variable, but always
set it at restart if the slot's inactive_timeout has non-zero value
and reset it as soon as someone acquires that slot so that if the slot
doesn't get acquired  till inactive_timeout, checkpointer will
invalidate the slot.
2. Ensure with pg_alter_replication_slot one could "only" alter the
timeout property for the time being, if not that could lead to the
subscription inconsistency.
3. Have some notes in the CREATE and ALTER SUBSCRIPTION docs about
using an existing slot to leverage inactive_timeout feature.
4. last_inactive_at should also be set to the current time during slot
creation because if one creates a slot and does nothing with it then
it's the time it starts to be inactive.
5. We don't set last_inactive_at to GetCurrentTimestamp() for failover slots.
6. Leave the patch that added support for inactive_timeout in subscriptions.

Please see the attached v14 patch set. No change in the attached
v14-0001 from the previous patch.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  103. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-22T08:57:33Z

    Hi,

On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
> On Fri, Mar 22, 2024 at 12:39 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > > Please find the v14-0001 patch for now.
> >
> > Thanks!
> >
> > > LGTM. Let's wait for Bertrand to see if he has more comments on 0001
> > > and then I'll push it.
> >
> > LGTM too.
> 
> Thanks. Here I'm implementing the following:

Thanks!

> 0001 Track invalidation_reason in pg_replication_slots
> 0002 Track last_inactive_at in pg_replication_slots
> 0003 Allow setting inactive_timeout for replication slots via SQL API
> 0004 Introduce new SQL funtion pg_alter_replication_slot
> 0005 Allow setting inactive_timeout in the replication command
> 0006 Add inactive_timeout based replication slot invalidation
> 
> 1. Keep it last_inactive_at as a shared memory variable, but always
> set it at restart if the slot's inactive_timeout has non-zero value
> and reset it as soon as someone acquires that slot so that if the slot
> doesn't get acquired  till inactive_timeout, checkpointer will
> invalidate the slot.
> 4. last_inactive_at should also be set to the current time during slot
> creation because if one creates a slot and does nothing with it then
> it's the time it starts to be inactive.

I did not look at the code yet but just tested the behavior. It works as you
describe it but I think this behavior is weird because:

- when we create a slot without a timeout then last_inactive_at is set. I think
that's fine, but then:
- when we restart the engine, then last_inactive_at is gone (as timeout is not
set).

I think last_inactive_at should be set also at engine restart even if there is
no timeout. I don't think we should link both. Changing my mind here on this
subject due to the testing.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  104. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-22T09:29:21Z

    On Fri, Mar 22, 2024 at 2:27 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
>
> > 0001 Track invalidation_reason in pg_replication_slots
> > 0002 Track last_inactive_at in pg_replication_slots
> > 0003 Allow setting inactive_timeout for replication slots via SQL API
> > 0004 Introduce new SQL funtion pg_alter_replication_slot
> > 0005 Allow setting inactive_timeout in the replication command
> > 0006 Add inactive_timeout based replication slot invalidation
> >
> > 1. Keep it last_inactive_at as a shared memory variable, but always
> > set it at restart if the slot's inactive_timeout has non-zero value
> > and reset it as soon as someone acquires that slot so that if the slot
> > doesn't get acquired  till inactive_timeout, checkpointer will
> > invalidate the slot.
> > 4. last_inactive_at should also be set to the current time during slot
> > creation because if one creates a slot and does nothing with it then
> > it's the time it starts to be inactive.
>
> I did not look at the code yet but just tested the behavior. It works as you
> describe it but I think this behavior is weird because:
>
> - when we create a slot without a timeout then last_inactive_at is set. I think
> that's fine, but then:
> - when we restart the engine, then last_inactive_at is gone (as timeout is not
> set).
>
> I think last_inactive_at should be set also at engine restart even if there is
> no timeout.

I think it is the opposite. Why do we need to set  'last_inactive_at'
when inactive_timeout is not set? BTW, haven't we discussed that we
don't need to set 'last_inactive_at' at the time of slot creation as
it is sufficient to set it at the time ReplicationSlotRelease()?

A few other comments:
==================
1.
@@ -1027,7 +1027,8 @@ CREATE VIEW pg_replication_slots AS
             L.invalidation_reason,
             L.failover,
             L.synced,
-            L.last_inactive_at
+            L.last_inactive_at,
+            L.inactive_timeout

I think it would be better to keep 'inactive_timeout' ahead of
'last_inactive_at' as that is the primary field. In major versions, we
don't have to strictly keep the new fields at the end. In this case,
it seems better to keep these two new fields after two_phase so that
these are before invalidation_reason where we can show the
invalidation due to these fields.

2.
 void
-ReplicationSlotRelease(void)
+ReplicationSlotRelease(bool set_last_inactive_at)

Why do we need a parameter here? Can't we directly check from the slot
whether 'inactive_timeout' has a non-zero value?

3.
+ /*
+ * There's no point in allowing failover slots to get invalidated
+ * based on slot's inactive_timeout parameter on standby. The failover
+ * slots simply get synced from the primary on the standby.
+ */
+ if (!(RecoveryInProgress() && slot->data.failover))

I think you need to check 'sync' flag instead of 'failover'.
Generally, failover marker slots should be invalidated either on
primary or standby unless on standby the 'failover' marked slot is
synced from the primary.

4. I feel the patches should be arranged like 0003->0001, 0002->0002,
0006->0003. We can leave remaining for the time being till we get
these three patches (all three need to be committed as one but it is
okay to keep them separate for review) committed.

-- 
With Regards,
Amit Kapila.

  105. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-22T09:45:29Z

    Hi,

On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
> On Fri, Mar 22, 2024 at 12:39 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > > Please find the v14-0001 patch for now.
> >
> > Thanks!
> >
> > > LGTM. Let's wait for Bertrand to see if he has more comments on 0001
> > > and then I'll push it.
> >
> > LGTM too.
> 
> 
> Please see the attached v14 patch set. No change in the attached
> v14-0001 from the previous patch.

Looking at v14-0002:

1 ===

@@ -691,6 +699,13 @@ ReplicationSlotRelease(void)
                ConditionVariableBroadcast(&slot->active_cv);
        }

+       if (slot->data.persistency == RS_PERSISTENT)
+       {
+               SpinLockAcquire(&slot->mutex);
+               slot->last_inactive_at = GetCurrentTimestamp();
+               SpinLockRelease(&slot->mutex);
+       }

I'm not sure we should do system calls while we're holding a spinlock.
Assign a variable before?

2 ===

Also, what about moving this here?

"
    if (slot->data.persistency == RS_PERSISTENT)
    {
        /*
         * Mark persistent slot inactive.  We're not freeing it, just
         * disconnecting, but wake up others that may be waiting for it.
         */
        SpinLockAcquire(&slot->mutex);
        slot->active_pid = 0;
        SpinLockRelease(&slot->mutex);
        ConditionVariableBroadcast(&slot->active_cv);
    }
"

That would avoid testing twice "slot->data.persistency == RS_PERSISTENT".

3 ===

@@ -2341,6 +2356,7 @@ RestoreSlotFromDisk(const char *name)

                slot->in_use = true;
                slot->active_pid = 0;
+               slot->last_inactive_at = 0;

I think we should put GetCurrentTimestamp() here. It's done in v14-0006 but I
think it's better to do it in 0002 (and not taking care of inactive_timeout).

4 ===

    Track last_inactive_at in pg_replication_slots

 doc/src/sgml/system-views.sgml       | 11 +++++++++++
 src/backend/catalog/system_views.sql |  3 ++-
 src/backend/replication/slot.c       | 16 ++++++++++++++++
 src/backend/replication/slotfuncs.c  |  7 ++++++-
 src/include/catalog/pg_proc.dat      |  6 +++---
 src/include/replication/slot.h       |  3 +++
 src/test/regress/expected/rules.out  |  5 +++--
 7 files changed, 44 insertions(+), 7 deletions(-)

Worth to add some tests too (or we postpone them in future commits because we're
confident enough they will follow soon)?

5 ===

Most of the fields that reflect a time (not duration) in the system views are
xxxx_time, so I'm wondering if instead of "last_inactive_at" we should use
something like "last_inactive_time"?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  106. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-22T09:53:13Z

    Hi,

On Fri, Mar 22, 2024 at 02:59:21PM +0530, Amit Kapila wrote:
> On Fri, Mar 22, 2024 at 2:27 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
> >
> > > 0001 Track invalidation_reason in pg_replication_slots
> > > 0002 Track last_inactive_at in pg_replication_slots
> > > 0003 Allow setting inactive_timeout for replication slots via SQL API
> > > 0004 Introduce new SQL funtion pg_alter_replication_slot
> > > 0005 Allow setting inactive_timeout in the replication command
> > > 0006 Add inactive_timeout based replication slot invalidation
> > >
> > > 1. Keep it last_inactive_at as a shared memory variable, but always
> > > set it at restart if the slot's inactive_timeout has non-zero value
> > > and reset it as soon as someone acquires that slot so that if the slot
> > > doesn't get acquired  till inactive_timeout, checkpointer will
> > > invalidate the slot.
> > > 4. last_inactive_at should also be set to the current time during slot
> > > creation because if one creates a slot and does nothing with it then
> > > it's the time it starts to be inactive.
> >
> > I did not look at the code yet but just tested the behavior. It works as you
> > describe it but I think this behavior is weird because:
> >
> > - when we create a slot without a timeout then last_inactive_at is set. I think
> > that's fine, but then:
> > - when we restart the engine, then last_inactive_at is gone (as timeout is not
> > set).
> >
> > I think last_inactive_at should be set also at engine restart even if there is
> > no timeout.
> 
> I think it is the opposite. Why do we need to set  'last_inactive_at'
> when inactive_timeout is not set?

I think those are unrelated, one could want to know when a slot has been inactive
even if no timeout is set. I understand that for this patch series we have in mind 
to use them both to invalidate slots but I think that there is use case to not
use both in correlation. Also not setting last_inactive_at could give the "false"
impression that the slot is active.

> BTW, haven't we discussed that we
> don't need to set 'last_inactive_at' at the time of slot creation as
> it is sufficient to set it at the time ReplicationSlotRelease()?

Right.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  107. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-22T10:26:23Z

    On Fri, Mar 22, 2024 at 3:15 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
>
> 1 ===
>
> @@ -691,6 +699,13 @@ ReplicationSlotRelease(void)
>                 ConditionVariableBroadcast(&slot->active_cv);
>         }
>
> +       if (slot->data.persistency == RS_PERSISTENT)
> +       {
> +               SpinLockAcquire(&slot->mutex);
> +               slot->last_inactive_at = GetCurrentTimestamp();
> +               SpinLockRelease(&slot->mutex);
> +       }
>
> I'm not sure we should do system calls while we're holding a spinlock.
> Assign a variable before?
>
> 2 ===
>
> Also, what about moving this here?
>
> "
>     if (slot->data.persistency == RS_PERSISTENT)
>     {
>         /*
>          * Mark persistent slot inactive.  We're not freeing it, just
>          * disconnecting, but wake up others that may be waiting for it.
>          */
>         SpinLockAcquire(&slot->mutex);
>         slot->active_pid = 0;
>         SpinLockRelease(&slot->mutex);
>         ConditionVariableBroadcast(&slot->active_cv);
>     }
> "
>
> That would avoid testing twice "slot->data.persistency == RS_PERSISTENT".
>

That sounds like a good idea. Also, don't we need to consider physical
slots where we don't reserve WAL during slot creation? I don't think
there is a need to set inactive_at for such slots. If we agree,
probably checking restart_lsn should suffice the need to know whether
the WAL is reserved or not.

>
> 5 ===
>
> Most of the fields that reflect a time (not duration) in the system views are
> xxxx_time, so I'm wondering if instead of "last_inactive_at" we should use
> something like "last_inactive_time"?
>

How about naming it as last_active_time? This will indicate the time
at which the slot was last active.

-- 
With Regards,
Amit Kapila.

  108. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-22T10:46:19Z

    On Fri, Mar 22, 2024 at 3:23 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 02:59:21PM +0530, Amit Kapila wrote:
> > On Fri, Mar 22, 2024 at 2:27 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
> > >
> > > > 0001 Track invalidation_reason in pg_replication_slots
> > > > 0002 Track last_inactive_at in pg_replication_slots
> > > > 0003 Allow setting inactive_timeout for replication slots via SQL API
> > > > 0004 Introduce new SQL funtion pg_alter_replication_slot
> > > > 0005 Allow setting inactive_timeout in the replication command
> > > > 0006 Add inactive_timeout based replication slot invalidation
> > > >
> > > > 1. Keep it last_inactive_at as a shared memory variable, but always
> > > > set it at restart if the slot's inactive_timeout has non-zero value
> > > > and reset it as soon as someone acquires that slot so that if the slot
> > > > doesn't get acquired  till inactive_timeout, checkpointer will
> > > > invalidate the slot.
> > > > 4. last_inactive_at should also be set to the current time during slot
> > > > creation because if one creates a slot and does nothing with it then
> > > > it's the time it starts to be inactive.
> > >
> > > I did not look at the code yet but just tested the behavior. It works as you
> > > describe it but I think this behavior is weird because:
> > >
> > > - when we create a slot without a timeout then last_inactive_at is set. I think
> > > that's fine, but then:
> > > - when we restart the engine, then last_inactive_at is gone (as timeout is not
> > > set).
> > >
> > > I think last_inactive_at should be set also at engine restart even if there is
> > > no timeout.
> >
> > I think it is the opposite. Why do we need to set  'last_inactive_at'
> > when inactive_timeout is not set?
>
> I think those are unrelated, one could want to know when a slot has been inactive
> even if no timeout is set. I understand that for this patch series we have in mind
> to use them both to invalidate slots but I think that there is use case to not
> use both in correlation. Also not setting last_inactive_at could give the "false"
> impression that the slot is active.
>

I see your point and agree with this. I feel we can commit this part
first then, probably that is the reason Bharath has kept it as a
separate patch. It would be good add the use case for this patch in
the commit message.

A minor comment:

  if (SlotIsLogical(s))
  pgstat_acquire_replslot(s);

+ if (s->data.persistency == RS_PERSISTENT)
+ {
+ SpinLockAcquire(&s->mutex);
+ s->last_inactive_at = 0;
+ SpinLockRelease(&s->mutex);
+ }
+

I think this part of the change needs a comment.

-- 
With Regards,
Amit Kapila.

  109. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-22T12:00:07Z

    Hi,

On Fri, Mar 22, 2024 at 03:56:23PM +0530, Amit Kapila wrote:
> On Fri, Mar 22, 2024 at 3:15 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
> >
> > 1 ===
> >
> > @@ -691,6 +699,13 @@ ReplicationSlotRelease(void)
> >                 ConditionVariableBroadcast(&slot->active_cv);
> >         }
> >
> > +       if (slot->data.persistency == RS_PERSISTENT)
> > +       {
> > +               SpinLockAcquire(&slot->mutex);
> > +               slot->last_inactive_at = GetCurrentTimestamp();
> > +               SpinLockRelease(&slot->mutex);
> > +       }
> >
> > I'm not sure we should do system calls while we're holding a spinlock.
> > Assign a variable before?
> >
> > 2 ===
> >
> > Also, what about moving this here?
> >
> > "
> >     if (slot->data.persistency == RS_PERSISTENT)
> >     {
> >         /*
> >          * Mark persistent slot inactive.  We're not freeing it, just
> >          * disconnecting, but wake up others that may be waiting for it.
> >          */
> >         SpinLockAcquire(&slot->mutex);
> >         slot->active_pid = 0;
> >         SpinLockRelease(&slot->mutex);
> >         ConditionVariableBroadcast(&slot->active_cv);
> >     }
> > "
> >
> > That would avoid testing twice "slot->data.persistency == RS_PERSISTENT".
> >
> 
> That sounds like a good idea. Also, don't we need to consider physical
> slots where we don't reserve WAL during slot creation? I don't think
> there is a need to set inactive_at for such slots.

If the slot is not active, why shouldn't we set inactive_at? I can understand
that such a slots do not present "any risks" but I think we should still set
inactive_at (also to not give the false impression that the slot is active).

> > 5 ===
> >
> > Most of the fields that reflect a time (not duration) in the system views are
> > xxxx_time, so I'm wondering if instead of "last_inactive_at" we should use
> > something like "last_inactive_time"?
> >
> 
> How about naming it as last_active_time? This will indicate the time
> at which the slot was last active.

I thought about it too but I think it could be missleading as one could think that 
it should be updated each time WAL record decoding is happening.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  110. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-22T12:03:36Z

    Hi,

On Fri, Mar 22, 2024 at 04:16:19PM +0530, Amit Kapila wrote:
> On Fri, Mar 22, 2024 at 3:23 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Fri, Mar 22, 2024 at 02:59:21PM +0530, Amit Kapila wrote:
> > > On Fri, Mar 22, 2024 at 2:27 PM Bertrand Drouvot
> > > <bertranddrouvot.pg@gmail.com> wrote:
> > > >
> > > > On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
> > > >
> > > > > 0001 Track invalidation_reason in pg_replication_slots
> > > > > 0002 Track last_inactive_at in pg_replication_slots
> > > > > 0003 Allow setting inactive_timeout for replication slots via SQL API
> > > > > 0004 Introduce new SQL funtion pg_alter_replication_slot
> > > > > 0005 Allow setting inactive_timeout in the replication command
> > > > > 0006 Add inactive_timeout based replication slot invalidation
> > > > >
> > > > > 1. Keep it last_inactive_at as a shared memory variable, but always
> > > > > set it at restart if the slot's inactive_timeout has non-zero value
> > > > > and reset it as soon as someone acquires that slot so that if the slot
> > > > > doesn't get acquired  till inactive_timeout, checkpointer will
> > > > > invalidate the slot.
> > > > > 4. last_inactive_at should also be set to the current time during slot
> > > > > creation because if one creates a slot and does nothing with it then
> > > > > it's the time it starts to be inactive.
> > > >
> > > > I did not look at the code yet but just tested the behavior. It works as you
> > > > describe it but I think this behavior is weird because:
> > > >
> > > > - when we create a slot without a timeout then last_inactive_at is set. I think
> > > > that's fine, but then:
> > > > - when we restart the engine, then last_inactive_at is gone (as timeout is not
> > > > set).
> > > >
> > > > I think last_inactive_at should be set also at engine restart even if there is
> > > > no timeout.
> > >
> > > I think it is the opposite. Why do we need to set  'last_inactive_at'
> > > when inactive_timeout is not set?
> >
> > I think those are unrelated, one could want to know when a slot has been inactive
> > even if no timeout is set. I understand that for this patch series we have in mind
> > to use them both to invalidate slots but I think that there is use case to not
> > use both in correlation. Also not setting last_inactive_at could give the "false"
> > impression that the slot is active.
> >
> 
> I see your point and agree with this. I feel we can commit this part
> first then,

Agree that in this case the current ordering makes sense (as setting
last_inactive_at would be completly unrelated to the timeout).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  111. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Ajin Cherian <itsajin@gmail.com> — 2024-03-22T12:24:43Z

    On Fri, Mar 22, 2024 at 7:15 PM Bharath Rupireddy <
bharath.rupireddyforpostgres@gmail.com> wrote:

> On Fri, Mar 22, 2024 at 12:39 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > > Please find the v14-0001 patch for now.
> >
> > Thanks!
> >
> > > LGTM. Let's wait for Bertrand to see if he has more comments on 0001
> > > and then I'll push it.
> >
> > LGTM too.
>
> Thanks. Here I'm implementing the following:
>
> 0001 Track invalidation_reason in pg_replication_slots
> 0002 Track last_inactive_at in pg_replication_slots
> 0003 Allow setting inactive_timeout for replication slots via SQL API
> 0004 Introduce new SQL funtion pg_alter_replication_slot
> 0005 Allow setting inactive_timeout in the replication command
> 0006 Add inactive_timeout based replication slot invalidation
>
> 1. Keep it last_inactive_at as a shared memory variable, but always
> set it at restart if the slot's inactive_timeout has non-zero value
> and reset it as soon as someone acquires that slot so that if the slot
> doesn't get acquired  till inactive_timeout, checkpointer will
> invalidate the slot.
> 2. Ensure with pg_alter_replication_slot one could "only" alter the
> timeout property for the time being, if not that could lead to the
> subscription inconsistency.
> 3. Have some notes in the CREATE and ALTER SUBSCRIPTION docs about
> using an existing slot to leverage inactive_timeout feature.
> 4. last_inactive_at should also be set to the current time during slot
> creation because if one creates a slot and does nothing with it then
> it's the time it starts to be inactive.
> 5. We don't set last_inactive_at to GetCurrentTimestamp() for failover
> slots.
> 6. Leave the patch that added support for inactive_timeout in
> subscriptions.
>
> Please see the attached v14 patch set. No change in the attached
> v14-0001 from the previous patch.
>
>
>
Some comments:
1. In patch 0005:
In ReplicationSlotAlter():
+ lock_acquired = false;
  if (MyReplicationSlot->data.failover != failover)
  {
  SpinLockAcquire(&MyReplicationSlot->mutex);
+ lock_acquired = true;
  MyReplicationSlot->data.failover = failover;
+ }
+
+ if (MyReplicationSlot->data.inactive_timeout != inactive_timeout)
+ {
+ if (!lock_acquired)
+ {
+ SpinLockAcquire(&MyReplicationSlot->mutex);
+ lock_acquired = true;
+ }
+
+ MyReplicationSlot->data.inactive_timeout = inactive_timeout;
+ }
+
+ if (lock_acquired)
+ {
  SpinLockRelease(&MyReplicationSlot->mutex);

Can't you make it shorter like below:
lock_acquired = false;

if (MyReplicationSlot->data.failover != failover ||
MyReplicationSlot->data.inactive_timeout != inactive_timeout) {
    SpinLockAcquire(&MyReplicationSlot->mutex);
    lock_acquired = true;
}

if (MyReplicationSlot->data.failover != failover) {
    MyReplicationSlot->data.failover = failover;
}

if (MyReplicationSlot->data.inactive_timeout != inactive_timeout) {
    MyReplicationSlot->data.inactive_timeout = inactive_timeout;
}

if (lock_acquired) {
    SpinLockRelease(&MyReplicationSlot->mutex);
    ReplicationSlotMarkDirty();
    ReplicationSlotSave();
}

2. In patch 0005:  why change walrcv_alter_slot option? it doesn't seem to
be used anywhere, any use case for it? If required, would the intention be
to add this as a Create Subscription option?

regards,
Ajin Cherian
Fujitsu Australia

  112. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-22T12:32:11Z

    On Fri, Mar 22, 2024 at 5:30 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 03:56:23PM +0530, Amit Kapila wrote:
> > On Fri, Mar 22, 2024 at 3:15 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote:
> > >
> > > 1 ===
> > >
> > > @@ -691,6 +699,13 @@ ReplicationSlotRelease(void)
> > >                 ConditionVariableBroadcast(&slot->active_cv);
> > >         }
> > >
> > > +       if (slot->data.persistency == RS_PERSISTENT)
> > > +       {
> > > +               SpinLockAcquire(&slot->mutex);
> > > +               slot->last_inactive_at = GetCurrentTimestamp();
> > > +               SpinLockRelease(&slot->mutex);
> > > +       }
> > >
> > > I'm not sure we should do system calls while we're holding a spinlock.
> > > Assign a variable before?
> > >
> > > 2 ===
> > >
> > > Also, what about moving this here?
> > >
> > > "
> > >     if (slot->data.persistency == RS_PERSISTENT)
> > >     {
> > >         /*
> > >          * Mark persistent slot inactive.  We're not freeing it, just
> > >          * disconnecting, but wake up others that may be waiting for it.
> > >          */
> > >         SpinLockAcquire(&slot->mutex);
> > >         slot->active_pid = 0;
> > >         SpinLockRelease(&slot->mutex);
> > >         ConditionVariableBroadcast(&slot->active_cv);
> > >     }
> > > "
> > >
> > > That would avoid testing twice "slot->data.persistency == RS_PERSISTENT".
> > >
> >
> > That sounds like a good idea. Also, don't we need to consider physical
> > slots where we don't reserve WAL during slot creation? I don't think
> > there is a need to set inactive_at for such slots.
>
> If the slot is not active, why shouldn't we set inactive_at? I can understand
> that such a slots do not present "any risks" but I think we should still set
> inactive_at (also to not give the false impression that the slot is active).
>

But OTOH, there is a chance that we will invalidate such slots even
though they have never reserved WAL in the first place which doesn't
appear to be a good thing.

> > > 5 ===
> > >
> > > Most of the fields that reflect a time (not duration) in the system views are
> > > xxxx_time, so I'm wondering if instead of "last_inactive_at" we should use
> > > something like "last_inactive_time"?
> > >
> >
> > How about naming it as last_active_time? This will indicate the time
> > at which the slot was last active.
>
> I thought about it too but I think it could be missleading as one could think that
> it should be updated each time WAL record decoding is happening.
>

Fair enough.

-- 
With Regards,
Amit Kapila.

  113. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-22T13:47:34Z

    Hi,

On Fri, Mar 22, 2024 at 06:02:11PM +0530, Amit Kapila wrote:
> On Fri, Mar 22, 2024 at 5:30 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> > On Fri, Mar 22, 2024 at 03:56:23PM +0530, Amit Kapila wrote:
> > > >
> > > > That would avoid testing twice "slot->data.persistency == RS_PERSISTENT".
> > > >
> > >
> > > That sounds like a good idea. Also, don't we need to consider physical
> > > slots where we don't reserve WAL during slot creation? I don't think
> > > there is a need to set inactive_at for such slots.
> >
> > If the slot is not active, why shouldn't we set inactive_at? I can understand
> > that such a slots do not present "any risks" but I think we should still set
> > inactive_at (also to not give the false impression that the slot is active).
> >
> 
> But OTOH, there is a chance that we will invalidate such slots even
> though they have never reserved WAL in the first place which doesn't
> appear to be a good thing.

That's right but I don't think it is not a good thing. I think we should treat
inactive_at as an independent field (like if the timeout one does not exist at
all) and just focus on its meaning (slot being inactive). If one sets a timeout
(> 0) and gets an invalidation then I think it works as designed (even if the
slot does not present any "risk" as it does not hold any rows or WAL). 

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  114. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-22T21:32:26Z

    On Fri, Mar 22, 2024 at 3:15 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Looking at v14-0002:

Thanks for reviewing. I agree that 0002 with last_inactive_at can go
independently and be of use on its own in addition to helping
implement inactive_timeout based invalidation.

> 1 ===
>
> @@ -691,6 +699,13 @@ ReplicationSlotRelease(void)
>                 ConditionVariableBroadcast(&slot->active_cv);
>         }
>
> +       if (slot->data.persistency == RS_PERSISTENT)
> +       {
> +               SpinLockAcquire(&slot->mutex);
> +               slot->last_inactive_at = GetCurrentTimestamp();
> +               SpinLockRelease(&slot->mutex);
> +       }
>
> I'm not sure we should do system calls while we're holding a spinlock.
> Assign a variable before?

Can do that. Then, the last_inactive_at = current_timestamp + mutex
acquire time. But, that shouldn't be a problem than doing system calls
while holding the mutex. So, done that way.

> 2 ===
>
> Also, what about moving this here?
>
> "
>     if (slot->data.persistency == RS_PERSISTENT)
>     {
>         /*
>          * Mark persistent slot inactive.  We're not freeing it, just
>          * disconnecting, but wake up others that may be waiting for it.
>          */
>         SpinLockAcquire(&slot->mutex);
>         slot->active_pid = 0;
>         SpinLockRelease(&slot->mutex);
>         ConditionVariableBroadcast(&slot->active_cv);
>     }
> "
>
> That would avoid testing twice "slot->data.persistency == RS_PERSISTENT".

Ugh. Done that now.

> 3 ===
>
> @@ -2341,6 +2356,7 @@ RestoreSlotFromDisk(const char *name)
>
>                 slot->in_use = true;
>                 slot->active_pid = 0;
> +               slot->last_inactive_at = 0;
>
> I think we should put GetCurrentTimestamp() here. It's done in v14-0006 but I
> think it's better to do it in 0002 (and not taking care of inactive_timeout).

Done.

> 4 ===
>
>     Track last_inactive_at in pg_replication_slots
>
>  doc/src/sgml/system-views.sgml       | 11 +++++++++++
>  src/backend/catalog/system_views.sql |  3 ++-
>  src/backend/replication/slot.c       | 16 ++++++++++++++++
>  src/backend/replication/slotfuncs.c  |  7 ++++++-
>  src/include/catalog/pg_proc.dat      |  6 +++---
>  src/include/replication/slot.h       |  3 +++
>  src/test/regress/expected/rules.out  |  5 +++--
>  7 files changed, 44 insertions(+), 7 deletions(-)
>
> Worth to add some tests too (or we postpone them in future commits because we're
> confident enough they will follow soon)?

Yes. Added some tests in a new TAP test file named
src/test/recovery/t/043_replslot_misc.pl. This new file can be used to
add miscellaneous replication tests in future as well. I couldn't find
a better place in existing test files - tried having the new tests for
physical slots in t/001_stream_rep.pl and I didn't find a right place
for logical slots.

> 5 ===
>
> Most of the fields that reflect a time (not duration) in the system views are
> xxxx_time, so I'm wondering if instead of "last_inactive_at" we should use
> something like "last_inactive_time"?

Yeah, I can see that. So, I changed it to last_inactive_time.

I agree with treating last_inactive_time as a separate property of the
slot having its own use in addition to helping implement
inactive_timeout based invalidation. I think it can go separately.

I tried to address the review comments received for this patch alone
and attached v15-0001. I'll post other patches soon.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  115. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-23T05:06:18Z

    On Fri, Mar 22, 2024 at 7:17 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 06:02:11PM +0530, Amit Kapila wrote:
> > On Fri, Mar 22, 2024 at 5:30 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > > On Fri, Mar 22, 2024 at 03:56:23PM +0530, Amit Kapila wrote:
> > > > >
> > > > > That would avoid testing twice "slot->data.persistency == RS_PERSISTENT".
> > > > >
> > > >
> > > > That sounds like a good idea. Also, don't we need to consider physical
> > > > slots where we don't reserve WAL during slot creation? I don't think
> > > > there is a need to set inactive_at for such slots.
> > >
> > > If the slot is not active, why shouldn't we set inactive_at? I can understand
> > > that such a slots do not present "any risks" but I think we should still set
> > > inactive_at (also to not give the false impression that the slot is active).
> > >
> >
> > But OTOH, there is a chance that we will invalidate such slots even
> > though they have never reserved WAL in the first place which doesn't
> > appear to be a good thing.
>
> That's right but I don't think it is not a good thing. I think we should treat
> inactive_at as an independent field (like if the timeout one does not exist at
> all) and just focus on its meaning (slot being inactive). If one sets a timeout
> (> 0) and gets an invalidation then I think it works as designed (even if the
> slot does not present any "risk" as it does not hold any rows or WAL).
>

Fair point.

-- 
With Regards,
Amit Kapila.

  116. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-23T05:57:20Z

    On Sat, Mar 23, 2024 at 3:02 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 3:15 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> >
> > Worth to add some tests too (or we postpone them in future commits because we're
> > confident enough they will follow soon)?
>
> Yes. Added some tests in a new TAP test file named
> src/test/recovery/t/043_replslot_misc.pl. This new file can be used to
> add miscellaneous replication tests in future as well. I couldn't find
> a better place in existing test files - tried having the new tests for
> physical slots in t/001_stream_rep.pl and I didn't find a right place
> for logical slots.
>

How about adding the test in 019_replslot_limit? It is not a direct
fit but I feel later we can even add 'invalid_timeout' related tests
in this file which will use last_inactive_time feature. It is also
possible that some of the tests added by the 'invalid_timeout' feature
will obviate the need for some of these tests.

Review of v15
==============
1.
@@ -1026,7 +1026,8 @@ CREATE VIEW pg_replication_slots AS
             L.conflicting,
             L.invalidation_reason,
             L.failover,
-            L.synced
+            L.synced,
+            L.last_inactive_time
     FROM pg_get_replication_slots() AS L

As mentioned previously, let's keep these new fields before
conflicting and after two_phase.

2.
+# Get last_inactive_time value after slot's creation. Note that the
slot is still
+# inactive unless it's used by the standby below.
+my $last_inactive_time_1 = $primary->safe_psql('postgres',
+ qq(SELECT last_inactive_time FROM pg_replication_slots WHERE
slot_name = '$sb_slot' AND last_inactive_time IS NOT NULL;)
+);

We should check $last_inactive_time_1 to be a valid value and add a
similar check for logical slots.

3. BTW, why don't we set last_inactive_time for temporary slots
(RS_TEMPORARY) as well? Don't we even invalidate temporary slots? If
so, then I think we should set last_inactive_time for those as well
and later allow them to be invalidated based on timeout parameter.

-- 
With Regards,
Amit Kapila.

  117. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-23T07:41:50Z

    On Sat, Mar 23, 2024 at 11:27 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> How about adding the test in 019_replslot_limit? It is not a direct
> fit but I feel later we can even add 'invalid_timeout' related tests
> in this file which will use last_inactive_time feature.

I'm thinking the other way. Now, the new TAP file 043_replslot_misc.pl
can have last_inactive_time tests, and later invalid_timeout ones too.
This way 019_replslot_limit.pl is not cluttered.

> It is also
> possible that some of the tests added by the 'invalid_timeout' feature
> will obviate the need for some of these tests.

Might be. But, I prefer to keep both these tests separate but in the
same file 043_replslot_misc.pl. Because we cover some corner cases the
last_inactive_time is set upon loading the slot from disk.

> Review of v15
> ==============
> 1.
> @@ -1026,7 +1026,8 @@ CREATE VIEW pg_replication_slots AS
>              L.conflicting,
>              L.invalidation_reason,
>              L.failover,
> -            L.synced
> +            L.synced,
> +            L.last_inactive_time
>      FROM pg_get_replication_slots() AS L
>
> As mentioned previously, let's keep these new fields before
> conflicting and after two_phase.

Sorry, I forgot to notice that comment (out of a flood of comments
really :)). Now, done that way.

> 2.
> +# Get last_inactive_time value after slot's creation. Note that the
> slot is still
> +# inactive unless it's used by the standby below.
> +my $last_inactive_time_1 = $primary->safe_psql('postgres',
> + qq(SELECT last_inactive_time FROM pg_replication_slots WHERE
> slot_name = '$sb_slot' AND last_inactive_time IS NOT NULL;)
> +);
>
> We should check $last_inactive_time_1 to be a valid value and add a
> similar check for logical slots.

That's taken care by the type cast we do, right? Isn't that enough?

is( $primary->safe_psql(
        'postgres',
        qq[SELECT last_inactive_time >
'$last_inactive_time'::timestamptz FROM pg_replication_slots WHERE
slot_name = '$sb_slot' AND last_inactive_time IS NOT NULL;]
    ),
    't',
    'last inactive time for an inactive physical slot is updated correctly');

For instance, setting last_inactive_time_1 to an invalid value fails
with the following error:

error running SQL: 'psql:<stdin>:1: ERROR:  invalid input syntax for
type timestamp with time zone: "foo"
LINE 1: SELECT last_inactive_time > 'foo'::timestamptz FROM pg_repli...

> 3. BTW, why don't we set last_inactive_time for temporary slots
> (RS_TEMPORARY) as well? Don't we even invalidate temporary slots? If
> so, then I think we should set last_inactive_time for those as well
> and later allow them to be invalidated based on timeout parameter.

WFM. Done that way.

Please see the attached v16 patch.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  118. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-23T09:04:45Z

    Hi,

On Sat, Mar 23, 2024 at 01:11:50PM +0530, Bharath Rupireddy wrote:
> On Sat, Mar 23, 2024 at 11:27 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > How about adding the test in 019_replslot_limit? It is not a direct
> > fit but I feel later we can even add 'invalid_timeout' related tests
> > in this file which will use last_inactive_time feature.
> 
> I'm thinking the other way. Now, the new TAP file 043_replslot_misc.pl
> can have last_inactive_time tests, and later invalid_timeout ones too.
> This way 019_replslot_limit.pl is not cluttered.

I share the same opinion as Amit: I think 019_replslot_limit would be a better
place, because I see the timeout as another kind of limit.

> 
> > It is also
> > possible that some of the tests added by the 'invalid_timeout' feature
> > will obviate the need for some of these tests.
> 
> Might be. But, I prefer to keep both these tests separate but in the
> same file 043_replslot_misc.pl. Because we cover some corner cases the
> last_inactive_time is set upon loading the slot from disk.

Right but I think that this test does not necessary have to be in the same .pl
as the one testing the timeout. Could be added in one of the existing .pl like
001_stream_rep.pl for example.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  119. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-24T02:30:00Z

    On Sat, Mar 23, 2024 at 2:34 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > > How about adding the test in 019_replslot_limit? It is not a direct
> > > fit but I feel later we can even add 'invalid_timeout' related tests
> > > in this file which will use last_inactive_time feature.
> >
> > I'm thinking the other way. Now, the new TAP file 043_replslot_misc.pl
> > can have last_inactive_time tests, and later invalid_timeout ones too.
> > This way 019_replslot_limit.pl is not cluttered.
>
> I share the same opinion as Amit: I think 019_replslot_limit would be a better
> place, because I see the timeout as another kind of limit.

Hm. Done that way.

Please see the attached v17 patch.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  120. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-24T05:10:19Z

    On Sat, Mar 23, 2024 at 1:12 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Sat, Mar 23, 2024 at 11:27 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> > 2.
> > +# Get last_inactive_time value after slot's creation. Note that the
> > slot is still
> > +# inactive unless it's used by the standby below.
> > +my $last_inactive_time_1 = $primary->safe_psql('postgres',
> > + qq(SELECT last_inactive_time FROM pg_replication_slots WHERE
> > slot_name = '$sb_slot' AND last_inactive_time IS NOT NULL;)
> > +);
> >
> > We should check $last_inactive_time_1 to be a valid value and add a
> > similar check for logical slots.
>
> That's taken care by the type cast we do, right? Isn't that enough?
>
> is( $primary->safe_psql(
>         'postgres',
>         qq[SELECT last_inactive_time >
> '$last_inactive_time'::timestamptz FROM pg_replication_slots WHERE
> slot_name = '$sb_slot' AND last_inactive_time IS NOT NULL;]
>     ),
>     't',
>     'last inactive time for an inactive physical slot is updated correctly');
>
> For instance, setting last_inactive_time_1 to an invalid value fails
> with the following error:
>
> error running SQL: 'psql:<stdin>:1: ERROR:  invalid input syntax for
> type timestamp with time zone: "foo"
> LINE 1: SELECT last_inactive_time > 'foo'::timestamptz FROM pg_repli...
>

It would be found at a later point. It would be probably better to
verify immediately after the test that fetches the last_inactive_time
value.

-- 
With Regards,
Amit Kapila.

  121. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-24T09:35:44Z

    On Sun, Mar 24, 2024 at 10:40 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > For instance, setting last_inactive_time_1 to an invalid value fails
> > with the following error:
> >
> > error running SQL: 'psql:<stdin>:1: ERROR:  invalid input syntax for
> > type timestamp with time zone: "foo"
> > LINE 1: SELECT last_inactive_time > 'foo'::timestamptz FROM pg_repli...
> >
>
> It would be found at a later point. It would be probably better to
> verify immediately after the test that fetches the last_inactive_time
> value.

Agree. I've added a few more checks explicitly to verify the
last_inactive_time is sane with the following:

        qq[SELECT '$last_inactive_time'::timestamptz > to_timestamp(0)
AND '$last_inactive_time'::timestamptz >
'$slot_creation_time'::timestamptz;]

I've attached the v18 patch set here. I've also addressed earlier
review comments from Amit, Ajin Cherian. Note that I've added new
invalidation mechanism tests in a separate TAP test file just because
I don't want to clutter or bloat any of the existing files and spread
tests for physical slots and logical slots into separate existing TAP
files.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  122. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-25T04:18:23Z

    On Sun, Mar 24, 2024 at 3:05 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Sun, Mar 24, 2024 at 10:40 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > For instance, setting last_inactive_time_1 to an invalid value fails
> > > with the following error:
> > >
> > > error running SQL: 'psql:<stdin>:1: ERROR:  invalid input syntax for
> > > type timestamp with time zone: "foo"
> > > LINE 1: SELECT last_inactive_time > 'foo'::timestamptz FROM pg_repli...
> > >
> >
> > It would be found at a later point. It would be probably better to
> > verify immediately after the test that fetches the last_inactive_time
> > value.
>
> Agree. I've added a few more checks explicitly to verify the
> last_inactive_time is sane with the following:
>
>         qq[SELECT '$last_inactive_time'::timestamptz > to_timestamp(0)
> AND '$last_inactive_time'::timestamptz >
> '$slot_creation_time'::timestamptz;]
>

Such a test looks reasonable but shall we add equal to in the second
part of the test (like '$last_inactive_time'::timestamptz >=
> '$slot_creation_time'::timestamptz;). This is just to be sure that even if the test ran fast enough to give the same time, the test shouldn't fail. I think it won't matter for correctness as well.


-- 
With Regards,
Amit Kapila.

  123. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-25T04:58:31Z

    On Mon, Mar 25, 2024 at 9:48 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
>
> Such a test looks reasonable but shall we add equal to in the second
> part of the test (like '$last_inactive_time'::timestamptz >=
> > '$slot_creation_time'::timestamptz;). This is just to be sure that even if the test ran fast enough to give the same time, the test shouldn't fail. I think it won't matter for correctness as well.
>

Apart from this, I have made minor changes in the comments. See and
let me know what you think of attached.

-- 
With Regards,
Amit Kapila.

  124. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-25T05:03:30Z

    On Sun, Mar 24, 2024 at 3:06 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> I've attached the v18 patch set here.

Thanks for the patches. Please find few comments:

patch 001:
--------

1)
slot.h:

+ /* The time at which this slot become inactive */
+ TimestampTz last_inactive_time;

become -->became

---------
patch 002:

2)
slotsync.c:

  ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY,
    remote_slot->two_phase,
    remote_slot->failover,
-   true);
+   true, 0);

+ slot->data.inactive_timeout = remote_slot->inactive_timeout;

Is there a reason we are not passing 'remote_slot->inactive_timeout'
to ReplicationSlotCreate() directly?

---------

3)
slotfuncs.c
pg_create_logical_replication_slot():
+ int inactive_timeout = PG_GETARG_INT32(5);

Can we mention here that timeout is in seconds either in comment or
rename variable to inactive_timeout_secs?

Please do this for create_physical_replication_slot(),
create_logical_replication_slot(),
pg_create_physical_replication_slot() as well.

---------
4)
+ int inactive_timeout; /* The amount of time in seconds the slot
+ * is allowed to be inactive. */
 } LogicalSlotInfo;

 Do we need to mention "before getting invalided" like other places
(in last patch)?

----------

 5)
Same at these two places. "before getting invalided" to be added in
the last patch otherwise the info is incompleted.

+
+ /* The amount of time in seconds the slot is allowed to be inactive */
+ int inactive_timeout;
 } ReplicationSlotPersistentData;


+ * inactive_timeout: The amount of time in seconds the slot is allowed to be
+ *     inactive.
  */
 void
 ReplicationSlotCreate(const char *name, bool db_specific,
 Same here. "before getting invalidated" ?

--------

Reviewing more..

thanks
Shveta

  125. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-25T06:23:53Z

    On Mon, Mar 25, 2024 at 10:33 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Sun, Mar 24, 2024 at 3:06 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > I've attached the v18 patch set here.
>

I have a question. Don't we allow creating subscriptions on an
existing slot with a non-null 'inactive_timeout' set where
'inactive_timeout' of the slot is retained even after subscription
creation?

I tried this:

===================
--On publisher, create slot with 120sec inactive_timeout:
SELECT * FROM pg_create_logical_replication_slot('logical_slot1',
'pgoutput', false, true, true, 120);

--On subscriber, create sub using logical_slot1
create subscription mysubnew1_1  connection 'dbname=newdb1
host=localhost user=shveta port=5433' publication mypubnew1_1 WITH
(failover = true, create_slot=false, slot_name='logical_slot1');

--Before creating sub, pg_replication_slots output:
   slot_name   | failover | synced | active | temp | conf |
   lat                | inactive_timeout
---------------+----------+--------+--------+------+------+----------------------------------+------------------
 logical_slot1 | t        | f      | f      | f    | f    | 2024-03-25
11:11:55.375736+05:30 |              120

--After creating sub pg_replication_slots output:  (inactive_timeout is 0 now):
   slot_name   |failover | synced | active | temp | conf | | lat |
inactive_timeout
---------------+---------+--------+--------+------+------+-+-----+------------------
 logical_slot1 |t        | f      | t      | f    | f    | |     |
           0
===================

In CreateSubscription, we call  'walrcv_alter_slot()' /
'ReplicationSlotAlter()' when create_slot is false. This call ends up
setting active_timeout from 120sec to 0. Is it intentional?

thanks
Shveta

  126. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-25T06:55:21Z

    On Mon, Mar 25, 2024 at 10:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 25, 2024 at 9:48 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Such a test looks reasonable but shall we add equal to in the second
> > part of the test (like '$last_inactive_time'::timestamptz >=
> > > '$slot_creation_time'::timestamptz;). This is just to be sure that even if the test ran fast enough to give the same time, the test shouldn't fail. I think it won't matter for correctness as well.

Agree. I added that in v19 patch. I was having that concern in my
mind. That's the reason I wasn't capturing current_time something like
below for the same worry that current_timestamp might be the same (or
nearly the same) as the slot creation time. That's why I ended up
capturing current_timestamp in a separate query than clubbing it up
with pg_create_physical_replication_slot.

SELECT current_timestamp FROM pg_create_physical_replication_slot('foo');

> Apart from this, I have made minor changes in the comments. See and
> let me know what you think of attached.

LGTM. I've merged the diff into v19 patch.

Please find the attached v19 patch.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  127. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-25T07:13:19Z

    On Mon, Mar 25, 2024 at 11:53 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Mar 25, 2024 at 10:33 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Sun, Mar 24, 2024 at 3:06 PM Bharath Rupireddy
> > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > >
> > > I've attached the v18 patch set here.

I have one concern, for synced slots on standby, how do we disallow
invalidation due to inactive-timeout immediately after promotion?

For synced slots, last_inactive_time and inactive_timeout are both
set. Let's say I bring down primary for promotion of standby and then
promote standby, there are chances that it may end up invalidating
synced slots (considering standby is not brought down during promotion
and thus inactive_timeout may already be past 'last_inactive_time'). I
tried with smaller unit of inactive_timeout:

--Shutdown primary to prepare for planned promotion.

--On standby, one synced slot with last_inactive_time (lat) as 12:21
   slot_name   | failover | synced | active | temp | conf | res |
         lat                                        | inactive_timeout
---------------+----------+--------+--------+------+------+-----+----------------------------------+------------------
 logical_slot1 | t           | t              | f         | f       |
f       |       | 2024-03-25 12:21:09.020757+05:30 |              60

--wait for some time, now the time is 12:24
postgres=# select now();
               now
----------------------------------
 2024-03-25 12:24:17.616716+05:30

-- promote immediately:
./pg_ctl -D ../../standbydb/ promote -w

--on promoted standby:
postgres=# select pg_is_in_recovery();
 pg_is_in_recovery
-------------------
 f

--synced slot is invalidated immediately on promotion.
   slot_name   | failover | synced | active | temp | conf
  |       res                |               lat                |
inactive_timeout
---------------+----------+--------+--------+------+------+------------------+----------------------------------+--------
 logical_slot1 | t             | t           | f         | f
| f                    | inactive_timeout | 2024-03-25
12:21:09.020757+05:30 |


thanks
Shveta

  128. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-25T07:29:52Z

    On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Mar 25, 2024 at 11:53 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Mon, Mar 25, 2024 at 10:33 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Sun, Mar 24, 2024 at 3:06 PM Bharath Rupireddy
> > > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > > >
> > > > I've attached the v18 patch set here.
>
> I have one concern, for synced slots on standby, how do we disallow
> invalidation due to inactive-timeout immediately after promotion?
>
> For synced slots, last_inactive_time and inactive_timeout are both
> set. Let's say I bring down primary for promotion of standby and then
> promote standby, there are chances that it may end up invalidating
> synced slots (considering standby is not brought down during promotion
> and thus inactive_timeout may already be past 'last_inactive_time').
>

This raises the question of whether we need to set
'last_inactive_time' synced slots on the standby?

-- 
With Regards,
Amit Kapila.

  129. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-25T07:35:45Z

    Hi,

On Mon, Mar 25, 2024 at 12:25:21PM +0530, Bharath Rupireddy wrote:
> On Mon, Mar 25, 2024 at 10:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Mar 25, 2024 at 9:48 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Such a test looks reasonable but shall we add equal to in the second
> > > part of the test (like '$last_inactive_time'::timestamptz >=
> > > > '$slot_creation_time'::timestamptz;). This is just to be sure that even if the test ran fast enough to give the same time, the test shouldn't fail. I think it won't matter for correctness as well.
> 
> Agree. I added that in v19 patch. I was having that concern in my
> mind. That's the reason I wasn't capturing current_time something like
> below for the same worry that current_timestamp might be the same (or
> nearly the same) as the slot creation time. That's why I ended up
> capturing current_timestamp in a separate query than clubbing it up
> with pg_create_physical_replication_slot.
> 
> SELECT current_timestamp FROM pg_create_physical_replication_slot('foo');
> 
> > Apart from this, I have made minor changes in the comments. See and
> > let me know what you think of attached.
> 

Thanks!

v19-0001 LGTM, just one Nit comment for 019_replslot_limit.pl:

The code for "Get last_inactive_time value after the slot's creation" and 
"Check that the captured time is sane" is somehow duplicated: is it worth creating
2 functions?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  130. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-25T08:07:35Z

    Hi,

On Mon, Mar 25, 2024 at 12:59:52PM +0530, Amit Kapila wrote:
> On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Mon, Mar 25, 2024 at 11:53 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Mon, Mar 25, 2024 at 10:33 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Sun, Mar 24, 2024 at 3:06 PM Bharath Rupireddy
> > > > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > > > >
> > > > > I've attached the v18 patch set here.
> >
> > I have one concern, for synced slots on standby, how do we disallow
> > invalidation due to inactive-timeout immediately after promotion?
> >
> > For synced slots, last_inactive_time and inactive_timeout are both
> > set.

Yeah, and I can see last_inactive_time is moving on the standby (while not the
case on the primary), probably due to the sync worker slot acquisition/release
which does not seem right.

> Let's say I bring down primary for promotion of standby and then
> > promote standby, there are chances that it may end up invalidating
> > synced slots (considering standby is not brought down during promotion
> > and thus inactive_timeout may already be past 'last_inactive_time').
> >
> 
> This raises the question of whether we need to set
> 'last_inactive_time' synced slots on the standby?

Yeah, I think that last_inactive_time should stay at 0 on synced slots on the
standby because such slots are not usable anyway (until the standby gets promoted).

So, I think that last_inactive_time does not make sense if the slot never had
the chance to be active.

OTOH I think the timeout invalidation (if any) should be synced from primary.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  131. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-25T08:37:21Z

    On Mon, Mar 25, 2024 at 1:37 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Mon, Mar 25, 2024 at 12:59:52PM +0530, Amit Kapila wrote:
> > On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Mon, Mar 25, 2024 at 11:53 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Mon, Mar 25, 2024 at 10:33 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > > >
> > > > > On Sun, Mar 24, 2024 at 3:06 PM Bharath Rupireddy
> > > > > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > > > > >
> > > > > > I've attached the v18 patch set here.
> > >
> > > I have one concern, for synced slots on standby, how do we disallow
> > > invalidation due to inactive-timeout immediately after promotion?
> > >
> > > For synced slots, last_inactive_time and inactive_timeout are both
> > > set.
>
> Yeah, and I can see last_inactive_time is moving on the standby (while not the
> case on the primary), probably due to the sync worker slot acquisition/release
> which does not seem right.
>
> > Let's say I bring down primary for promotion of standby and then
> > > promote standby, there are chances that it may end up invalidating
> > > synced slots (considering standby is not brought down during promotion
> > > and thus inactive_timeout may already be past 'last_inactive_time').
> > >
> >
> > This raises the question of whether we need to set
> > 'last_inactive_time' synced slots on the standby?
>
> Yeah, I think that last_inactive_time should stay at 0 on synced slots on the
> standby because such slots are not usable anyway (until the standby gets promoted).
>
> So, I think that last_inactive_time does not make sense if the slot never had
> the chance to be active.
>
> OTOH I think the timeout invalidation (if any) should be synced from primary.

Yes, even I feel that last_inactive_time makes sense only when the
slot is available to be used. Synced slots are not available to be
used until standby is promoted and thus last_inactive_time can be
skipped to be set for synced_slots. But once primay is invalidated due
to inactive-timeout, that invalidation should be synced to standby
(which is happening currently).

thanks
Shveta

  132. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-25T08:51:11Z

    Hi,

On Mon, Mar 25, 2024 at 02:07:21PM +0530, shveta malik wrote:
> On Mon, Mar 25, 2024 at 1:37 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Hi,
> >
> > On Mon, Mar 25, 2024 at 12:59:52PM +0530, Amit Kapila wrote:
> > > On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Mon, Mar 25, 2024 at 11:53 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > > >
> > > > > On Mon, Mar 25, 2024 at 10:33 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > > > >
> > > > > > On Sun, Mar 24, 2024 at 3:06 PM Bharath Rupireddy
> > > > > > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > > > > > >
> > > > > > > I've attached the v18 patch set here.
> > > >
> > > > I have one concern, for synced slots on standby, how do we disallow
> > > > invalidation due to inactive-timeout immediately after promotion?
> > > >
> > > > For synced slots, last_inactive_time and inactive_timeout are both
> > > > set.
> >
> > Yeah, and I can see last_inactive_time is moving on the standby (while not the
> > case on the primary), probably due to the sync worker slot acquisition/release
> > which does not seem right.
> >
> > > Let's say I bring down primary for promotion of standby and then
> > > > promote standby, there are chances that it may end up invalidating
> > > > synced slots (considering standby is not brought down during promotion
> > > > and thus inactive_timeout may already be past 'last_inactive_time').
> > > >
> > >
> > > This raises the question of whether we need to set
> > > 'last_inactive_time' synced slots on the standby?
> >
> > Yeah, I think that last_inactive_time should stay at 0 on synced slots on the
> > standby because such slots are not usable anyway (until the standby gets promoted).
> >
> > So, I think that last_inactive_time does not make sense if the slot never had
> > the chance to be active.
> >
> > OTOH I think the timeout invalidation (if any) should be synced from primary.
> 
> Yes, even I feel that last_inactive_time makes sense only when the
> slot is available to be used. Synced slots are not available to be
> used until standby is promoted and thus last_inactive_time can be
> skipped to be set for synced_slots. But once primay is invalidated due
> to inactive-timeout, that invalidation should be synced to standby
> (which is happening currently).
> 

yeah, syncing the invalidation and always keeping last_inactive_time to zero 
for synced slots looks right to me.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  133. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-25T09:09:50Z

    On Mon, Mar 25, 2024 at 1:37 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> Yeah, and I can see last_inactive_time is moving on the standby (while not the
> case on the primary), probably due to the sync worker slot acquisition/release
> which does not seem right.
>

Yes, you are right, last_inactive_time keeps on moving for synced
slots on standby.  Once I disabled slot-sync worker, then it is
constant. Then it only changes if I call pg_sync_replication_slots().

On a  different note, I noticed that we allow altering
inactive_timeout for synced-slots on standby. And again overwrite it
with the primary's value in the next sync cycle. Steps:

====================
--Check pg_replication_slots for synced slot on standby, inactive_timeout is 120
   slot_name   | failover | synced | active | inactive_timeout
---------------+----------+--------+--------+------------------
 logical_slot1 | t        | t      | f      |              120

--Alter on standby
SELECT 'alter' FROM pg_alter_replication_slot('logical_slot1', 900);

--Check pg_replication_slots:
   slot_name   | failover | synced | active | inactive_timeout
---------------+----------+--------+--------+------------------
 logical_slot1 | t        | t      | f      |              900

--Run sync function
SELECT pg_sync_replication_slots();

--check again, inactive_timeout is set back to primary's value.
   slot_name   | failover | synced | active | inactive_timeout
---------------+----------+--------+--------+------------------
 logical_slot1 | t        | t      | f      |              120

 ====================

I feel altering synced slot's inactive_timeout should be prohibited on
standby. It should be in sync with primary always. Thoughts?

I am listing the concerns raised by me:
1) create-subscription with create_slot=false overwriting
inactive_timeout of existing slot  ([1])
2) last_inactive_time set for synced slots may result in invalidation
of slot on promotion.  ([2])
3) alter replication slot to alter inactive_timout for synced slots on
standby, should this be allowed?

[1]: https://www.postgresql.org/message-id/CAJpy0uAqBi%2BGbNn2ngJ-A_Z905CD3ss896bqY2ACUjGiF1Gkng%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/CAJpy0uCLu%2BmqAwAMum%3DpXE9YYsy0BE7hOSw_Wno5vjwpFY%3D63g%40mail.gmail.com

thanks
Shveta

  134. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-25T09:23:53Z

    Hi,

On Mon, Mar 25, 2024 at 02:39:50PM +0530, shveta malik wrote:
> I am listing the concerns raised by me:
> 3) alter replication slot to alter inactive_timout for synced slots on
> standby, should this be allowed?

I don't think it should be allowed.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  135. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-25T10:01:15Z

    On Mon, Mar 25, 2024 at 1:37 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > > I have one concern, for synced slots on standby, how do we disallow
> > > invalidation due to inactive-timeout immediately after promotion?
> > >
> > > For synced slots, last_inactive_time and inactive_timeout are both
> > > set.
>
> Yeah, and I can see last_inactive_time is moving on the standby (while not the
> case on the primary), probably due to the sync worker slot acquisition/release
> which does not seem right.
>
> > Let's say I bring down primary for promotion of standby and then
> > > promote standby, there are chances that it may end up invalidating
> > > synced slots (considering standby is not brought down during promotion
> > > and thus inactive_timeout may already be past 'last_inactive_time').
> > >
> >
> > This raises the question of whether we need to set
> > 'last_inactive_time' synced slots on the standby?
>
> Yeah, I think that last_inactive_time should stay at 0 on synced slots on the
> standby because such slots are not usable anyway (until the standby gets promoted).
>
> So, I think that last_inactive_time does not make sense if the slot never had
> the chance to be active.

Right. Done that way i.e. not setting the last_inactive_time for slots
both while releasing the slot and restoring from the disk.

Also, I've added a TAP function to check if the captured times are
sane per Bertrand's review comment.

Please see the attached v20 patch.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  136. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-25T10:38:59Z

    On Mon, Mar 25, 2024 at 3:31 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Right. Done that way i.e. not setting the last_inactive_time for slots
> both while releasing the slot and restoring from the disk.
>
> Also, I've added a TAP function to check if the captured times are
> sane per Bertrand's review comment.
>
> Please see the attached v20 patch.

Thanks for the patch. The issue of unnecessary invalidation of synced
slots on promotion is resolved in this patch.

thanks
Shveta

  137. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-25T11:33:59Z

    On Mon, Mar 25, 2024 at 3:31 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Right. Done that way i.e. not setting the last_inactive_time for slots
> both while releasing the slot and restoring from the disk.
>
> Also, I've added a TAP function to check if the captured times are
> sane per Bertrand's review comment.
>
> Please see the attached v20 patch.
>

Pushed, after minor changes.

-- 
With Regards,
Amit Kapila.

  138. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-25T11:40:11Z

    On Mon, Mar 25, 2024 at 2:40 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Mar 25, 2024 at 1:37 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Hi,
> >
> > Yeah, and I can see last_inactive_time is moving on the standby (while not the
> > case on the primary), probably due to the sync worker slot acquisition/release
> > which does not seem right.
> >
>
> Yes, you are right, last_inactive_time keeps on moving for synced
> slots on standby.  Once I disabled slot-sync worker, then it is
> constant. Then it only changes if I call pg_sync_replication_slots().
>
> On a  different note, I noticed that we allow altering
> inactive_timeout for synced-slots on standby. And again overwrite it
> with the primary's value in the next sync cycle. Steps:
>
> ====================
> --Check pg_replication_slots for synced slot on standby, inactive_timeout is 120
>    slot_name   | failover | synced | active | inactive_timeout
> ---------------+----------+--------+--------+------------------
>  logical_slot1 | t        | t      | f      |              120
>
> --Alter on standby
> SELECT 'alter' FROM pg_alter_replication_slot('logical_slot1', 900);
>

I think we should keep pg_alter_replication_slot() as the last
priority among the remaining patches for this release. Let's try to
first finish the primary functionality of inactive_timeout patch.
Otherwise, I agree that the problem reported by you should be fixed.

-- 
With Regards,
Amit Kapila.

  139. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-25T11:54:25Z

    On Mon, Mar 25, 2024 at 5:10 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> I think we should keep pg_alter_replication_slot() as the last
> priority among the remaining patches for this release. Let's try to
> first finish the primary functionality of inactive_timeout patch.
> Otherwise, I agree that the problem reported by you should be fixed.

Noted. Will focus on v18-002 patch now.

I was debugging the flow and just noticed that RecoveryInProgress()
always returns 'true' during
StartupReplicationSlots()-->RestoreSlotFromDisk() (even on primary) as
'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at
that time. The 'xlogctl->SharedRecoveryState' is changed  to
'RECOVERY_STATE_DONE' on primary and to 'RECOVERY_STATE_ARCHIVE' on
standby at a later stage in StartupXLOG() (after we are done loading
slots).

The impact of this is, the condition in RestoreSlotFromDisk() in v20-001:

if (!(RecoveryInProgress() && slot->data.synced))
     slot->last_inactive_time = GetCurrentTimestamp();

is merely equivalent to:

if (!slot->data.synced)
    slot->last_inactive_time = GetCurrentTimestamp();

Thus on primary, after restart, last_inactive_at is set correctly,
while on promoted standby (new primary), last_inactive_at is always
NULL after restart for the synced slots.

thanks
Shveta

  140. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-25T19:54:43Z

    I apologize that I haven't been able to keep up with this thread for a
while, but I'm happy to see the continued interest in $SUBJECT.

On Sun, Mar 24, 2024 at 03:05:44PM +0530, Bharath Rupireddy wrote:
> This commit particularly lets one specify the inactive_timeout for
> a slot via SQL functions pg_create_physical_replication_slot and
> pg_create_logical_replication_slot.

Off-list, Bharath brought to my attention that the current proposal was to
set the timeout at the slot level.  While I think that is an entirely
reasonable thing to support, the main use-case I have in mind for this
feature is for an administrator that wants to prevent inactive slots from
causing problems (e.g., transaction ID wraparound) on a server or a number
of servers.  For that use-case, I think a GUC would be much more
convenient.  Perhaps there could be a default inactive slot timeout GUC
that would be used in the absence of a slot-level setting.  Thoughts?

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  141. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-26T04:00:32Z

    On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> I have one concern, for synced slots on standby, how do we disallow
> invalidation due to inactive-timeout immediately after promotion?
>
> For synced slots, last_inactive_time and inactive_timeout are both
> set. Let's say I bring down primary for promotion of standby and then
> promote standby, there are chances that it may end up invalidating
> synced slots (considering standby is not brought down during promotion
> and thus inactive_timeout may already be past 'last_inactive_time').
>

On standby, if we decide to maintain valid last_inactive_time for
synced slots, then invalidation is correctly restricted in
InvalidateSlotForInactiveTimeout() for synced slots using the check:

        if (RecoveryInProgress() && slot->data.synced)
                return false;

But immediately after promotion, we can not rely on the above check
and thus possibility of synced slots invalidation is there. To
maintain consistent behavior regarding the setting of
last_inactive_time for synced slots, similar to user slots, one
potential solution to prevent this invalidation issue is to update the
last_inactive_time of all synced slots within the ShutDownSlotSync()
function during FinishWalRecovery(). This approach ensures that
promotion doesn't immediately invalidate slots, and henceforth, we
possess a correct last_inactive_time as a basis for invalidation going
forward. This will be equivalent to updating last_inactive_time during
restart (but without actual restart during promotion).
The plus point of maintaining last_inactive_time for synced slots
could be, this can provide data to the user on when last time the sync
was attempted on that particular slot by background slot sync worker
or SQl function. Thoughts?

thanks
Shveta

  142. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-26T04:43:55Z

    On Tue, Mar 26, 2024 at 1:24 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
>
> On Sun, Mar 24, 2024 at 03:05:44PM +0530, Bharath Rupireddy wrote:
> > This commit particularly lets one specify the inactive_timeout for
> > a slot via SQL functions pg_create_physical_replication_slot and
> > pg_create_logical_replication_slot.
>
> Off-list, Bharath brought to my attention that the current proposal was to
> set the timeout at the slot level.  While I think that is an entirely
> reasonable thing to support, the main use-case I have in mind for this
> feature is for an administrator that wants to prevent inactive slots from
> causing problems (e.g., transaction ID wraparound) on a server or a number
> of servers.  For that use-case, I think a GUC would be much more
> convenient.  Perhaps there could be a default inactive slot timeout GUC
> that would be used in the absence of a slot-level setting.  Thoughts?
>

Yeah, that is a valid point. One of the reasons for keeping it at slot
level was to allow different subscribers/output plugins to have a
different setting for invalid_timeout for their respective slots based
on their usage. Now, having it as a GUC also has some valid use cases
as pointed out by you but I am not sure having both at slot level and
at GUC level is required. I was a bit inclined to have it at slot
level for now and then based on some field usage report we can later
add GUC as well.

-- 
With Regards,
Amit Kapila.

  143. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-26T05:37:51Z

    On Tue, Mar 26, 2024 at 9:30 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > I have one concern, for synced slots on standby, how do we disallow
> > invalidation due to inactive-timeout immediately after promotion?
> >
> > For synced slots, last_inactive_time and inactive_timeout are both
> > set. Let's say I bring down primary for promotion of standby and then
> > promote standby, there are chances that it may end up invalidating
> > synced slots (considering standby is not brought down during promotion
> > and thus inactive_timeout may already be past 'last_inactive_time').
> >
>
> On standby, if we decide to maintain valid last_inactive_time for
> synced slots, then invalidation is correctly restricted in
> InvalidateSlotForInactiveTimeout() for synced slots using the check:
>
>         if (RecoveryInProgress() && slot->data.synced)
>                 return false;
>
> But immediately after promotion, we can not rely on the above check
> and thus possibility of synced slots invalidation is there. To
> maintain consistent behavior regarding the setting of
> last_inactive_time for synced slots, similar to user slots, one
> potential solution to prevent this invalidation issue is to update the
> last_inactive_time of all synced slots within the ShutDownSlotSync()
> function during FinishWalRecovery(). This approach ensures that
> promotion doesn't immediately invalidate slots, and henceforth, we
> possess a correct last_inactive_time as a basis for invalidation going
> forward. This will be equivalent to updating last_inactive_time during
> restart (but without actual restart during promotion).
> The plus point of maintaining last_inactive_time for synced slots
> could be, this can provide data to the user on when last time the sync
> was attempted on that particular slot by background slot sync worker
> or SQl function. Thoughts?

Please find the attached v21 patch implementing the above idea. It
also has changes for renaming last_inactive_time to inactive_since.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  144. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T05:55:11Z

    Hi,

On Tue, Mar 26, 2024 at 09:30:32AM +0530, shveta malik wrote:
> On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > I have one concern, for synced slots on standby, how do we disallow
> > invalidation due to inactive-timeout immediately after promotion?
> >
> > For synced slots, last_inactive_time and inactive_timeout are both
> > set. Let's say I bring down primary for promotion of standby and then
> > promote standby, there are chances that it may end up invalidating
> > synced slots (considering standby is not brought down during promotion
> > and thus inactive_timeout may already be past 'last_inactive_time').
> >
> 
> On standby, if we decide to maintain valid last_inactive_time for
> synced slots, then invalidation is correctly restricted in
> InvalidateSlotForInactiveTimeout() for synced slots using the check:
> 
>         if (RecoveryInProgress() && slot->data.synced)
>                 return false;

Right.

> But immediately after promotion, we can not rely on the above check
> and thus possibility of synced slots invalidation is there. To
> maintain consistent behavior regarding the setting of
> last_inactive_time for synced slots, similar to user slots, one
> potential solution to prevent this invalidation issue is to update the
> last_inactive_time of all synced slots within the ShutDownSlotSync()
> function during FinishWalRecovery(). This approach ensures that
> promotion doesn't immediately invalidate slots, and henceforth, we
> possess a correct last_inactive_time as a basis for invalidation going
> forward. This will be equivalent to updating last_inactive_time during
> restart (but without actual restart during promotion).
> The plus point of maintaining last_inactive_time for synced slots
> could be, this can provide data to the user on when last time the sync
> was attempted on that particular slot by background slot sync worker
> or SQl function. Thoughts?

Yeah, another plus point is that if the primary is down then one could look
at the synced "active_since" on the standby to get an idea of it (depends of the
last sync though).

The issue that I can see with your proposal is: what if one synced the slots
manually (with pg_sync_replication_slots()) but does not use the sync worker?
Then I think ShutDownSlotSync() is not going to help in that case.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  145. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-26T05:56:38Z

    On Sun, Mar 24, 2024 at 3:05 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> I've attached the v18 patch set here. I've also addressed earlier
> review comments from Amit, Ajin Cherian. Note that I've added new
> invalidation mechanism tests in a separate TAP test file just because
> I don't want to clutter or bloat any of the existing files and spread
> tests for physical slots and logical slots into separate existing TAP
> files.
>

Review comments on v18_0002 and v18_0005
=======================================
1.
 ReplicationSlotCreate(const char *name, bool db_specific,
    ReplicationSlotPersistency persistency,
-   bool two_phase, bool failover, bool synced)
+   bool two_phase, bool failover, bool synced,
+   int inactive_timeout)
 {
  ReplicationSlot *slot = NULL;
  int i;
@@ -345,6 +348,18 @@ ReplicationSlotCreate(const char *name, bool db_specific,
  errmsg("cannot enable failover for a temporary replication slot"));
  }

+ if (inactive_timeout > 0)
+ {
+ /*
+ * Do not allow users to set inactive_timeout for temporary slots,
+ * because temporary slots will not be saved to the disk.
+ */
+ if (persistency == RS_TEMPORARY)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot set inactive_timeout for a temporary replication slot"));
+ }

We have decided to update inactive_since for temporary slots. So,
unless there is some reason, we should allow inactive_timeout to also
be set for temporary slots.

2.
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1024,6 +1024,7 @@ CREATE VIEW pg_replication_slots AS
             L.safe_wal_size,
             L.two_phase,
             L.last_inactive_time,
+            L.inactive_timeout,

Shall we keep inactive_timeout before
last_inactive_time/inactive_since? I don't have any strong reason to
propose that way apart from that the former is provided by the user.

3.
@@ -287,6 +288,13 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
  slot_contents = *slot;
  SpinLockRelease(&slot->mutex);

+ /*
+ * Here's an opportunity to invalidate inactive replication slots
+ * based on timeout, so let's do it.
+ */
+ if (InvalidateReplicationSlotForInactiveTimeout(slot, false, true, true))
+ invalidated = true;

I don't think we should try to invalidate the slots in
pg_get_replication_slots. This function's purpose is to get the
current information on slots and has no intention to perform any work
for slots. Any error due to invalidation won't be what the user would
be expecting here.

4.
+static bool
+InvalidateSlotForInactiveTimeout(ReplicationSlot *slot,
+ bool need_control_lock,
+ bool need_mutex)
{
...
...
+ if (need_control_lock)
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
+
+ /*
+ * Check if the slot needs to be invalidated due to inactive_timeout. We
+ * do this with the spinlock held to avoid race conditions -- for example
+ * the restart_lsn could move forward, or the slot could be dropped.
+ */
+ if (need_mutex)
+ SpinLockAcquire(&slot->mutex);
...

I find this combination of parameters a bit strange. Because, say if
need_mutex is false and need_control_lock is true then that means this
function will acquire LWlock after acquiring spinlock which is
unacceptable. Now, this may not happen in practice as the callers
won't pass such a combination but still, this functionality should be
improved.

-- 
With Regards,
Amit Kapila.

  146. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T06:06:12Z

    Hi,

On Tue, Mar 26, 2024 at 05:55:11AM +0000, Bertrand Drouvot wrote:
> Hi,
> 
> On Tue, Mar 26, 2024 at 09:30:32AM +0530, shveta malik wrote:
> > On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > I have one concern, for synced slots on standby, how do we disallow
> > > invalidation due to inactive-timeout immediately after promotion?
> > >
> > > For synced slots, last_inactive_time and inactive_timeout are both
> > > set. Let's say I bring down primary for promotion of standby and then
> > > promote standby, there are chances that it may end up invalidating
> > > synced slots (considering standby is not brought down during promotion
> > > and thus inactive_timeout may already be past 'last_inactive_time').
> > >
> > 
> > On standby, if we decide to maintain valid last_inactive_time for
> > synced slots, then invalidation is correctly restricted in
> > InvalidateSlotForInactiveTimeout() for synced slots using the check:
> > 
> >         if (RecoveryInProgress() && slot->data.synced)
> >                 return false;
> 
> Right.
> 
> > But immediately after promotion, we can not rely on the above check
> > and thus possibility of synced slots invalidation is there. To
> > maintain consistent behavior regarding the setting of
> > last_inactive_time for synced slots, similar to user slots, one
> > potential solution to prevent this invalidation issue is to update the
> > last_inactive_time of all synced slots within the ShutDownSlotSync()
> > function during FinishWalRecovery(). This approach ensures that
> > promotion doesn't immediately invalidate slots, and henceforth, we
> > possess a correct last_inactive_time as a basis for invalidation going
> > forward. This will be equivalent to updating last_inactive_time during
> > restart (but without actual restart during promotion).
> > The plus point of maintaining last_inactive_time for synced slots
> > could be, this can provide data to the user on when last time the sync
> > was attempted on that particular slot by background slot sync worker
> > or SQl function. Thoughts?
> 
> Yeah, another plus point is that if the primary is down then one could look
> at the synced "active_since" on the standby to get an idea of it (depends of the
> last sync though).
> 
> The issue that I can see with your proposal is: what if one synced the slots
> manually (with pg_sync_replication_slots()) but does not use the sync worker?
> Then I think ShutDownSlotSync() is not going to help in that case.

It looks like ShutDownSlotSync() is always called (even if sync_replication_slots = off),
so that sounds ok to me (I should have checked the code, I was under the impression
ShutDownSlotSync() was not called if sync_replication_slots = off).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  147. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-26T06:20:45Z

    On Tue, Mar 26, 2024 at 11:36 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
> >
> > The issue that I can see with your proposal is: what if one synced the slots
> > manually (with pg_sync_replication_slots()) but does not use the sync worker?
> > Then I think ShutDownSlotSync() is not going to help in that case.
>
> It looks like ShutDownSlotSync() is always called (even if sync_replication_slots = off),
> so that sounds ok to me (I should have checked the code, I was under the impression
> ShutDownSlotSync() was not called if sync_replication_slots = off).

Right, it is called irrespective of sync_replication_slots.

thanks
Shveta

  148. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-26T06:34:26Z

    On Tue, Mar 26, 2024 at 11:08 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 9:30 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Mon, Mar 25, 2024 at 12:43 PM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > I have one concern, for synced slots on standby, how do we disallow
> > > invalidation due to inactive-timeout immediately after promotion?
> > >
> > > For synced slots, last_inactive_time and inactive_timeout are both
> > > set. Let's say I bring down primary for promotion of standby and then
> > > promote standby, there are chances that it may end up invalidating
> > > synced slots (considering standby is not brought down during promotion
> > > and thus inactive_timeout may already be past 'last_inactive_time').
> > >
> >
> > On standby, if we decide to maintain valid last_inactive_time for
> > synced slots, then invalidation is correctly restricted in
> > InvalidateSlotForInactiveTimeout() for synced slots using the check:
> >
> >         if (RecoveryInProgress() && slot->data.synced)
> >                 return false;
> >
> > But immediately after promotion, we can not rely on the above check
> > and thus possibility of synced slots invalidation is there. To
> > maintain consistent behavior regarding the setting of
> > last_inactive_time for synced slots, similar to user slots, one
> > potential solution to prevent this invalidation issue is to update the
> > last_inactive_time of all synced slots within the ShutDownSlotSync()
> > function during FinishWalRecovery(). This approach ensures that
> > promotion doesn't immediately invalidate slots, and henceforth, we
> > possess a correct last_inactive_time as a basis for invalidation going
> > forward. This will be equivalent to updating last_inactive_time during
> > restart (but without actual restart during promotion).
> > The plus point of maintaining last_inactive_time for synced slots
> > could be, this can provide data to the user on when last time the sync
> > was attempted on that particular slot by background slot sync worker
> > or SQl function. Thoughts?
>
> Please find the attached v21 patch implementing the above idea. It
> also has changes for renaming last_inactive_time to inactive_since.
>

Thanks for the patch. I have tested this patch alone, and it does what
it says. One additional thing which I noticed is that now it sets
inactive_since for temp slots as well, but that idea looks fine to me.

I could not test 'invalidation on promotion bug' with this change, as
that needed rebasing of the rest of the patches.

Few trivial things:

1)
Commti msg:

ensures the value is set to current timestamp during the
shutdown to help correctly interpret the time if the standby gets
promoted without a restart.

shutdown --> shutdown of slot sync worker   (as it was not clear if it
is instance shutdown or something else)

2)
'The time since the slot has became inactive'.

has became-->has become
or just became

Please check it in all the files. There are multiple places.

thanks
Shveta

  149. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T07:45:19Z

    Hi,

On Tue, Mar 26, 2024 at 11:07:51AM +0530, Bharath Rupireddy wrote:
> On Tue, Mar 26, 2024 at 9:30 AM shveta malik <shveta.malik@gmail.com> wrote:
> > But immediately after promotion, we can not rely on the above check
> > and thus possibility of synced slots invalidation is there. To
> > maintain consistent behavior regarding the setting of
> > last_inactive_time for synced slots, similar to user slots, one
> > potential solution to prevent this invalidation issue is to update the
> > last_inactive_time of all synced slots within the ShutDownSlotSync()
> > function during FinishWalRecovery(). This approach ensures that
> > promotion doesn't immediately invalidate slots, and henceforth, we
> > possess a correct last_inactive_time as a basis for invalidation going
> > forward. This will be equivalent to updating last_inactive_time during
> > restart (but without actual restart during promotion).
> > The plus point of maintaining last_inactive_time for synced slots
> > could be, this can provide data to the user on when last time the sync
> > was attempted on that particular slot by background slot sync worker
> > or SQl function. Thoughts?
> 
> Please find the attached v21 patch implementing the above idea. It
> also has changes for renaming last_inactive_time to inactive_since.

Thanks!

A few comments:

1 ===

One trailing whitespace:

Applying: Fix review comments for slot's last_inactive_time property
.git/rebase-apply/patch:433: trailing whitespace.
# got a valid inactive_since value representing the last slot sync time.
warning: 1 line adds whitespace errors.

2 ===

It looks like inactive_since is set to the current timestamp on the standby
each time the sync worker does a cycle:

primary:

postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
  slot_name  |        inactive_since
-------------+-------------------------------
 lsub27_slot | 2024-03-26 07:39:19.745517+00
 lsub28_slot | 2024-03-26 07:40:24.953826+00

standby:

postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
  slot_name  |        inactive_since
-------------+-------------------------------
 lsub27_slot | 2024-03-26 07:43:56.387324+00
 lsub28_slot | 2024-03-26 07:43:56.387338+00

I don't think that should be the case.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  150. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-26T08:07:21Z

    On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> 2 ===
>
> It looks like inactive_since is set to the current timestamp on the standby
> each time the sync worker does a cycle:
>
> primary:
>
> postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
>   slot_name  |        inactive_since
> -------------+-------------------------------
>  lsub27_slot | 2024-03-26 07:39:19.745517+00
>  lsub28_slot | 2024-03-26 07:40:24.953826+00
>
> standby:
>
> postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
>   slot_name  |        inactive_since
> -------------+-------------------------------
>  lsub27_slot | 2024-03-26 07:43:56.387324+00
>  lsub28_slot | 2024-03-26 07:43:56.387338+00
>
> I don't think that should be the case.
>

But why? This is exactly what we discussed in another thread where we
agreed to update inactive_since even for sync slots. In each sync
cycle, we acquire/release the slot, so the inactive_since gets
updated. See synchronize_one_slot().

-- 
With Regards,
Amit Kapila.

  151. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T08:24:00Z

    Hi,

On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote:
> On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > 2 ===
> >
> > It looks like inactive_since is set to the current timestamp on the standby
> > each time the sync worker does a cycle:
> >
> > primary:
> >
> > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
> >   slot_name  |        inactive_since
> > -------------+-------------------------------
> >  lsub27_slot | 2024-03-26 07:39:19.745517+00
> >  lsub28_slot | 2024-03-26 07:40:24.953826+00
> >
> > standby:
> >
> > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
> >   slot_name  |        inactive_since
> > -------------+-------------------------------
> >  lsub27_slot | 2024-03-26 07:43:56.387324+00
> >  lsub28_slot | 2024-03-26 07:43:56.387338+00
> >
> > I don't think that should be the case.
> >
> 
> But why? This is exactly what we discussed in another thread where we
> agreed to update inactive_since even for sync slots.

Hum, I thought we agreed to "sync" it and to "update it to current time"
only at promotion time.

I don't think updating inactive_since to current time during each cycle makes
sense (I mean I understand the use case: being able to say when slots have been
sync, but if this is what we want then we should consider an extra view or an
extra field but not relying on the inactive_since one).

If the primary goes down, not updating inactive_since to the current time could
also provide benefit such as knowing the inactive_since of the primary slots
(from the standby) the last time it has been synced. If we update it to the current
time then this information is lost.

> In each sync
> cycle, we acquire/release the slot, so the inactive_since gets
> updated. See synchronize_one_slot().

Right, and I think we should put an extra condition if in recovery.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  152. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-26T08:57:17Z

    On Tue, Mar 26, 2024 at 11:26 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Review comments on v18_0002 and v18_0005
> =======================================
>
> 1.
> We have decided to update inactive_since for temporary slots. So,
> unless there is some reason, we should allow inactive_timeout to also
> be set for temporary slots.

WFM. A temporary slot that's inactive for a long time before even the
server isn't shutdown can utilize this inactive_timeout based
invalidation mechanism. And, I'd also vote for we being consistent for
temporary and synced slots.

>              L.last_inactive_time,
> +            L.inactive_timeout,
>
> Shall we keep inactive_timeout before
> last_inactive_time/inactive_since? I don't have any strong reason to
> propose that way apart from that the former is provided by the user.

Done.

> + if (InvalidateReplicationSlotForInactiveTimeout(slot, false, true, true))
> + invalidated = true;
>
> I don't think we should try to invalidate the slots in
> pg_get_replication_slots. This function's purpose is to get the
> current information on slots and has no intention to perform any work
> for slots. Any error due to invalidation won't be what the user would
> be expecting here.

Agree. Removed.

> 4.
> +static bool
> +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot,
> + bool need_control_lock,
> + bool need_mutex)
> {
> ...
> ...
> + if (need_control_lock)
> + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
> +
> + Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
> +
> + /*
> + * Check if the slot needs to be invalidated due to inactive_timeout. We
> + * do this with the spinlock held to avoid race conditions -- for example
> + * the restart_lsn could move forward, or the slot could be dropped.
> + */
> + if (need_mutex)
> + SpinLockAcquire(&slot->mutex);
> ...
>
> I find this combination of parameters a bit strange. Because, say if
> need_mutex is false and need_control_lock is true then that means this
> function will acquire LWlock after acquiring spinlock which is
> unacceptable. Now, this may not happen in practice as the callers
> won't pass such a combination but still, this functionality should be
> improved.

Right. Either we need two locks or not. So, changed it to use just one
bool need_locks, upon set both control lock and spin lock are acquired
and released.

On Mon, Mar 25, 2024 at 10:33 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> patch 002:
>
> 2)
> slotsync.c:
>
>   ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY,
>     remote_slot->two_phase,
>     remote_slot->failover,
> -   true);
> +   true, 0);
>
> + slot->data.inactive_timeout = remote_slot->inactive_timeout;
>
> Is there a reason we are not passing 'remote_slot->inactive_timeout'
> to ReplicationSlotCreate() directly?

The slot there gets created temporarily for which we were not
supporting inactive_timeout being set. But, in the latest v22 patch we
are supporting, so passing the remote_slot->inactive_timeout directly.

> 3)
> slotfuncs.c
> pg_create_logical_replication_slot():
> + int inactive_timeout = PG_GETARG_INT32(5);
>
> Can we mention here that timeout is in seconds either in comment or
> rename variable to inactive_timeout_secs?
>
> Please do this for create_physical_replication_slot(),
> create_logical_replication_slot(),
> pg_create_physical_replication_slot() as well.

Added /* in seconds */ next the variable declaration.

> ---------
> 4)
> + int inactive_timeout; /* The amount of time in seconds the slot
> + * is allowed to be inactive. */
>  } LogicalSlotInfo;
>
>  Do we need to mention "before getting invalided" like other places
> (in last patch)?

Done.

>  5)
> Same at these two places. "before getting invalided" to be added in
> the last patch otherwise the info is incompleted.
>
> +
> + /* The amount of time in seconds the slot is allowed to be inactive */
> + int inactive_timeout;
>  } ReplicationSlotPersistentData;
>
>
> + * inactive_timeout: The amount of time in seconds the slot is allowed to be
> + *     inactive.
>   */
>  void
>  ReplicationSlotCreate(const char *name, bool db_specific,
>  Same here. "before getting invalidated" ?

Done.

On Tue, Mar 26, 2024 at 12:04 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> > Please find the attached v21 patch implementing the above idea. It
> > also has changes for renaming last_inactive_time to inactive_since.
>
> Thanks for the patch. I have tested this patch alone, and it does what
> it says. One additional thing which I noticed is that now it sets
> inactive_since for temp slots as well, but that idea looks fine to me.

Right. Let's be consistent by treating all slots the same.

> I could not test 'invalidation on promotion bug' with this change, as
> that needed rebasing of the rest of the patches.

Please use the v22 patch set.

> Few trivial things:
>
> 1)
> Commti msg:
>
> ensures the value is set to current timestamp during the
> shutdown to help correctly interpret the time if the standby gets
> promoted without a restart.
>
> shutdown --> shutdown of slot sync worker   (as it was not clear if it
> is instance shutdown or something else)

Changed it to "shutdown of slot sync machinery" to be consistent with
the comments.

> 2)
> 'The time since the slot has became inactive'.
>
> has became-->has become
> or just became
>
> Please check it in all the files. There are multiple places.

Fixed.

Please see the attached v23 patches. I've addressed all the review
comments received so far from Amit and Shveta.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  153. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-26T09:22:11Z

    On Tue, Mar 26, 2024 at 2:27 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> >
> > 1)
> > Commti msg:
> >
> > ensures the value is set to current timestamp during the
> > shutdown to help correctly interpret the time if the standby gets
> > promoted without a restart.
> >
> > shutdown --> shutdown of slot sync worker   (as it was not clear if it
> > is instance shutdown or something else)
>
> Changed it to "shutdown of slot sync machinery" to be consistent with
> the comments.

Thanks for addressing the comments. Just to give more clarity here (so
that you take a informed decision), I am not sure if we actually shut
down slot-sync machinery. We only shot down slot sync worker.
Slot-sync machinery can still be used using
'pg_sync_replication_slots' SQL function. I can easily reproduce the
scenario where SQL function and  reset_synced_slots_info() are going
in parallel where the latter hits 'Assert(s->active_pid == 0)' due to
the fact that  parallel SQL sync function is active on that slot.

thanks
Shveta

  154. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T09:42:40Z

    Hi,

On Tue, Mar 26, 2024 at 02:27:17PM +0530, Bharath Rupireddy wrote:
> Please use the v22 patch set.

Thanks!

1 ===

+reset_synced_slots_info(void)

I'm not sure "reset" is the right word, what about slot_sync_shutdown_update()?

2 ===

+       for (int i = 0; i < max_replication_slots; i++)
+       {
+               ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+               /* Check if it is a synchronized slot */
+               if (s->in_use && s->data.synced)
+               {
+                       TimestampTz now;
+
+                       Assert(SlotIsLogical(s));
+                       Assert(s->active_pid == 0);
+
+                       /*
+                        * Set the time since the slot has become inactive after shutting
+                        * down slot sync machinery. This helps correctly interpret the
+                        * time if the standby gets promoted without a restart. We get the
+                        * current time beforehand to avoid a system call while holding
+                        * the lock.
+                        */
+                       now = GetCurrentTimestamp();

What about moving "now = GetCurrentTimestamp()" outside of the for loop? (it
would be less costly and probably good enough).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  155. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-26T09:47:36Z

    On Tue, Mar 26, 2024 at 1:54 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote:
> > On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > 2 ===
> > >
> > > It looks like inactive_since is set to the current timestamp on the standby
> > > each time the sync worker does a cycle:
> > >
> > > primary:
> > >
> > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
> > >   slot_name  |        inactive_since
> > > -------------+-------------------------------
> > >  lsub27_slot | 2024-03-26 07:39:19.745517+00
> > >  lsub28_slot | 2024-03-26 07:40:24.953826+00
> > >
> > > standby:
> > >
> > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
> > >   slot_name  |        inactive_since
> > > -------------+-------------------------------
> > >  lsub27_slot | 2024-03-26 07:43:56.387324+00
> > >  lsub28_slot | 2024-03-26 07:43:56.387338+00
> > >
> > > I don't think that should be the case.
> > >
> >
> > But why? This is exactly what we discussed in another thread where we
> > agreed to update inactive_since even for sync slots.
>
> Hum, I thought we agreed to "sync" it and to "update it to current time"
> only at promotion time.

I think there may have been some misunderstanding here. But now if I
rethink this, I am fine with 'inactive_since' getting synced from
primary to standby. But if we do that, we need to add docs stating
"inactive_since" represents primary's inactivity and not standby's
slots inactivity for synced slots. The reason for this clarification
is that the synced slot might be generated much later, yet
'inactive_since' is synced from the primary, potentially indicating a
time considerably earlier than when the synced slot was actually
created.

Another approach could be that "inactive_since" for synced slot
actually gives its own inactivity data rather than giving primary's
slot data. We update inactive_since on standby only at 3 occasions:
1) at the time of creation of the synced slot.
2) during standby restart.
3) during promotion of standby.

I have attached a sample patch for this idea as.txt file.

I am fine with any of these approaches.  One gives data synced from
primary for synced slots, while another gives actual inactivity data
of synced slots.

thanks
Shveta

  156. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T10:20:50Z

    Hi,

On Tue, Mar 26, 2024 at 03:17:36PM +0530, shveta malik wrote:
> On Tue, Mar 26, 2024 at 1:54 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Hi,
> >
> > On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote:
> > > On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot
> > > <bertranddrouvot.pg@gmail.com> wrote:
> > > >
> > > > 2 ===
> > > >
> > > > It looks like inactive_since is set to the current timestamp on the standby
> > > > each time the sync worker does a cycle:
> > > >
> > > > primary:
> > > >
> > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
> > > >   slot_name  |        inactive_since
> > > > -------------+-------------------------------
> > > >  lsub27_slot | 2024-03-26 07:39:19.745517+00
> > > >  lsub28_slot | 2024-03-26 07:40:24.953826+00
> > > >
> > > > standby:
> > > >
> > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't';
> > > >   slot_name  |        inactive_since
> > > > -------------+-------------------------------
> > > >  lsub27_slot | 2024-03-26 07:43:56.387324+00
> > > >  lsub28_slot | 2024-03-26 07:43:56.387338+00
> > > >
> > > > I don't think that should be the case.
> > > >
> > >
> > > But why? This is exactly what we discussed in another thread where we
> > > agreed to update inactive_since even for sync slots.
> >
> > Hum, I thought we agreed to "sync" it and to "update it to current time"
> > only at promotion time.
> 
> I think there may have been some misunderstanding here.

Indeed ;-)

> But now if I
> rethink this, I am fine with 'inactive_since' getting synced from
> primary to standby. But if we do that, we need to add docs stating
> "inactive_since" represents primary's inactivity and not standby's
> slots inactivity for synced slots.

Yeah sure.

> The reason for this clarification
> is that the synced slot might be generated much later, yet
> 'inactive_since' is synced from the primary, potentially indicating a
> time considerably earlier than when the synced slot was actually
> created.

Right.

> Another approach could be that "inactive_since" for synced slot
> actually gives its own inactivity data rather than giving primary's
> slot data. We update inactive_since on standby only at 3 occasions:
> 1) at the time of creation of the synced slot.
> 2) during standby restart.
> 3) during promotion of standby.
> 
> I have attached a sample patch for this idea as.txt file.

Thanks!

> I am fine with any of these approaches.  One gives data synced from
> primary for synced slots, while another gives actual inactivity data
> of synced slots.

What about another approach?: inactive_since gives data synced from primary for
synced slots and another dedicated field (could be added later...) could
represent what you suggest as the other option.

Another cons of updating inactive_since at the current time during each slot
sync cycle is that calling GetCurrentTimestamp() very frequently
(during each sync cycle of very active slots) could be too costly.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  157. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-26T10:21:02Z

    On Tue, Mar 26, 2024 at 3:12 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 02:27:17PM +0530, Bharath Rupireddy wrote:
> > Please use the v22 patch set.
>
> Thanks!
>
> 1 ===
>
> +reset_synced_slots_info(void)
>
> I'm not sure "reset" is the right word, what about slot_sync_shutdown_update()?
>

*shutdown_update() sounds generic. How about
update_synced_slots_inactive_time()? I think it is a bit longer but
conveys the meaning.

-- 
With Regards,
Amit Kapila.

  158. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Ajin Cherian <itsajin@gmail.com> — 2024-03-26T10:22:25Z

    On Tue, Mar 26, 2024 at 7:57 PM Bharath Rupireddy <
bharath.rupireddyforpostgres@gmail.com> wrote:

> Please see the attached v23 patches. I've addressed all the review
> comments received so far from Amit and Shveta.
>
>
In patch 0003:
+ SpinLockAcquire(&slot->mutex);
+ }
+
+ Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
+
+ if (slot->inactive_since > 0 &&
+ slot->data.inactive_timeout > 0)
+ {
+ TimestampTz now;
+
+ /* inactive_since is only tracked for inactive slots */
+ Assert(slot->active_pid == 0);
+
+ now = GetCurrentTimestamp();
+ if (TimestampDifferenceExceeds(slot->inactive_since, now,
+   slot->data.inactive_timeout * 1000))
+ inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT;
+ }
+
+ if (need_locks)
+ {
+ SpinLockRelease(&slot->mutex);

Here, GetCurrentTimestamp() is still called with SpinLock held. Maybe do
this prior to acquiring the spinlock.

regards,
Ajin Cherian
Fujitsu Australia

  159. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-26T10:47:53Z

    On Tue, Mar 26, 2024 at 3:50 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> > I think there may have been some misunderstanding here.
>
> Indeed ;-)
>
> > But now if I
> > rethink this, I am fine with 'inactive_since' getting synced from
> > primary to standby. But if we do that, we need to add docs stating
> > "inactive_since" represents primary's inactivity and not standby's
> > slots inactivity for synced slots.
>
> Yeah sure.
>
> > The reason for this clarification
> > is that the synced slot might be generated much later, yet
> > 'inactive_since' is synced from the primary, potentially indicating a
> > time considerably earlier than when the synced slot was actually
> > created.
>
> Right.
>
> > Another approach could be that "inactive_since" for synced slot
> > actually gives its own inactivity data rather than giving primary's
> > slot data. We update inactive_since on standby only at 3 occasions:
> > 1) at the time of creation of the synced slot.
> > 2) during standby restart.
> > 3) during promotion of standby.
> >
> > I have attached a sample patch for this idea as.txt file.
>
> Thanks!
>
> > I am fine with any of these approaches.  One gives data synced from
> > primary for synced slots, while another gives actual inactivity data
> > of synced slots.
>
> What about another approach?: inactive_since gives data synced from primary for
> synced slots and another dedicated field (could be added later...) could
> represent what you suggest as the other option.

Yes, okay with me. I think there is some confusion here as well. In my
second approach above, I have not suggested anything related to
sync-worker. We can think on that later if we really need another
field which give us sync time.  In my second approach, I have tried to
avoid updating inactive_since for synced slots during sync process. We
update that field during creation of synced slot so that
inactive_since reflects correct info even for synced slots (rather
than copying from primary). Please have a look at my patch and let me
know your thoughts. I am fine with copying it from primary as well and
documenting this behaviour.

> Another cons of updating inactive_since at the current time during each slot
> sync cycle is that calling GetCurrentTimestamp() very frequently
> (during each sync cycle of very active slots) could be too costly.

Right.

thanks
Shveta

  160. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-26T11:05:16Z

    On Tue, Mar 26, 2024 at 4:18 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> > What about another approach?: inactive_since gives data synced from primary for
> > synced slots and another dedicated field (could be added later...) could
> > represent what you suggest as the other option.
>
> Yes, okay with me. I think there is some confusion here as well. In my
> second approach above, I have not suggested anything related to
> sync-worker. We can think on that later if we really need another
> field which give us sync time.  In my second approach, I have tried to
> avoid updating inactive_since for synced slots during sync process. We
> update that field during creation of synced slot so that
> inactive_since reflects correct info even for synced slots (rather
> than copying from primary). Please have a look at my patch and let me
> know your thoughts. I am fine with copying it from primary as well and
> documenting this behaviour.

I took a look at your patch.

--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
remote_dbid)
         SpinLockAcquire(&slot->mutex);
         slot->effective_catalog_xmin = xmin_horizon;
         slot->data.catalog_xmin = xmin_horizon;
+        slot->inactive_since = GetCurrentTimestamp();
         SpinLockRelease(&slot->mutex);

If we just sync inactive_since value for synced slots while in
recovery from the primary, so be it. Why do we need to update it to
the current time when the slot is being created? We don't expose slot
creation time, no? Aren't we fine if we just sync the value from
primary and document that fact? After the promotion, we can reset it
to the current time so that it gets its own time. Do you see any
issues with it?

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  161. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-26T11:19:18Z

    On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 4:18 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > What about another approach?: inactive_since gives data synced from primary for
> > > synced slots and another dedicated field (could be added later...) could
> > > represent what you suggest as the other option.
> >
> > Yes, okay with me. I think there is some confusion here as well. In my
> > second approach above, I have not suggested anything related to
> > sync-worker. We can think on that later if we really need another
> > field which give us sync time.  In my second approach, I have tried to
> > avoid updating inactive_since for synced slots during sync process. We
> > update that field during creation of synced slot so that
> > inactive_since reflects correct info even for synced slots (rather
> > than copying from primary). Please have a look at my patch and let me
> > know your thoughts. I am fine with copying it from primary as well and
> > documenting this behaviour.
>
> I took a look at your patch.
>
> --- a/src/backend/replication/logical/slotsync.c
> +++ b/src/backend/replication/logical/slotsync.c
> @@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
> remote_dbid)
>          SpinLockAcquire(&slot->mutex);
>          slot->effective_catalog_xmin = xmin_horizon;
>          slot->data.catalog_xmin = xmin_horizon;
> +        slot->inactive_since = GetCurrentTimestamp();
>          SpinLockRelease(&slot->mutex);
>
> If we just sync inactive_since value for synced slots while in
> recovery from the primary, so be it. Why do we need to update it to
> the current time when the slot is being created?

If we update inactive_since  at synced slot's creation or during
restart (skipping setting it during sync), then this time reflects
actual 'inactive_since' for that particular synced slot.  Isn't that a
clear info for the user and in alignment of what the name
'inactive_since' actually suggests?

> We don't expose slot
> creation time, no?

No, we don't. But for synced slot, that is the time since that slot is
inactive  (unless promoted), so we are exposing inactive_since and not
creation time.

>Aren't we fine if we just sync the value from
> primary and document that fact? After the promotion, we can reset it
> to the current time so that it gets its own time. Do you see any
> issues with it?

Yes, we can do that. But curious to know, do we see any additional
benefit of reflecting primary's inactive_since at standby which I
might be missing?

thanks
Shveta

  162. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T12:31:08Z

    Hi,

On Tue, Mar 26, 2024 at 04:49:18PM +0530, shveta malik wrote:
> On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Tue, Mar 26, 2024 at 4:18 PM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > > What about another approach?: inactive_since gives data synced from primary for
> > > > synced slots and another dedicated field (could be added later...) could
> > > > represent what you suggest as the other option.
> > >
> > > Yes, okay with me. I think there is some confusion here as well. In my
> > > second approach above, I have not suggested anything related to
> > > sync-worker. We can think on that later if we really need another
> > > field which give us sync time.  In my second approach, I have tried to
> > > avoid updating inactive_since for synced slots during sync process. We
> > > update that field during creation of synced slot so that
> > > inactive_since reflects correct info even for synced slots (rather
> > > than copying from primary). Please have a look at my patch and let me
> > > know your thoughts. I am fine with copying it from primary as well and
> > > documenting this behaviour.
> >
> > I took a look at your patch.
> >
> > --- a/src/backend/replication/logical/slotsync.c
> > +++ b/src/backend/replication/logical/slotsync.c
> > @@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
> > remote_dbid)
> >          SpinLockAcquire(&slot->mutex);
> >          slot->effective_catalog_xmin = xmin_horizon;
> >          slot->data.catalog_xmin = xmin_horizon;
> > +        slot->inactive_since = GetCurrentTimestamp();
> >          SpinLockRelease(&slot->mutex);
> >
> > If we just sync inactive_since value for synced slots while in
> > recovery from the primary, so be it. Why do we need to update it to
> > the current time when the slot is being created?
> 
> If we update inactive_since  at synced slot's creation or during
> restart (skipping setting it during sync), then this time reflects
> actual 'inactive_since' for that particular synced slot.  Isn't that a
> clear info for the user and in alignment of what the name
> 'inactive_since' actually suggests?
> 
> > We don't expose slot
> > creation time, no?
> 
> No, we don't. But for synced slot, that is the time since that slot is
> inactive  (unless promoted), so we are exposing inactive_since and not
> creation time.
> 
> >Aren't we fine if we just sync the value from
> > primary and document that fact? After the promotion, we can reset it
> > to the current time so that it gets its own time. Do you see any
> > issues with it?
> 
> Yes, we can do that. But curious to know, do we see any additional
> benefit of reflecting primary's inactive_since at standby which I
> might be missing?

In case the primary goes down, then one could use the value on the standby
to get the value coming from the primary. I think that could be useful info to
have.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  163. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T12:35:04Z

    Hi,

On Tue, Mar 26, 2024 at 04:17:53PM +0530, shveta malik wrote:
> On Tue, Mar 26, 2024 at 3:50 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Hi,
> >
> > > I think there may have been some misunderstanding here.
> >
> > Indeed ;-)
> >
> > > But now if I
> > > rethink this, I am fine with 'inactive_since' getting synced from
> > > primary to standby. But if we do that, we need to add docs stating
> > > "inactive_since" represents primary's inactivity and not standby's
> > > slots inactivity for synced slots.
> >
> > Yeah sure.
> >
> > > The reason for this clarification
> > > is that the synced slot might be generated much later, yet
> > > 'inactive_since' is synced from the primary, potentially indicating a
> > > time considerably earlier than when the synced slot was actually
> > > created.
> >
> > Right.
> >
> > > Another approach could be that "inactive_since" for synced slot
> > > actually gives its own inactivity data rather than giving primary's
> > > slot data. We update inactive_since on standby only at 3 occasions:
> > > 1) at the time of creation of the synced slot.
> > > 2) during standby restart.
> > > 3) during promotion of standby.
> > >
> > > I have attached a sample patch for this idea as.txt file.
> >
> > Thanks!
> >
> > > I am fine with any of these approaches.  One gives data synced from
> > > primary for synced slots, while another gives actual inactivity data
> > > of synced slots.
> >
> > What about another approach?: inactive_since gives data synced from primary for
> > synced slots and another dedicated field (could be added later...) could
> > represent what you suggest as the other option.
> 
> Yes, okay with me. I think there is some confusion here as well. In my
> second approach above, I have not suggested anything related to
> sync-worker.

Yeah, no confusion, understood that way.

> We can think on that later if we really need another
> field which give us sync time.

I think that calling GetCurrentTimestamp() so frequently could be too costly, so
I'm not sure we should.

> In my second approach, I have tried to
> avoid updating inactive_since for synced slots during sync process. We
> update that field during creation of synced slot so that
> inactive_since reflects correct info even for synced slots (rather
> than copying from primary). 

Yeah, and I think we could create a dedicated field with this information
if we feel the need.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  164. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-26T16:29:23Z

    On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> If we just sync inactive_since value for synced slots while in
> recovery from the primary, so be it. Why do we need to update it to
> the current time when the slot is being created? We don't expose slot
> creation time, no? Aren't we fine if we just sync the value from
> primary and document that fact? After the promotion, we can reset it
> to the current time so that it gets its own time.

I'm attaching v24 patches. It implements the above idea proposed
upthread for synced slots. I've now separated
s/last_inactive_time/inactive_since and synced slots behaviour. Please
have a look.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  165. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-26T17:52:12Z

    Hi,

On Tue, Mar 26, 2024 at 09:59:23PM +0530, Bharath Rupireddy wrote:
> On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > If we just sync inactive_since value for synced slots while in
> > recovery from the primary, so be it. Why do we need to update it to
> > the current time when the slot is being created? We don't expose slot
> > creation time, no? Aren't we fine if we just sync the value from
> > primary and document that fact? After the promotion, we can reset it
> > to the current time so that it gets its own time.
> 
> I'm attaching v24 patches. It implements the above idea proposed
> upthread for synced slots. I've now separated
> s/last_inactive_time/inactive_since and synced slots behaviour. Please
> have a look.

Thanks!

==== v24-0001

It's now pure mechanical changes and it looks good to me.

==== v24-0002

1 ===

    This commit does two things:
    1) Updates inactive_since for sync slots with the value
    received from the primary's slot.

Tested it and it does that.

2 ===

    2) Ensures the value is set to current timestamp during the
    shutdown of slot sync machinery to help correctly interpret the
    time if the standby gets promoted without a restart.

Tested it and it does that.

3 ===

+/*
+ * Reset the synced slots info such as inactive_since after shutting
+ * down the slot sync machinery.
+ */
+static void
+update_synced_slots_inactive_time(void)

Looks like the comment "reset" is not matching the name of the function and
what it does.

4 ===

+                       /*
+                        * We get the current time beforehand and only once to avoid
+                        * system calls overhead while holding the lock.
+                        */
+                       if (now == 0)
+                               now = GetCurrentTimestamp();

Also +1 of having GetCurrentTimestamp() just called one time within the loop.

5 ===

-               if (!(RecoveryInProgress() && slot->data.synced))
+               if (!(InRecovery && slot->data.synced))
                        slot->inactive_since = GetCurrentTimestamp();
                else
                        slot->inactive_since = 0;

Not related to this change but more the way RestoreSlotFromDisk() behaves here:

For a sync slot on standby it will be set to zero and then later will be
synchronized with the one coming from the primary. I think that's fine to have
it to zero for this window of time.

Now, if the standby is down and one sets sync_replication_slots to off,
then inactive_since will be set to zero on the standby at startup and not 
synchronized (unless one triggers a manual sync). I also think that's fine but
it might be worth to document this behavior (that after a standby startup
inactive_since is zero until the next sync...). 

6 ===

+       print "HI  $slot_name $name $inactive_since $slot_creation_time\n";

garbage?

7 ===

+# Capture and validate inactive_since of a given slot.
+sub capture_and_validate_slot_inactive_since
+{
+       my ($node, $slot_name, $slot_creation_time) = @_;
+       my $name = $node->name;

We know have capture_and_validate_slot_inactive_since at 2 places:
040_standby_failover_slots_sync.pl and 019_replslot_limit.pl.

Worth to create a sub in Cluster.pm?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  166. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-27T03:31:50Z

    On Tue, Mar 26, 2024 at 9:59 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > If we just sync inactive_since value for synced slots while in
> > recovery from the primary, so be it. Why do we need to update it to
> > the current time when the slot is being created? We don't expose slot
> > creation time, no? Aren't we fine if we just sync the value from
> > primary and document that fact? After the promotion, we can reset it
> > to the current time so that it gets its own time.
>
> I'm attaching v24 patches. It implements the above idea proposed
> upthread for synced slots. I've now separated
> s/last_inactive_time/inactive_since and synced slots behaviour. Please
> have a look.

Thanks for the patches. Few trivial comments for v24-002:

1)
slot.c:
+ * data from the remote slot. We use InRecovery flag instead of
+ * RecoveryInProgress() as it always returns true even for normal
+ * server startup.

a) Not clear what 'it' refers to. Better to use 'the latter'
b) Is it better to mention the primary here:
 'as the latter always returns true even on the primary server during startup'.


2)
update_local_synced_slot():

- strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0)
+ strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0 &&
+ remote_slot->inactive_since == slot->inactive_since)

When this code was written initially, the intent was to do strcmp at
the end (only if absolutely needed). It will be good if we maintain
the same and add new checks before strcmp.

3)
update_synced_slots_inactive_time():

This assert is removed, is it intentional?
Assert(s->active_pid == 0);


4)
040_standby_failover_slots_sync.pl:

+# Capture the inactive_since of the slot from the standby the logical failover
+# slots are synced/created on the standby.

The comment is unclear, something seems missing.

thanks
Shveta

  167. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-27T04:38:33Z

    On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > I'm attaching v24 patches. It implements the above idea proposed
> > upthread for synced slots.
>
> ==== v24-0002
>
> 1 ===
>
>     This commit does two things:
>     1) Updates inactive_since for sync slots with the value
>     received from the primary's slot.
>
> Tested it and it does that.

Thanks. I've added a test case for this.

> 2 ===
>
>     2) Ensures the value is set to current timestamp during the
>     shutdown of slot sync machinery to help correctly interpret the
>     time if the standby gets promoted without a restart.
>
> Tested it and it does that.

Thanks. I've added a test case for this.

> 3 ===
>
> +/*
> + * Reset the synced slots info such as inactive_since after shutting
> + * down the slot sync machinery.
> + */
> +static void
> +update_synced_slots_inactive_time(void)
>
> Looks like the comment "reset" is not matching the name of the function and
> what it does.

Changed. I've also changed the function name to
update_synced_slots_inactive_since to be precise on what it exactly
does.

> 4 ===
>
> +                       /*
> +                        * We get the current time beforehand and only once to avoid
> +                        * system calls overhead while holding the lock.
> +                        */
> +                       if (now == 0)
> +                               now = GetCurrentTimestamp();
>
> Also +1 of having GetCurrentTimestamp() just called one time within the loop.

Right.

> 5 ===
>
> -               if (!(RecoveryInProgress() && slot->data.synced))
> +               if (!(InRecovery && slot->data.synced))
>                         slot->inactive_since = GetCurrentTimestamp();
>                 else
>                         slot->inactive_since = 0;
>
> Not related to this change but more the way RestoreSlotFromDisk() behaves here:
>
> For a sync slot on standby it will be set to zero and then later will be
> synchronized with the one coming from the primary. I think that's fine to have
> it to zero for this window of time.

Right.

> Now, if the standby is down and one sets sync_replication_slots to off,
> then inactive_since will be set to zero on the standby at startup and not
> synchronized (unless one triggers a manual sync). I also think that's fine but
> it might be worth to document this behavior (that after a standby startup
> inactive_since is zero until the next sync...).

Isn't this behaviour applicable for other slot parameters that the
slot syncs from the remote slot on the primary?

I've added the following note in the comments when we update
inactive_since in RestoreSlotFromDisk.

         * Note that for synced slots after the standby starts up (i.e. after
         * the slots are loaded from the disk), the inactive_since will remain
         * zero until the next slot sync cycle.
         */
        if (!(InRecovery && slot->data.synced))
            slot->inactive_since = GetCurrentTimestamp();
        else
            slot->inactive_since = 0;

> 6 ===
>
> +       print "HI  $slot_name $name $inactive_since $slot_creation_time\n";
>
> garbage?

Removed.

> 7 ===
>
> +# Capture and validate inactive_since of a given slot.
> +sub capture_and_validate_slot_inactive_since
> +{
> +       my ($node, $slot_name, $slot_creation_time) = @_;
> +       my $name = $node->name;
>
> We know have capture_and_validate_slot_inactive_since at 2 places:
> 040_standby_failover_slots_sync.pl and 019_replslot_limit.pl.
>
> Worth to create a sub in Cluster.pm?

I'd second that thought for now. We might have to debate first if it's
useful for all the nodes even without replication, and if yes, the
naming stuff and all that. Historically, we've had such duplicated
functions until recently, for instance advance_wal and log_contains.
We
moved them over to a common perl library Cluster.pm very recently. I'm
sure we can come back later to move it to Cluster.pm.

On Wed, Mar 27, 2024 at 9:02 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> 1)
> slot.c:
> + * data from the remote slot. We use InRecovery flag instead of
> + * RecoveryInProgress() as it always returns true even for normal
> + * server startup.
>
> a) Not clear what 'it' refers to. Better to use 'the latter'
> b) Is it better to mention the primary here:
>  'as the latter always returns true even on the primary server during startup'.

Modified.

> 2)
> update_local_synced_slot():
>
> - strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0)
> + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0 &&
> + remote_slot->inactive_since == slot->inactive_since)
>
> When this code was written initially, the intent was to do strcmp at
> the end (only if absolutely needed). It will be good if we maintain
> the same and add new checks before strcmp.

Done.

> 3)
> update_synced_slots_inactive_time():
>
> This assert is removed, is it intentional?
> Assert(s->active_pid == 0);

Yes, the slot can get acquired in the corner case when someone runs
pg_sync_replication_slots concurrently at this time. I'm referring to
the issue reported upthread. We don't prevent one running
pg_sync_replication_slots in promotion/ShutDownSlotSync phase right?
Maybe we should prevent that otherwise some of the slots are synced
and the standby gets promoted while others are yet-to-be-synced.

> 4)
> 040_standby_failover_slots_sync.pl:
>
> +# Capture the inactive_since of the slot from the standby the logical failover
> +# slots are synced/created on the standby.
>
> The comment is unclear, something seems missing.

Nice catch. Yes, that was wrong. I've modified it now.

Please find the attached v25-0001 (made this 0001 patch now as
inactive_since patch is committed) patch with the above changes.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  168. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-27T04:52:30Z

    On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
>
> > 3)
> > update_synced_slots_inactive_time():
> >
> > This assert is removed, is it intentional?
> > Assert(s->active_pid == 0);
>
> Yes, the slot can get acquired in the corner case when someone runs
> pg_sync_replication_slots concurrently at this time. I'm referring to
> the issue reported upthread. We don't prevent one running
> pg_sync_replication_slots in promotion/ShutDownSlotSync phase right?
> Maybe we should prevent that otherwise some of the slots are synced
> and the standby gets promoted while others are yet-to-be-synced.
>

We should do something about it but that shouldn't be done in this
patch. We can handle it separately and then add such an assert.

-- 
With Regards,
Amit Kapila.

  169. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-27T04:54:32Z

    On Wed, Mar 27, 2024 at 10:22 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > 3)
> > > update_synced_slots_inactive_time():
> > >
> > > This assert is removed, is it intentional?
> > > Assert(s->active_pid == 0);
> >
> > Yes, the slot can get acquired in the corner case when someone runs
> > pg_sync_replication_slots concurrently at this time. I'm referring to
> > the issue reported upthread. We don't prevent one running
> > pg_sync_replication_slots in promotion/ShutDownSlotSync phase right?
> > Maybe we should prevent that otherwise some of the slots are synced
> > and the standby gets promoted while others are yet-to-be-synced.
> >
>
> We should do something about it but that shouldn't be done in this
> patch. We can handle it separately and then add such an assert.

Agreed. Once this patch is concluded, I can fix the slot sync shutdown
issue and will also add this 'assert' back.

thanks
Shveta

  170. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-27T04:56:01Z

    On Wed, Mar 27, 2024 at 10:24 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Mar 27, 2024 at 10:22 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy
> > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > >
> > > On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot
> > > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > > 3)
> > > > update_synced_slots_inactive_time():
> > > >
> > > > This assert is removed, is it intentional?
> > > > Assert(s->active_pid == 0);
> > >
> > > Yes, the slot can get acquired in the corner case when someone runs
> > > pg_sync_replication_slots concurrently at this time. I'm referring to
> > > the issue reported upthread. We don't prevent one running
> > > pg_sync_replication_slots in promotion/ShutDownSlotSync phase right?
> > > Maybe we should prevent that otherwise some of the slots are synced
> > > and the standby gets promoted while others are yet-to-be-synced.
> > >
> >
> > We should do something about it but that shouldn't be done in this
> > patch. We can handle it separately and then add such an assert.
>
> Agreed. Once this patch is concluded, I can fix the slot sync shutdown
> issue and will also add this 'assert' back.

Agreed. Thanks.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  171. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-27T05:25:29Z

    On Tue, Mar 26, 2024 at 6:05 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
>
> > We can think on that later if we really need another
> > field which give us sync time.
>
> I think that calling GetCurrentTimestamp() so frequently could be too costly, so
> I'm not sure we should.

Agreed.

> > In my second approach, I have tried to
> > avoid updating inactive_since for synced slots during sync process. We
> > update that field during creation of synced slot so that
> > inactive_since reflects correct info even for synced slots (rather
> > than copying from primary).
>
> Yeah, and I think we could create a dedicated field with this information
> if we feel the need.

Okay.

thanks
Shveta

  172. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-27T05:35:04Z

    On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Please find the attached v25-0001 (made this 0001 patch now as
> inactive_since patch is committed) patch with the above changes.

Fixed an issue in synchronize_slots where DatumGetLSN is being used in
place of DatumGetTimestampTz. Found this via CF bot member [1], not on
my dev system.

Please find the attached v6 patch.


[1]
[05:14:39.281] #7  DatumGetLSN (X=<optimized out>) at
../src/include/utils/pg_lsn.h:24
[05:14:39.281] No locals.
[05:14:39.281] #8  synchronize_slots (wrconn=wrconn@entry=0x583cd170)
at ../src/backend/replication/logical/slotsync.c:757
[05:14:39.281]         isnull = false
[05:14:39.281]         remote_slot = 0x583ce1a8
[05:14:39.281]         d = <optimized out>
[05:14:39.281]         col = 10
[05:14:39.281]         slotRow = {25, 25, 3220, 3220, 28, 16, 16, 25, 25, 1184}
[05:14:39.281]         res = 0x583cd1b8
[05:14:39.281]         tupslot = 0x583ce11c
[05:14:39.281]         remote_slot_list = 0x0
[05:14:39.281]         some_slot_updated = false
[05:14:39.281]         started_tx = false
[05:14:39.281]         query = 0x57692bc4 "SELECT slot_name, plugin,
confirmed_flush_lsn, restart_lsn, catalog_xmin, two_phase, failover,
database, invalidation_reason, inactive_since FROM
pg_catalog.pg_replication_slots WHERE failover and NOT"...
[05:14:39.281]         __func__ = "synchronize_slots"
[05:14:39.281] #9  0x56ff9d1e in SyncReplicationSlots
(wrconn=0x583cd170) at
../src/backend/replication/logical/slotsync.c:1504

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  173. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-27T05:48:46Z

    Hi,

On Wed, Mar 27, 2024 at 10:08:33AM +0530, Bharath Rupireddy wrote:
> On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > -               if (!(RecoveryInProgress() && slot->data.synced))
> > +               if (!(InRecovery && slot->data.synced))
> >                         slot->inactive_since = GetCurrentTimestamp();
> >                 else
> >                         slot->inactive_since = 0;
> >
> > Not related to this change but more the way RestoreSlotFromDisk() behaves here:
> >
> > For a sync slot on standby it will be set to zero and then later will be
> > synchronized with the one coming from the primary. I think that's fine to have
> > it to zero for this window of time.
> 
> Right.
> 
> > Now, if the standby is down and one sets sync_replication_slots to off,
> > then inactive_since will be set to zero on the standby at startup and not
> > synchronized (unless one triggers a manual sync). I also think that's fine but
> > it might be worth to document this behavior (that after a standby startup
> > inactive_since is zero until the next sync...).
> 
> Isn't this behaviour applicable for other slot parameters that the
> slot syncs from the remote slot on the primary?

No they are persisted on disk. If not, we'd not know where to resume the decoding
from on the standby in case primary is down and/or sync is off.

> I've added the following note in the comments when we update
> inactive_since in RestoreSlotFromDisk.
> 
>          * Note that for synced slots after the standby starts up (i.e. after
>          * the slots are loaded from the disk), the inactive_since will remain
>          * zero until the next slot sync cycle.
>          */
>         if (!(InRecovery && slot->data.synced))
>             slot->inactive_since = GetCurrentTimestamp();
>         else
>             slot->inactive_since = 0;

I think we should add some words in the doc too and also about what the meaning
of inactive_since on the standby is (as suggested by Shveta in [1]).

[1]: https://www.postgresql.org/message-id/CAJpy0uDkTW%2Bt1k3oPkaipFBzZePfFNB5DmiA%3D%3DpxRGcAdpF%3DPg%40mail.gmail.com

> > 7 ===
> >
> > +# Capture and validate inactive_since of a given slot.
> > +sub capture_and_validate_slot_inactive_since
> > +{
> > +       my ($node, $slot_name, $slot_creation_time) = @_;
> > +       my $name = $node->name;
> >
> > We know have capture_and_validate_slot_inactive_since at 2 places:
> > 040_standby_failover_slots_sync.pl and 019_replslot_limit.pl.
> >
> > Worth to create a sub in Cluster.pm?
> 
> I'd second that thought for now. We might have to debate first if it's
> useful for all the nodes even without replication, and if yes, the
> naming stuff and all that. Historically, we've had such duplicated
> functions until recently, for instance advance_wal and log_contains.
> We
> moved them over to a common perl library Cluster.pm very recently. I'm
> sure we can come back later to move it to Cluster.pm.

I thought that would be the right time not to introduce duplicated code.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  174. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-27T06:09:04Z

    On Wed, Mar 27, 2024 at 11:05 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Fixed an issue in synchronize_slots where DatumGetLSN is being used in
> place of DatumGetTimestampTz. Found this via CF bot member [1], not on
> my dev system.
>
> Please find the attached v6 patch.

Thanks for the patch. Few trivial things:

----------
1)
system-views.sgml:

a) "Note that the slots" --> "Note that the slots on the standbys,"
--it is good to mention "standbys" as synced could be true on primary
as well (promoted standby)

b) If you plan to add more info which Bertrand suggested, then it will
be better to make a <note> section instead of using "Note"

2)
commit msg:

"The impact of this
on a promoted standby inactive_since is always NULL for all
synced slots even after server restart.
"
Sentence looks broken.
---------

Apart from the above trivial things, v26-001 looks good to me.

thanks
Shveta

  175. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-27T09:25:17Z

    On Wed, Mar 27, 2024 at 11:39 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> Thanks for the patch. Few trivial things:

Thanks for reviewing.

> ----------
> 1)
> system-views.sgml:
>
> a) "Note that the slots" --> "Note that the slots on the standbys,"
> --it is good to mention "standbys" as synced could be true on primary
> as well (promoted standby)

Done.

> b) If you plan to add more info which Bertrand suggested, then it will
> be better to make a <note> section instead of using "Note"

I added the note that Bertrand specified upthread. But, I couldn't
find an instance of adding <note> ... </note> within a table. Hence
with "Note that ...." statments just like any other notes in the
system-views.sgml. pg_replication_slot in system-vews.sgml renders as
table, so having <note> ... </note> may not be a great idea.

> 2)
> commit msg:
>
> "The impact of this
> on a promoted standby inactive_since is always NULL for all
> synced slots even after server restart.
> "
> Sentence looks broken.
> ---------

Reworded.

> Apart from the above trivial things, v26-001 looks good to me.

Please check the attached v27 patch which also has Bertrand's comment
on deduplicating the TAP function. I've now moved it to Cluster.pm.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  176. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-27T10:11:58Z

    Hi,

On Wed, Mar 27, 2024 at 02:55:17PM +0530, Bharath Rupireddy wrote:
> Please check the attached v27 patch which also has Bertrand's comment
> on deduplicating the TAP function. I've now moved it to Cluster.pm.

Thanks!

1 ===

+        Note that the slots on the standbys that are being synced from a
+        primary server (whose <structfield>synced</structfield> field is
+        <literal>true</literal>), will get the
+        <structfield>inactive_since</structfield> value from the
+        corresponding remote slot on the primary. Also, note that for the
+        synced slots on the standby, after the standby starts up (i.e. after
+        the slots are loaded from the disk), the inactive_since will remain
+        zero until the next slot sync cycle.

Not sure we should mention the "(i.e. after the slots are loaded from the disk)"
and also "cycle" (as that does not sound right in case of manual sync).

My proposal (in text) but feel free to reword it:

Note that the slots on the standbys that are being synced from a
primary server (whose synced field is true), will get the inactive_since value
from the corresponding remote slot on the primary. Also, after the standby starts
up, the inactive_since (for such synced slots) will remain zero until the next
synchronization.

2 ===

+=item $node->create_logical_slot_on_standby(self, primary, slot_name, dbname)

get_slot_inactive_since_value instead?

3 ===

+against given reference time.

s/given reference/optional given reference/?


Apart from the above, LGTM.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  177. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-27T10:13:22Z

    On Wed, Mar 27, 2024 at 2:55 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Mar 27, 2024 at 11:39 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > Thanks for the patch. Few trivial things:
>
> Thanks for reviewing.
>
> > ----------
> > 1)
> > system-views.sgml:
> >
> > a) "Note that the slots" --> "Note that the slots on the standbys,"
> > --it is good to mention "standbys" as synced could be true on primary
> > as well (promoted standby)
>
> Done.
>
> > b) If you plan to add more info which Bertrand suggested, then it will
> > be better to make a <note> section instead of using "Note"
>
> I added the note that Bertrand specified upthread. But, I couldn't
> find an instance of adding <note> ... </note> within a table. Hence
> with "Note that ...." statments just like any other notes in the
> system-views.sgml. pg_replication_slot in system-vews.sgml renders as
> table, so having <note> ... </note> may not be a great idea.
>
> > 2)
> > commit msg:
> >
> > "The impact of this
> > on a promoted standby inactive_since is always NULL for all
> > synced slots even after server restart.
> > "
> > Sentence looks broken.
> > ---------
>
> Reworded.
>
> > Apart from the above trivial things, v26-001 looks good to me.
>
> Please check the attached v27 patch which also has Bertrand's comment
> on deduplicating the TAP function. I've now moved it to Cluster.pm.
>

Thanks for the patch. Regarding doc, I have few comments.

+        Note that the slots on the standbys that are being synced from a
+        primary server (whose <structfield>synced</structfield> field is
+        <literal>true</literal>), will get the
+        <structfield>inactive_since</structfield> value from the
+        corresponding remote slot on the primary. Also, note that for the
+        synced slots on the standby, after the standby starts up (i.e. after
+        the slots are loaded from the disk), the inactive_since will remain
+        zero until the next slot sync cycle.

a)  "inactive_since will remain  zero"
Since it is user exposed info and the user finds it NULL in
pg_replication_slots, shall we mention NULL instead of 0?

b) Since we are referring to the sync cycle here, I feel it will be
good to give a link to that page.
+        zero until the next slot sync cycle (see
+        <xref linkend="logicaldecoding-replication-slots-synchronization"/> for
+        slot synchronization details).

thanks
Shveta

  178. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-27T12:25:05Z

    On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> 1 ===
>
> My proposal (in text) but feel free to reword it:
>
> Note that the slots on the standbys that are being synced from a
> primary server (whose synced field is true), will get the inactive_since value
> from the corresponding remote slot on the primary. Also, after the standby starts
> up, the inactive_since (for such synced slots) will remain zero until the next
> synchronization.

WFM.

> 2 ===
>
> +=item $node->create_logical_slot_on_standby(self, primary, slot_name, dbname)
>
> get_slot_inactive_since_value instead?

Ugh. Changed.

> 3 ===
>
> +against given reference time.
>
> s/given reference/optional given reference/?

Done.

> Apart from the above, LGTM.

Thanks for reviewing.

On Wed, Mar 27, 2024 at 3:43 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> Thanks for the patch. Regarding doc, I have few comments.

Thanks for reviewing.

> a)  "inactive_since will remain  zero"
> Since it is user exposed info and the user finds it NULL in
> pg_replication_slots, shall we mention NULL instead of 0?

Right. Changed.

> b) Since we are referring to the sync cycle here, I feel it will be
> good to give a link to that page.
> +        zero until the next slot sync cycle (see
> +        <xref linkend="logicaldecoding-replication-slots-synchronization"/> for
> +        slot synchronization details).

WFM.

Please see the attached v28 patch.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  179. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-27T13:24:52Z

    Hi,

On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote:
> On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot
> Please see the attached v28 patch.

Thanks!

1 === sorry I missed it in the previous review

        if (!(RecoveryInProgress() && slot->data.synced))
+       {
                now = GetCurrentTimestamp();
+               update_inactive_since = true;
+       }
+       else
+               update_inactive_since = false;

I think update_inactive_since is not needed, we could rely on (now > 0) instead.

2 ===

+=item $node->get_slot_inactive_since_value(self, primary, slot_name, dbname)
+
+Get inactive_since column value for a given replication slot validating it
+against optional reference time.
+
+=cut
+
+sub get_slot_inactive_since_value
+{

shouldn't be "=item $node->get_slot_inactive_since_value(self, slot_name, reference_time)"
instead?

Apart from the above, LGTM.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  180. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-27T15:30:37Z

    On Wed, Mar 27, 2024 at 6:54 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote:
> > On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot
> > Please see the attached v28 patch.
>
> Thanks!
>
> 1 === sorry I missed it in the previous review
>
>         if (!(RecoveryInProgress() && slot->data.synced))
> +       {
>                 now = GetCurrentTimestamp();
> +               update_inactive_since = true;
> +       }
> +       else
> +               update_inactive_since = false;
>
> I think update_inactive_since is not needed, we could rely on (now > 0) instead.

Thought of using it, but, at the expense of readability. I prefer to
use a variable instead. However, I changed the variable to be more
meaningful to is_slot_being_synced.

> 2 ===
>
> +=item $node->get_slot_inactive_since_value(self, primary, slot_name, dbname)
> +
> +Get inactive_since column value for a given replication slot validating it
> +against optional reference time.
> +
> +=cut
> +
> +sub get_slot_inactive_since_value
> +{
>
> shouldn't be "=item $node->get_slot_inactive_since_value(self, slot_name, reference_time)"
> instead?

Ugh. Changed.

> Apart from the above, LGTM.

Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the
standby for sync slots. 0002 implementing inactive timeout GUC based
invalidation mechanism.

Please have a look.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  181. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-27T16:12:23Z

    Hi,

On Wed, Mar 27, 2024 at 09:00:37PM +0530, Bharath Rupireddy wrote:
> On Wed, Mar 27, 2024 at 6:54 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Hi,
> >
> > On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote:
> > > On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot
> > > Please see the attached v28 patch.
> >
> > Thanks!
> >
> > 1 === sorry I missed it in the previous review
> >
> >         if (!(RecoveryInProgress() && slot->data.synced))
> > +       {
> >                 now = GetCurrentTimestamp();
> > +               update_inactive_since = true;
> > +       }
> > +       else
> > +               update_inactive_since = false;
> >
> > I think update_inactive_since is not needed, we could rely on (now > 0) instead.
> 
> Thought of using it, but, at the expense of readability. I prefer to
> use a variable instead.

That's fine too.

> However, I changed the variable to be more meaningful to is_slot_being_synced.

Yeah makes sense and even easier to read.

v29-0001 LGTM.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  182. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-03-28T03:46:17Z

    On Wed, Mar 27, 2024 at 9:00 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the
> standby for sync slots. 0002 implementing inactive timeout GUC based
> invalidation mechanism.
>
> Please have a look.

Thanks for the patches. v29-001 looks good to me.

thanks
Shveta

  183. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-28T09:43:44Z

    Hi,

On Wed, Mar 27, 2024 at 09:00:37PM +0530, Bharath Rupireddy wrote:
> standby for sync slots. 0002 implementing inactive timeout GUC based
> invalidation mechanism.
> 
> Please have a look.

Thanks!

Regarding 0002:

Some testing:

T1 ===

When the slot is invalidated on the primary, then the reason is propagated to
the sync slot (if any). That's fine but we are loosing the inactive_since on the
standby:

Primary:

postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot';
  slot_name  |        inactive_since         | conflicting | invalidation_reason
-------------+-------------------------------+-------------+---------------------
 lsub29_slot | 2024-03-28 08:24:51.672528+00 | f           | inactive_timeout
(1 row)

Standby:

postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot';
  slot_name  | inactive_since | conflicting | invalidation_reason
-------------+----------------+-------------+---------------------
 lsub29_slot |                | f           | inactive_timeout
(1 row)

I think in this case it should always reflect the value from the primary (so
that one can understand why it is invalidated).

T2 ===

And it is set to a value during promotion:

postgres=# select pg_promote();
 pg_promote
------------
 t
(1 row)

postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot';
  slot_name  |        inactive_since        | conflicting | invalidation_reason
-------------+------------------------------+-------------+---------------------
 lsub29_slot | 2024-03-28 08:30:11.74505+00 | f           | inactive_timeout
(1 row)

I think when it is invalidated it should always reflect the value from the
primary (so that one can understand why it is invalidated).

T3 ===

As far the slot invalidation on the primary:

postgres=# SELECT * FROM pg_logical_slot_get_changes('lsub29_slot', NULL, NULL, 'include-xids', '0');
ERROR:  cannot acquire invalidated replication slot "lsub29_slot"

Can we make the message more consistent with what can be found in CreateDecodingContext()
for example?

T4 ===

Also, it looks like querying pg_replication_slots() does not trigger an
invalidation: I think it should if the slot is not invalidated yet (and matches
the invalidation criteria).

Code review:

CR1 ===

+        Invalidate replication slots that are inactive for longer than this
+        amount of time. If this value is specified without units, it is taken

s/Invalidate/Invalidates/?

Should we mention the relationship with inactive_since?

CR2 ===

+ *
+ * If check_for_invalidation is true, the slot is checked for invalidation
+ * based on replication_slot_inactive_timeout GUC and an error is raised after making the slot ours.
  */
 void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+                                          bool check_for_invalidation)


s/check_for_invalidation/check_for_timeout_invalidation/?

CR3 ===

+       if (slot->inactive_since == 0 ||
+               replication_slot_inactive_timeout == 0)
+               return false;

Better to test replication_slot_inactive_timeout first? (I mean there is no
point of testing inactive_since if replication_slot_inactive_timeout == 0)

CR4 ===

+       if (slot->inactive_since > 0 &&
+               replication_slot_inactive_timeout > 0)
+       {

Same.

So, instead of CR3 === and CR4 ===, I wonder if it wouldn't be better to do
something like:

if (replication_slot_inactive_timeout == 0)
	return false;
else if (slot->inactive_since > 0)
.
.
.
.
else
	return false;

That would avoid checking replication_slot_inactive_timeout and inactive_since
multiple times.

CR5 ===

+        * held to avoid race conditions -- for example the restart_lsn could move
+        * forward, or the slot could be dropped.

Does the restart_lsn example makes sense here?

CR6 ===

+static bool
+InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks)
+{

InvalidatePossiblyInactiveSlot() maybe?

CR7 ===

+       /* Make sure the invalidated state persists across server restart */
+       slot->just_dirtied = true;
+       slot->dirty = true;
+       SpinLockRelease(&slot->mutex);

Maybe we could create a new function say MarkGivenReplicationSlotDirty()
with a slot as parameter, that ReplicationSlotMarkDirty could call too?

Then maybe we could set slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT in
InvalidateSlotForInactiveTimeout()? (to avoid multiple SpinLockAcquire/SpinLockRelease).

CR8 ===

+       if (persist_state)
+       {
+               char            path[MAXPGPATH];
+
+               sprintf(path, "pg_replslot/%s", NameStr(slot->data.name));
+               SaveSlotToPath(slot, path, ERROR);
+       }

Maybe we could create a new function say GivenReplicationSlotSave()
with a slot as parameter, that ReplicationSlotSave() could call too?

CR9 ===

+       if (check_for_invalidation)
+       {
+               /* The slot is ours by now */
+               Assert(s->active_pid == MyProcPid);
+
+               /*
+                * Well, the slot is not yet ours really unless we check for the
+                * invalidation below.
+                */
+               s->active_pid = 0;
+               if (InvalidateReplicationSlotForInactiveTimeout(s, true, true))
+               {
+                       /*
+                        * If the slot has been invalidated, recalculate the resource
+                        * limits.
+                        */
+                       ReplicationSlotsComputeRequiredXmin(false);
+                       ReplicationSlotsComputeRequiredLSN();
+
+                       /* Might need it for slot clean up on error, so restore it */
+                       s->active_pid = MyProcPid;
+                       ereport(ERROR,
+                                       (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+                                        errmsg("cannot acquire invalidated replication slot \"%s\"",
+                                                       NameStr(MyReplicationSlot->data.name))));
+               }
+               s->active_pid = MyProcPid;

Are we not missing some SpinLockAcquire/Release on the slot's mutex here? (the
places where we set the active_pid).

CR10 ===

@@ -1628,6 +1674,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
                                        if (SlotIsLogical(s))
                                                invalidation_cause = cause;
                                        break;
+                               case RS_INVAL_INACTIVE_TIMEOUT:
+                                       if (InvalidateReplicationSlotForInactiveTimeout(s, false, false))
+                                               invalidation_cause = cause;
+                                       break;

InvalidatePossiblyObsoleteSlot() is not called with such a reason, better to use
an Assert here and in the caller too?

CR11 ===

+++ b/src/test/recovery/t/050_invalidate_slots.pl

why not using 019_replslot_limit.pl?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  184. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-29T04:09:31Z

    On Wed, Mar 27, 2024 at 9:00 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
>
> Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the
> standby for sync slots.
>

Commit message states: "why we can't just update inactive_since for
synced slots on the standby with the value received from remote slot
on the primary. This is consistent with any other slot parameter i.e.
all of them are synced from the primary."

The inactive_since is not consistent with other slot parameters which
we copy. We don't perform anything related to those other parameters
like say two_phase phase which can change that property. However, we
do acquire the slot, advance the slot (as per recent discussion [1]),
and release it. Since these operations can impact inactive_since, it
seems to me that inactive_since is not the same as other parameters.
It can have a different value than the primary. Why would anyone want
to know the value of inactive_since from primary after the standby is
promoted? Now, the other concern is that calling GetCurrentTimestamp()
could be costly when the values for the slot are not going to be
updated but if that happens we can optimize such that before acquiring
the slot we can have some minimal pre-checks to ensure whether we need
to update the slot or not.

[1] - https://www.postgresql.org/message-id/OS0PR01MB571615D35F486080616CA841943A2%40OS0PR01MB5716.jpnprd01.prod.outlook.com

-- 
With Regards,
Amit Kapila.

  185. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-29T06:19:05Z

    Hi,

On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote:
> On Wed, Mar 27, 2024 at 9:00 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> >
> > Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the
> > standby for sync slots.
> >
> 
> Commit message states: "why we can't just update inactive_since for
> synced slots on the standby with the value received from remote slot
> on the primary. This is consistent with any other slot parameter i.e.
> all of them are synced from the primary."
> 
> The inactive_since is not consistent with other slot parameters which
> we copy. We don't perform anything related to those other parameters
> like say two_phase phase which can change that property. However, we
> do acquire the slot, advance the slot (as per recent discussion [1]),
> and release it. Since these operations can impact inactive_since, it
> seems to me that inactive_since is not the same as other parameters.
> It can have a different value than the primary. Why would anyone want
> to know the value of inactive_since from primary after the standby is
> promoted?

I think it can be useful "before" it is promoted and in case the primary is down.
I agree that tracking the activity time of a synced slot can be useful, why
not creating a dedicated field for that purpose (and keep inactive_since a
perfect "copy" of the primary)?

> Now, the other concern is that calling GetCurrentTimestamp()
> could be costly when the values for the slot are not going to be
> updated but if that happens we can optimize such that before acquiring
> the slot we can have some minimal pre-checks to ensure whether we need
> to update the slot or not.

Right, but for a very active slot it is likely that we call GetCurrentTimestamp()
during almost each sync cycle.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  186. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-03-29T09:33:01Z

    On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote:
> >
> > Commit message states: "why we can't just update inactive_since for
> > synced slots on the standby with the value received from remote slot
> > on the primary. This is consistent with any other slot parameter i.e.
> > all of them are synced from the primary."
> >
> > The inactive_since is not consistent with other slot parameters which
> > we copy. We don't perform anything related to those other parameters
> > like say two_phase phase which can change that property. However, we
> > do acquire the slot, advance the slot (as per recent discussion [1]),
> > and release it. Since these operations can impact inactive_since, it
> > seems to me that inactive_since is not the same as other parameters.
> > It can have a different value than the primary. Why would anyone want
> > to know the value of inactive_since from primary after the standby is
> > promoted?
>
> I think it can be useful "before" it is promoted and in case the primary is down.
>

It is not clear to me what is user going to do by checking the
inactivity time for slots when the corresponding server is down. I
thought the idea was to check such slots and see if they need to be
dropped or enabled again to avoid excessive disk usage, etc.

> I agree that tracking the activity time of a synced slot can be useful, why
> not creating a dedicated field for that purpose (and keep inactive_since a
> perfect "copy" of the primary)?
>

We can have a separate field for this but not sure if it is worth it.

> > Now, the other concern is that calling GetCurrentTimestamp()
> > could be costly when the values for the slot are not going to be
> > updated but if that happens we can optimize such that before acquiring
> > the slot we can have some minimal pre-checks to ensure whether we need
> > to update the slot or not.
>
> Right, but for a very active slot it is likely that we call GetCurrentTimestamp()
> during almost each sync cycle.
>

True, but if we have to save a slot to disk each time to persist the
changes (for an active slot) then probably GetCurrentTimestamp()
shouldn't be costly enough to matter.

-- 
With Regards,
Amit Kapila.

  187. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-03-29T12:47:51Z

    Hi,

On Fri, Mar 29, 2024 at 03:03:01PM +0530, Amit Kapila wrote:
> On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote:
> > >
> > > Commit message states: "why we can't just update inactive_since for
> > > synced slots on the standby with the value received from remote slot
> > > on the primary. This is consistent with any other slot parameter i.e.
> > > all of them are synced from the primary."
> > >
> > > The inactive_since is not consistent with other slot parameters which
> > > we copy. We don't perform anything related to those other parameters
> > > like say two_phase phase which can change that property. However, we
> > > do acquire the slot, advance the slot (as per recent discussion [1]),
> > > and release it. Since these operations can impact inactive_since, it
> > > seems to me that inactive_since is not the same as other parameters.
> > > It can have a different value than the primary. Why would anyone want
> > > to know the value of inactive_since from primary after the standby is
> > > promoted?
> >
> > I think it can be useful "before" it is promoted and in case the primary is down.
> >
> 
> It is not clear to me what is user going to do by checking the
> inactivity time for slots when the corresponding server is down.

Say a failover needs to be done, then it could be useful to know for which
slots the activity needs to be resumed (thinking about external logical decoding
plugin, not about pub/sub here). If one see an inactive slot (since long "enough")
then he can start to reasonate about what to do with it.

> I thought the idea was to check such slots and see if they need to be
> dropped or enabled again to avoid excessive disk usage, etc.

Yeah that's the case but it does not mean inactive_since can't be useful in other
ways.

Also, say the slot has been invalidated on the primary (due to inactivity timeout),
primary is down and there is a failover. By keeping the inactive_since from
the primary, one could know when the inactivity that lead to the timeout started.

Again, more concerned about external logical decoding plugin than pub/sub here.

> > I agree that tracking the activity time of a synced slot can be useful, why
> > not creating a dedicated field for that purpose (and keep inactive_since a
> > perfect "copy" of the primary)?
> >
> 
> We can have a separate field for this but not sure if it is worth it.

OTOH I'm not sure that erasing this information from the primary is useful. I
think that 2 fields would be the best option and would be less subject of
misinterpretation.

> > > Now, the other concern is that calling GetCurrentTimestamp()
> > > could be costly when the values for the slot are not going to be
> > > updated but if that happens we can optimize such that before acquiring
> > > the slot we can have some minimal pre-checks to ensure whether we need
> > > to update the slot or not.
> >
> > Right, but for a very active slot it is likely that we call GetCurrentTimestamp()
> > during almost each sync cycle.
> >
> 
> True, but if we have to save a slot to disk each time to persist the
> changes (for an active slot) then probably GetCurrentTimestamp()
> shouldn't be costly enough to matter.

Right, persisting the changes to disk would be even more costly.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  188. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-03-31T04:55:46Z

    On Thu, Mar 28, 2024 at 3:13 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Regarding 0002:

Thanks for reviewing it.

> Some testing:
>
> T1 ===
>
> When the slot is invalidated on the primary, then the reason is propagated to
> the sync slot (if any). That's fine but we are loosing the inactive_since on the
> standby:
>
> Primary:
>
> postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot';
>   slot_name  |        inactive_since         | conflicting | invalidation_reason
> -------------+-------------------------------+-------------+---------------------
>  lsub29_slot | 2024-03-28 08:24:51.672528+00 | f           | inactive_timeout
> (1 row)
>
> Standby:
>
> postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot';
>   slot_name  | inactive_since | conflicting | invalidation_reason
> -------------+----------------+-------------+---------------------
>  lsub29_slot |                | f           | inactive_timeout
> (1 row)
>
> I think in this case it should always reflect the value from the primary (so
> that one can understand why it is invalidated).

I'll come back to this as soon as we all agree on inactive_since
behavior for synced slots.

> T2 ===
>
> And it is set to a value during promotion:
>
> postgres=# select pg_promote();
>  pg_promote
> ------------
>  t
> (1 row)
>
> postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot';
>   slot_name  |        inactive_since        | conflicting | invalidation_reason
> -------------+------------------------------+-------------+---------------------
>  lsub29_slot | 2024-03-28 08:30:11.74505+00 | f           | inactive_timeout
> (1 row)
>
> I think when it is invalidated it should always reflect the value from the
> primary (so that one can understand why it is invalidated).

I'll come back to this as soon as we all agree on inactive_since
behavior for synced slots.

> T3 ===
>
> As far the slot invalidation on the primary:
>
> postgres=# SELECT * FROM pg_logical_slot_get_changes('lsub29_slot', NULL, NULL, 'include-xids', '0');
> ERROR:  cannot acquire invalidated replication slot "lsub29_slot"
>
> Can we make the message more consistent with what can be found in CreateDecodingContext()
> for example?

Hm, that makes sense because slot acquisition and release is something
internal to the server.

> T4 ===
>
> Also, it looks like querying pg_replication_slots() does not trigger an
> invalidation: I think it should if the slot is not invalidated yet (and matches
> the invalidation criteria).

There's a different opinion on this, check comment #3 from
https://www.postgresql.org/message-id/CAA4eK1LLj%2BeaMN-K8oeOjfG%2BUuzTY%3DL5PXbcMJURZbFm%2B_aJSA%40mail.gmail.com.

> Code review:
>
> CR1 ===
>
> +        Invalidate replication slots that are inactive for longer than this
> +        amount of time. If this value is specified without units, it is taken
>
> s/Invalidate/Invalidates/?

Done.

> Should we mention the relationship with inactive_since?

Done.

> CR2 ===
>
> + *
> + * If check_for_invalidation is true, the slot is checked for invalidation
> + * based on replication_slot_inactive_timeout GUC and an error is raised after making the slot ours.
>   */
>  void
> -ReplicationSlotAcquire(const char *name, bool nowait)
> +ReplicationSlotAcquire(const char *name, bool nowait,
> +                                          bool check_for_invalidation)
>
>
> s/check_for_invalidation/check_for_timeout_invalidation/?

Done.

> CR3 ===
>
> +       if (slot->inactive_since == 0 ||
> +               replication_slot_inactive_timeout == 0)
> +               return false;
>
> Better to test replication_slot_inactive_timeout first? (I mean there is no
> point of testing inactive_since if replication_slot_inactive_timeout == 0)
>
> CR4 ===
>
> +       if (slot->inactive_since > 0 &&
> +               replication_slot_inactive_timeout > 0)
> +       {
>
> Same.
>
> So, instead of CR3 === and CR4 ===, I wonder if it wouldn't be better to do
> something like:
>
> if (replication_slot_inactive_timeout == 0)
>         return false;
> else if (slot->inactive_since > 0)
> .
> else
>         return false;
>
> That would avoid checking replication_slot_inactive_timeout and inactive_since
> multiple times.

Done.

> CR5 ===
>
> +        * held to avoid race conditions -- for example the restart_lsn could move
> +        * forward, or the slot could be dropped.
>
> Does the restart_lsn example makes sense here?

No, it doesn't. Modified that.

> CR6 ===
>
> +static bool
> +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks)
> +{
>
> InvalidatePossiblyInactiveSlot() maybe?

I think we will lose the essence i.e. timeout from the suggested
function name, otherwise just the inactive doesn't give a clearer
meaning. I kept it that way unless anyone suggests otherwise.

> CR7 ===
>
> +       /* Make sure the invalidated state persists across server restart */
> +       slot->just_dirtied = true;
> +       slot->dirty = true;
> +       SpinLockRelease(&slot->mutex);
>
> Maybe we could create a new function say MarkGivenReplicationSlotDirty()
> with a slot as parameter, that ReplicationSlotMarkDirty could call too?

Done that.

> Then maybe we could set slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT in
> InvalidateSlotForInactiveTimeout()? (to avoid multiple SpinLockAcquire/SpinLockRelease).

Done that.

> CR8 ===
>
> +       if (persist_state)
> +       {
> +               char            path[MAXPGPATH];
> +
> +               sprintf(path, "pg_replslot/%s", NameStr(slot->data.name));
> +               SaveSlotToPath(slot, path, ERROR);
> +       }
>
> Maybe we could create a new function say GivenReplicationSlotSave()
> with a slot as parameter, that ReplicationSlotSave() could call too?

Done that.

> CR9 ===
>
> +       if (check_for_invalidation)
> +       {
> +               /* The slot is ours by now */
> +               Assert(s->active_pid == MyProcPid);
> +
> +               /*
> +                * Well, the slot is not yet ours really unless we check for the
> +                * invalidation below.
> +                */
> +               s->active_pid = 0;
> +               if (InvalidateReplicationSlotForInactiveTimeout(s, true, true))
> +               {
> +                       /*
> +                        * If the slot has been invalidated, recalculate the resource
> +                        * limits.
> +                        */
> +                       ReplicationSlotsComputeRequiredXmin(false);
> +                       ReplicationSlotsComputeRequiredLSN();
> +
> +                       /* Might need it for slot clean up on error, so restore it */
> +                       s->active_pid = MyProcPid;
> +                       ereport(ERROR,
> +                                       (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> +                                        errmsg("cannot acquire invalidated replication slot \"%s\"",
> +                                                       NameStr(MyReplicationSlot->data.name))));
> +               }
> +               s->active_pid = MyProcPid;
>
> Are we not missing some SpinLockAcquire/Release on the slot's mutex here? (the
> places where we set the active_pid).

Hm, yes. But, shall I acquire the mutex, set active_pid to 0 for a
moment just to satisfy Assert(slot->active_pid == 0); in
InvalidateReplicationSlotForInactiveTimeout and
InvalidateSlotForInactiveTimeout? I just removed the assertions
because being replication_slot_inactive_timeout > 0 and inactive_since
> 0 is enough for these functions to think and decide on inactive
timeout invalidation.

> CR10 ===
>
> @@ -1628,6 +1674,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
>                                         if (SlotIsLogical(s))
>                                                 invalidation_cause = cause;
>                                         break;
> +                               case RS_INVAL_INACTIVE_TIMEOUT:
> +                                       if (InvalidateReplicationSlotForInactiveTimeout(s, false, false))
> +                                               invalidation_cause = cause;
> +                                       break;
>
> InvalidatePossiblyObsoleteSlot() is not called with such a reason, better to use
> an Assert here and in the caller too?

Done.

> CR11 ===
>
> +++ b/src/test/recovery/t/050_invalidate_slots.pl
>
> why not using 019_replslot_limit.pl?

I understand that 019_replslot_limit covers wal_removed related
invalidations. But, I don't want to kludge it with a bunch of other
tests. The new tests anyway need a bunch of new nodes and a couple of
helper functions. Any future invalidation mechanisms can be added here
in this new file. Also, having a separate file quickly helps isolate
any test failures that BF animals might report in future. I don't
think a separate test file here hurts anyone unless there's a strong
reason against it.

Please see the attached v30 patch. 0002 is where all of the above
review comments have been addressed.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  189. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-01T03:17:59Z

    On Fri, Mar 29, 2024 at 9:39 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Commit message states: "why we can't just update inactive_since for
> synced slots on the standby with the value received from remote slot
> on the primary. This is consistent with any other slot parameter i.e.
> all of them are synced from the primary."
>
> The inactive_since is not consistent with other slot parameters which
> we copy. We don't perform anything related to those other parameters
> like say two_phase phase which can change that property. However, we
> do acquire the slot, advance the slot (as per recent discussion [1]),
> and release it. Since these operations can impact inactive_since, it
> seems to me that inactive_since is not the same as other parameters.
> It can have a different value than the primary. Why would anyone want
> to know the value of inactive_since from primary after the standby is
> promoted?

After thinking about it for a while now, it feels to me that the
synced slots (slots on the standby that are being synced from the
primary) can have their own inactive_sicne value. Fundamentally,
inactive_sicne is set to 0 when slot is acquired and set to current
time when slot is released, no matter who acquires and releases it -
be it walsenders for replication, or backends for slot advance, or
backends for slot sync using pg_sync_replication_slots, or backends
for other slot functions, or background sync worker. Remember the
earlier patch was updating inactive_since just for walsenders, but
then the suggestion was to update it unconditionally -
https://www.postgresql.org/message-id/CAJpy0uD64X%3D2ENmbHaRiWTKeQawr-rbGoy_GdhQQLVXzUSKTMg%40mail.gmail.com.
Whoever syncs the slot, *acutally* acquires the slot i.e. makes it
theirs, syncs it from the primary, and releases it. IMO, no
differentiation is to be made for synced slots.

There was a suggestion on using inactive_since of the synced slot on
the standby to know the inactivity of the slot on the primary. If one
wants to do that, they better look at/monitor the primary slot
info/logs/pg_replication_slot/whatever. I really don't see a point in
having two different meanings for a single property of a replication
slot - inactive_since for a regular slot tells since when this slot
has become inactive, and for a synced slot since when the
corresponding remote slot has become inactive. I think this will
confuse users for sure.

Also, if inactive_since is being changed on the primary so frequently,
and none of the other parameters are changing, if we copy
inactive_since to the synced slots, then standby will just be doing
*sync* work (mark the slots dirty and save to disk) for updating
inactive_since. I think this is unnecessary behaviour for sure.

Coming to a future patch for inactive timeout based slot invalidation,
we can either allow invalidation without any differentiation for
synced slots or restrict invalidation to avoid more sync work. For
instance, if inactive timeout is kept low on the standby, the sync
worker will be doing more work as it drops and recreates a slot
repeatedly if it keeps getting invalidated. Another thing is that the
standby takes independent invalidation decisions for synced slots.
AFAICS, invalidation due to wal_removal is the only sole reason (out
of all available invalidation reasons) for a synced slot to get
invalidated independently of the primary. Check
https://www.postgresql.org/message-id/CAA4eK1JXBwTaDRD_%3D8t6UB1fhRNjC1C%2BgH4YdDxj_9U6djLnXw%40mail.gmail.com
for the suggestion on we better not differentiaing invalidation
decisions for synced slots.

The assumption of letting synced slots have their own inactive_since
not only simplifies the code, but also looks less-confusing and more
meaningful to the user. The only code that we put in on top of the
committed code is to use InRecovery in place of
RecoveryInProgress() in RestoreSlotFromDisk() to fix the issue raised
by Shveta upthread.

> Now, the other concern is that calling GetCurrentTimestamp()
> could be costly when the values for the slot are not going to be
> updated but if that happens we can optimize such that before acquiring
> the slot we can have some minimal pre-checks to ensure whether we need
> to update the slot or not.
>
> [1] - https://www.postgresql.org/message-id/OS0PR01MB571615D35F486080616CA841943A2%40OS0PR01MB5716.jpnprd01.prod.outlook.com

A quick test with a function to measure the cost of
GetCurrentTimestamp [1] on my Ubuntu dev system (an AWS EC2 c5.4xlarge
instance), gives me [2]. It took 0.388 ms, 2.269 ms, 21.144 ms,
209.333 ms, 2091.174 ms, 20908.942 ms for 10K, 100K, 1million,
10million, 100million, 1billion times respectively. Costs might be
different on various systems with different OS, but it gives us a
rough idea.

If we are too much concerned about the cost of GetCurrentTimestamp(),
a possible approach is just don't set inactive_since for slots being
synced on the standby. Just let the first acquisition and release
after the promotion do that job. We can always call this out in the
docs saying "replication slots on the streaming standbys which are
being synced from the primary are not inactive in practice, so the
inactive_since is always NULL for them unless the standby is
promoted".

[1]
Datum
pg_get_current_timestamp(PG_FUNCTION_ARGS)
{
    int         loops = PG_GETARG_INT32(0);
    TimestampTz ctime;

    for (int i = 0; i < loops; i++)
        ctime = GetCurrentTimestamp();

    PG_RETURN_TIMESTAMPTZ(ctime);
}

[2]
postgres=# \timing
Timing is on.
postgres=# SELECT pg_get_current_timestamp(1000000000);
   pg_get_current_timestamp
-------------------------------
 2024-03-30 19:07:57.374797+00
(1 row)

Time: 20908.942 ms (00:20.909)
postgres=# SELECT pg_get_current_timestamp(100000000);
   pg_get_current_timestamp
-------------------------------
 2024-03-30 19:08:21.038064+00
(1 row)

Time: 2091.174 ms (00:02.091)
postgres=# SELECT pg_get_current_timestamp(10000000);
   pg_get_current_timestamp
-------------------------------
 2024-03-30 19:08:24.329949+00
(1 row)

Time: 209.333 ms
postgres=# SELECT pg_get_current_timestamp(1000000);
   pg_get_current_timestamp
-------------------------------
 2024-03-30 19:08:26.978016+00
(1 row)

Time: 21.144 ms
postgres=# SELECT pg_get_current_timestamp(100000);
   pg_get_current_timestamp
-------------------------------
 2024-03-30 19:08:29.142248+00
(1 row)

Time: 2.269 ms
postgres=# SELECT pg_get_current_timestamp(10000);
   pg_get_current_timestamp
------------------------------
 2024-03-30 19:08:31.34621+00
(1 row)

Time: 0.388 ms

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  190. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-01T03:34:43Z

    On Fri, Mar 29, 2024 at 6:17 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Fri, Mar 29, 2024 at 03:03:01PM +0530, Amit Kapila wrote:
> > On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > >
> > > On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote:
> > > >
> > > > Commit message states: "why we can't just update inactive_since for
> > > > synced slots on the standby with the value received from remote slot
> > > > on the primary. This is consistent with any other slot parameter i.e.
> > > > all of them are synced from the primary."
> > > >
> > > > The inactive_since is not consistent with other slot parameters which
> > > > we copy. We don't perform anything related to those other parameters
> > > > like say two_phase phase which can change that property. However, we
> > > > do acquire the slot, advance the slot (as per recent discussion [1]),
> > > > and release it. Since these operations can impact inactive_since, it
> > > > seems to me that inactive_since is not the same as other parameters.
> > > > It can have a different value than the primary. Why would anyone want
> > > > to know the value of inactive_since from primary after the standby is
> > > > promoted?
> > >
> > > I think it can be useful "before" it is promoted and in case the primary is down.
> > >
> >
> > It is not clear to me what is user going to do by checking the
> > inactivity time for slots when the corresponding server is down.
>
> Say a failover needs to be done, then it could be useful to know for which
> slots the activity needs to be resumed (thinking about external logical decoding
> plugin, not about pub/sub here). If one see an inactive slot (since long "enough")
> then he can start to reasonate about what to do with it.
>
> > I thought the idea was to check such slots and see if they need to be
> > dropped or enabled again to avoid excessive disk usage, etc.
>
> Yeah that's the case but it does not mean inactive_since can't be useful in other
> ways.
>
> Also, say the slot has been invalidated on the primary (due to inactivity timeout),
> primary is down and there is a failover. By keeping the inactive_since from
> the primary, one could know when the inactivity that lead to the timeout started.
>

So, this means at promotion, we won't set the current_time for
inactive_since which is not what the currently proposed patch is
doing. Moreover, doing the invalidation on promoted standby based on
inactive_since of the primary node sounds debatable because the
inactive_timeout could be different on the new node (promoted
standby).

> Again, more concerned about external logical decoding plugin than pub/sub here.
>
> > > I agree that tracking the activity time of a synced slot can be useful, why
> > > not creating a dedicated field for that purpose (and keep inactive_since a
> > > perfect "copy" of the primary)?
> > >
> >
> > We can have a separate field for this but not sure if it is worth it.
>
> OTOH I'm not sure that erasing this information from the primary is useful. I
> think that 2 fields would be the best option and would be less subject of
> misinterpretation.
>
> > > > Now, the other concern is that calling GetCurrentTimestamp()
> > > > could be costly when the values for the slot are not going to be
> > > > updated but if that happens we can optimize such that before acquiring
> > > > the slot we can have some minimal pre-checks to ensure whether we need
> > > > to update the slot or not.
> > >
> > > Right, but for a very active slot it is likely that we call GetCurrentTimestamp()
> > > during almost each sync cycle.
> > >
> >
> > True, but if we have to save a slot to disk each time to persist the
> > changes (for an active slot) then probably GetCurrentTimestamp()
> > shouldn't be costly enough to matter.
>
> Right, persisting the changes to disk would be even more costly.
>

The point I was making is that currently after copying the
remote_node's values, we always persist the slots to disk, so the cost
of current_time shouldn't be much. Now, if the values won't change
then probably there is some cost but in most cases (active slots), the
values will always change. Also, if all the slots are inactive then we
will slow down the speed of sync. We also need to consider if we want
to copy the value of inactive_since from the primary and if that is
the only value changed then shall we persist the slot or not?

-- 
With Regards,
Amit Kapila.

  191. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-01T06:51:08Z

    Hi,

On Mon, Apr 01, 2024 at 09:04:43AM +0530, Amit Kapila wrote:
> On Fri, Mar 29, 2024 at 6:17 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > On Fri, Mar 29, 2024 at 03:03:01PM +0530, Amit Kapila wrote:
> > > On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot
> > > <bertranddrouvot.pg@gmail.com> wrote:
> > > >
> > > > On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote:
> > > > >
> > > > > Commit message states: "why we can't just update inactive_since for
> > > > > synced slots on the standby with the value received from remote slot
> > > > > on the primary. This is consistent with any other slot parameter i.e.
> > > > > all of them are synced from the primary."
> > > > >
> > > > > The inactive_since is not consistent with other slot parameters which
> > > > > we copy. We don't perform anything related to those other parameters
> > > > > like say two_phase phase which can change that property. However, we
> > > > > do acquire the slot, advance the slot (as per recent discussion [1]),
> > > > > and release it. Since these operations can impact inactive_since, it
> > > > > seems to me that inactive_since is not the same as other parameters.
> > > > > It can have a different value than the primary. Why would anyone want
> > > > > to know the value of inactive_since from primary after the standby is
> > > > > promoted?
> > > >
> > > > I think it can be useful "before" it is promoted and in case the primary is down.
> > > >
> > >
> > > It is not clear to me what is user going to do by checking the
> > > inactivity time for slots when the corresponding server is down.
> >
> > Say a failover needs to be done, then it could be useful to know for which
> > slots the activity needs to be resumed (thinking about external logical decoding
> > plugin, not about pub/sub here). If one see an inactive slot (since long "enough")
> > then he can start to reasonate about what to do with it.
> >
> > > I thought the idea was to check such slots and see if they need to be
> > > dropped or enabled again to avoid excessive disk usage, etc.
> >
> > Yeah that's the case but it does not mean inactive_since can't be useful in other
> > ways.
> >
> > Also, say the slot has been invalidated on the primary (due to inactivity timeout),
> > primary is down and there is a failover. By keeping the inactive_since from
> > the primary, one could know when the inactivity that lead to the timeout started.
> >
> 
> So, this means at promotion, we won't set the current_time for
> inactive_since which is not what the currently proposed patch is
> doing.

Yeah, that's why I made the comment T2 in [1].

> Moreover, doing the invalidation on promoted standby based on
> inactive_since of the primary node sounds debatable because the
> inactive_timeout could be different on the new node (promoted
> standby).

I think that if the slot is not invalidated before the promotion then we should
erase the value from the primary and use the promotion time.

> > Again, more concerned about external logical decoding plugin than pub/sub here.
> >
> > > > I agree that tracking the activity time of a synced slot can be useful, why
> > > > not creating a dedicated field for that purpose (and keep inactive_since a
> > > > perfect "copy" of the primary)?
> > > >
> > >
> > > We can have a separate field for this but not sure if it is worth it.
> >
> > OTOH I'm not sure that erasing this information from the primary is useful. I
> > think that 2 fields would be the best option and would be less subject of
> > misinterpretation.
> >
> > > > > Now, the other concern is that calling GetCurrentTimestamp()
> > > > > could be costly when the values for the slot are not going to be
> > > > > updated but if that happens we can optimize such that before acquiring
> > > > > the slot we can have some minimal pre-checks to ensure whether we need
> > > > > to update the slot or not.
> > > >
> > > > Right, but for a very active slot it is likely that we call GetCurrentTimestamp()
> > > > during almost each sync cycle.
> > > >
> > >
> > > True, but if we have to save a slot to disk each time to persist the
> > > changes (for an active slot) then probably GetCurrentTimestamp()
> > > shouldn't be costly enough to matter.
> >
> > Right, persisting the changes to disk would be even more costly.
> >
> 
> The point I was making is that currently after copying the
> remote_node's values, we always persist the slots to disk, so the cost
> of current_time shouldn't be much.

Oh right, I missed this (was focusing only on inactive_since that we don't persist
to disk IIRC).

BTW, If we are going this way, maybe we could accept a bit less accuracy
and use GetCurrentTransactionStopTimestamp() instead?

> Now, if the values won't change
> then probably there is some cost but in most cases (active slots), the
> values will always change.

Right.

> Also, if all the slots are inactive then we
> will slow down the speed of sync.

Yes.

> We also need to consider if we want
> to copy the value of inactive_since from the primary and if that is
> the only value changed then shall we persist the slot or not?

Good point, then I don't think we should as inactive_since is not persisted on disk.

[1]: https://www.postgresql.org/message-id/ZgU70MjdOfO6l0O0%40ip-10-97-1-34.eu-west-3.compute.internal

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  192. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-01T07:18:55Z

    Hi,

On Mon, Apr 01, 2024 at 08:47:59AM +0530, Bharath Rupireddy wrote:
> On Fri, Mar 29, 2024 at 9:39 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Commit message states: "why we can't just update inactive_since for
> > synced slots on the standby with the value received from remote slot
> > on the primary. This is consistent with any other slot parameter i.e.
> > all of them are synced from the primary."
> >
> > The inactive_since is not consistent with other slot parameters which
> > we copy. We don't perform anything related to those other parameters
> > like say two_phase phase which can change that property. However, we
> > do acquire the slot, advance the slot (as per recent discussion [1]),
> > and release it. Since these operations can impact inactive_since, it
> > seems to me that inactive_since is not the same as other parameters.
> > It can have a different value than the primary. Why would anyone want
> > to know the value of inactive_since from primary after the standby is
> > promoted?
> 
> After thinking about it for a while now, it feels to me that the
> synced slots (slots on the standby that are being synced from the
> primary) can have their own inactive_sicne value. Fundamentally,
> inactive_sicne is set to 0 when slot is acquired and set to current
> time when slot is released, no matter who acquires and releases it -
> be it walsenders for replication, or backends for slot advance, or
> backends for slot sync using pg_sync_replication_slots, or backends
> for other slot functions, or background sync worker. Remember the
> earlier patch was updating inactive_since just for walsenders, but
> then the suggestion was to update it unconditionally -
> https://www.postgresql.org/message-id/CAJpy0uD64X%3D2ENmbHaRiWTKeQawr-rbGoy_GdhQQLVXzUSKTMg%40mail.gmail.com.
> Whoever syncs the slot, *acutally* acquires the slot i.e. makes it
> theirs, syncs it from the primary, and releases it. IMO, no
> differentiation is to be made for synced slots.
> 
> There was a suggestion on using inactive_since of the synced slot on
> the standby to know the inactivity of the slot on the primary. If one
> wants to do that, they better look at/monitor the primary slot
> info/logs/pg_replication_slot/whatever.

Yeah but the use case was in case the primary is down for whatever reason.

> I really don't see a point in
> having two different meanings for a single property of a replication
> slot - inactive_since for a regular slot tells since when this slot
> has become inactive, and for a synced slot since when the
> corresponding remote slot has become inactive. I think this will
> confuse users for sure.

I'm not sure as we are speaking about "synced" slots. I can also see some confusion
if this value is not "synced".

> Also, if inactive_since is being changed on the primary so frequently,
> and none of the other parameters are changing, if we copy
> inactive_since to the synced slots, then standby will just be doing
> *sync* work (mark the slots dirty and save to disk) for updating
> inactive_since. I think this is unnecessary behaviour for sure.

Right, I think we should avoid the save slot to disk in that case (question raised
by Amit in [1]).

> Coming to a future patch for inactive timeout based slot invalidation,
> we can either allow invalidation without any differentiation for
> synced slots or restrict invalidation to avoid more sync work. For
> instance, if inactive timeout is kept low on the standby, the sync
> worker will be doing more work as it drops and recreates a slot
> repeatedly if it keeps getting invalidated. Another thing is that the
> standby takes independent invalidation decisions for synced slots.
> AFAICS, invalidation due to wal_removal is the only sole reason (out
> of all available invalidation reasons) for a synced slot to get
> invalidated independently of the primary. Check
> https://www.postgresql.org/message-id/CAA4eK1JXBwTaDRD_%3D8t6UB1fhRNjC1C%2BgH4YdDxj_9U6djLnXw%40mail.gmail.com
> for the suggestion on we better not differentiaing invalidation
> decisions for synced slots.

Yeah, I think the invalidation decision on the standby is highly linked to
what inactive_since on the standby is: synced from primary or not.

> The assumption of letting synced slots have their own inactive_since
> not only simplifies the code, but also looks less-confusing and more
> meaningful to the user.

I'm not sure at all. But if the majority of us thinks it's the case then let's
go that way.

> > Now, the other concern is that calling GetCurrentTimestamp()
> > could be costly when the values for the slot are not going to be
> > updated but if that happens we can optimize such that before acquiring
> > the slot we can have some minimal pre-checks to ensure whether we need
> > to update the slot or not.

Also maybe we could accept a bit less accuracy and use
GetCurrentTransactionStopTimestamp() instead?

> If we are too much concerned about the cost of GetCurrentTimestamp(),
> a possible approach is just don't set inactive_since for slots being
> synced on the standby.
> Just let the first acquisition and release
> after the promotion do that job. We can always call this out in the
> docs saying "replication slots on the streaming standbys which are
> being synced from the primary are not inactive in practice, so the
> inactive_since is always NULL for them unless the standby is
> promoted".

I think that was the initial behavior that lead to Robert's remark (see [2]):

"
And I'm suspicious that having an exception for slots being synced is
a bad idea. That makes too much of a judgement about how the user will
use this field. It's usually better to just expose the data, and if
the user needs helps to make sense of that data, then give them that
help separately.
"

[1]: https://www.postgresql.org/message-id/CAA4eK1JtKieWMivbswYg5FVVB5FugCftLvQKVsxh%3Dm_8nk04vw%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/CA%2BTgmob_Ta-t2ty8QrKHBGnNLrf4ZYcwhGHGFsuUoFrAEDw4sA%40mail.gmail.com

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  193. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-01T09:59:51Z

    Hi,

On Sun, Mar 31, 2024 at 10:25:46AM +0530, Bharath Rupireddy wrote:
> On Thu, Mar 28, 2024 at 3:13 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> > I think in this case it should always reflect the value from the primary (so
> > that one can understand why it is invalidated).
> 
> I'll come back to this as soon as we all agree on inactive_since
> behavior for synced slots.

Makes sense. Also if the majority of us thinks it's not needed for inactive_since
to be an exact copy of the primary, then let's go that way.

> > I think when it is invalidated it should always reflect the value from the
> > primary (so that one can understand why it is invalidated).
> 
> I'll come back to this as soon as we all agree on inactive_since
> behavior for synced slots.

Yeah.

> > T4 ===
> >
> > Also, it looks like querying pg_replication_slots() does not trigger an
> > invalidation: I think it should if the slot is not invalidated yet (and matches
> > the invalidation criteria).
> 
> There's a different opinion on this, check comment #3 from
> https://www.postgresql.org/message-id/CAA4eK1LLj%2BeaMN-K8oeOjfG%2BUuzTY%3DL5PXbcMJURZbFm%2B_aJSA%40mail.gmail.com.

Oh right, I can see Amit's point too. Let's put pg_replication_slots() out of
the game then.

> > CR6 ===
> >
> > +static bool
> > +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks)
> > +{
> >
> > InvalidatePossiblyInactiveSlot() maybe?
> 
> I think we will lose the essence i.e. timeout from the suggested
> function name, otherwise just the inactive doesn't give a clearer
> meaning. I kept it that way unless anyone suggests otherwise.

Right. OTOH I think that "Possibly" adds some nuance (like InvalidatePossiblyObsoleteSlot()
is already doing).

> Please see the attached v30 patch. 0002 is where all of the above
> review comments have been addressed.

Thanks! FYI, I did not look at the content yet, just replied to the above
comments.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  194. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-04-02T03:07:54Z

    On Mon, Apr 1, 2024 at 12:18 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Fri, Mar 29, 2024 at 9:39 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Commit message states: "why we can't just update inactive_since for
> > synced slots on the standby with the value received from remote slot
> > on the primary. This is consistent with any other slot parameter i.e.
> > all of them are synced from the primary."
> >
> > The inactive_since is not consistent with other slot parameters which
> > we copy. We don't perform anything related to those other parameters
> > like say two_phase phase which can change that property. However, we
> > do acquire the slot, advance the slot (as per recent discussion [1]),
> > and release it. Since these operations can impact inactive_since, it
> > seems to me that inactive_since is not the same as other parameters.
> > It can have a different value than the primary. Why would anyone want
> > to know the value of inactive_since from primary after the standby is
> > promoted?
>
> After thinking about it for a while now, it feels to me that the
> synced slots (slots on the standby that are being synced from the
> primary) can have their own inactive_sicne value. Fundamentally,
> inactive_sicne is set to 0 when slot is acquired and set to current
> time when slot is released, no matter who acquires and releases it -
> be it walsenders for replication, or backends for slot advance, or
> backends for slot sync using pg_sync_replication_slots, or backends
> for other slot functions, or background sync worker. Remember the
> earlier patch was updating inactive_since just for walsenders, but
> then the suggestion was to update it unconditionally -
> https://www.postgresql.org/message-id/CAJpy0uD64X%3D2ENmbHaRiWTKeQawr-rbGoy_GdhQQLVXzUSKTMg%40mail.gmail.com.
> Whoever syncs the slot, *acutally* acquires the slot i.e. makes it
> theirs, syncs it from the primary, and releases it. IMO, no
> differentiation is to be made for synced slots.

FWIW, coming to this thread late, I think that the inactive_since
should not be synchronized from the primary. The wall clocks are
different on the primary and the standby so having the primary's
timestamp on the standby can confuse users, especially when there is a
big clock drift. Also, as Amit mentioned, inactive_since seems not to
be consistent with other parameters we copy. The
replication_slot_inactive_timeout feature should work on the standby
independent from the primary, like other slot invalidation mechanisms,
and it should be based on its own local clock.

> Coming to a future patch for inactive timeout based slot invalidation,
> we can either allow invalidation without any differentiation for
> synced slots or restrict invalidation to avoid more sync work. For
> instance, if inactive timeout is kept low on the standby, the sync
> worker will be doing more work as it drops and recreates a slot
> repeatedly if it keeps getting invalidated. Another thing is that the
> standby takes independent invalidation decisions for synced slots.
> AFAICS, invalidation due to wal_removal is the only sole reason (out
> of all available invalidation reasons) for a synced slot to get
> invalidated independently of the primary. Check
> https://www.postgresql.org/message-id/CAA4eK1JXBwTaDRD_%3D8t6UB1fhRNjC1C%2BgH4YdDxj_9U6djLnXw%40mail.gmail.com
> for the suggestion on we better not differentiaing invalidation
> decisions for synced slots.
>
> The assumption of letting synced slots have their own inactive_since
> not only simplifies the code, but also looks less-confusing and more
> meaningful to the user. The only code that we put in on top of the
> committed code is to use InRecovery in place of
> RecoveryInProgress() in RestoreSlotFromDisk() to fix the issue raised
> by Shveta upthread.

If we want to invalidate the synced slots due to the timeout, I think
we need to define what is "inactive" for synced slots.

Suppose that the slotsync worker updates the local (synced) slot's
inactive_since whenever releasing the slot, irrespective of the actual
LSNs (or other slot parameters) having been updated. I think that this
idea cannot handle a slot that is not acquired on the primary. In this
case, the remote slot is inactive but the local slot is regarded as
active.  WAL files are piled up on the standby (and on the primary) as
the slot's LSNs don't move forward. I think we want to regard such a
slot as "inactive" also on the standby and invalidate it because of
the timeout.

>
> > Now, the other concern is that calling GetCurrentTimestamp()
> > could be costly when the values for the slot are not going to be
> > updated but if that happens we can optimize such that before acquiring
> > the slot we can have some minimal pre-checks to ensure whether we need
> > to update the slot or not.

If we use such pre-checks, another problem might happen; it cannot
handle a case where the slot is acquired on the primary but its LSNs
don't move forward. Imagine a logical replication conflict happened on
the subscriber, and the logical replication enters the retry loop. In
this case, the remote slot's inactive_since gets updated for every
retry, but it looks inactive from the standby since the slot LSNs
don't change. Therefore, only the local slot could be invalidated due
to the timeout but probably we don't want to regard such a slot as
"inactive".

Another idea I came up with is that the slotsync worker updates the
local slot's inactive_since to the local timestamp only when the
remote slot might have got inactive. If the remote slot is acquired by
someone, the local slot's inactive_since is also NULL. If the remote
slot gets inactive, the slotsync worker sets the local timestamp to
the local slot's inactive_since. Since the remote slot could be
acquired and released before the slotsync worker gets the remote slot
data again, if the remote slot's inactive_since > the local slot's
inactive_since, the slotsync worker updates the local one. IOW, we
detect whether the remote slot was acquired and released since the
last synchronization, by checking the remote slot's inactive_since.
This idea seems to handle these cases I mentioned unless I'm missing
something, but it requires for the slotsync worker to update
inactive_since in a different way than other parameters.

Or a simple solution is that the slotsync worker updates
inactive_since as it does for non-synced slots, and disables
timeout-based slot invalidation for synced slots.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  195. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-02T06:28:40Z

    Hi,

On Tue, Apr 02, 2024 at 12:07:54PM +0900, Masahiko Sawada wrote:
> On Mon, Apr 1, 2024 at 12:18 PM Bharath Rupireddy
> 
> FWIW, coming to this thread late, I think that the inactive_since
> should not be synchronized from the primary. The wall clocks are
> different on the primary and the standby so having the primary's
> timestamp on the standby can confuse users, especially when there is a
> big clock drift. Also, as Amit mentioned, inactive_since seems not to
> be consistent with other parameters we copy. The
> replication_slot_inactive_timeout feature should work on the standby
> independent from the primary, like other slot invalidation mechanisms,
> and it should be based on its own local clock.

Thanks for sharing your thoughts! So, it looks like that most of us agree to not
sync inactive_since from the primary, I'm fine with that.

> If we want to invalidate the synced slots due to the timeout, I think
> we need to define what is "inactive" for synced slots.
> 
> Suppose that the slotsync worker updates the local (synced) slot's
> inactive_since whenever releasing the slot, irrespective of the actual
> LSNs (or other slot parameters) having been updated. I think that this
> idea cannot handle a slot that is not acquired on the primary. In this
> case, the remote slot is inactive but the local slot is regarded as
> active.  WAL files are piled up on the standby (and on the primary) as
> the slot's LSNs don't move forward. I think we want to regard such a
> slot as "inactive" also on the standby and invalidate it because of
> the timeout.

I think that makes sense to somehow link inactive_since on the standby to 
the actual LSNs (or other slot parameters) being updated or not.

> > > Now, the other concern is that calling GetCurrentTimestamp()
> > > could be costly when the values for the slot are not going to be
> > > updated but if that happens we can optimize such that before acquiring
> > > the slot we can have some minimal pre-checks to ensure whether we need
> > > to update the slot or not.
> 
> If we use such pre-checks, another problem might happen; it cannot
> handle a case where the slot is acquired on the primary but its LSNs
> don't move forward. Imagine a logical replication conflict happened on
> the subscriber, and the logical replication enters the retry loop. In
> this case, the remote slot's inactive_since gets updated for every
> retry, but it looks inactive from the standby since the slot LSNs
> don't change. Therefore, only the local slot could be invalidated due
> to the timeout but probably we don't want to regard such a slot as
> "inactive".
> 
> Another idea I came up with is that the slotsync worker updates the
> local slot's inactive_since to the local timestamp only when the
> remote slot might have got inactive. If the remote slot is acquired by
> someone, the local slot's inactive_since is also NULL. If the remote
> slot gets inactive, the slotsync worker sets the local timestamp to
> the local slot's inactive_since. Since the remote slot could be
> acquired and released before the slotsync worker gets the remote slot
> data again, if the remote slot's inactive_since > the local slot's
> inactive_since, the slotsync worker updates the local one.

Then I think we would need to be careful about time zone comparison.

> IOW, we
> detect whether the remote slot was acquired and released since the
> last synchronization, by checking the remote slot's inactive_since.
> This idea seems to handle these cases I mentioned unless I'm missing
> something, but it requires for the slotsync worker to update
> inactive_since in a different way than other parameters.
> 
> Or a simple solution is that the slotsync worker updates
> inactive_since as it does for non-synced slots, and disables
> timeout-based slot invalidation for synced slots.

Yeah, I think the main question to help us decide is: do we want to invalidate
"inactive" synced slots locally (in addition to synchronizing the invalidation
from the primary)? 

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  196. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-02T07:11:35Z

    On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > Or a simple solution is that the slotsync worker updates
> > inactive_since as it does for non-synced slots, and disables
> > timeout-based slot invalidation for synced slots.
>
> Yeah, I think the main question to help us decide is: do we want to invalidate
> "inactive" synced slots locally (in addition to synchronizing the invalidation
> from the primary)?

I think this approach looks way simpler than the other one. The other
approach of linking inactive_since on the standby for synced slots to
the actual LSNs (or other slot parameters) being updated or not looks
more complicated, and might not go well with the end user.  However,
we need to be able to say why we don't invalidate synced slots due to
inactive timeout unlike the wal_removed invalidation that can happen
right now on the standby for synced slots. This leads us to define
actually what a slot being active means. Is syncing the data from the
remote slot considered as the slot being active?

On the other hand, it may not sound great if we don't invalidate
synced slots due to inactive timeout even though they hold resources
such as WAL and XIDs.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  197. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-02T08:33:40Z

    Hi,

On Tue, Apr 02, 2024 at 12:41:35PM +0530, Bharath Rupireddy wrote:
> On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > > Or a simple solution is that the slotsync worker updates
> > > inactive_since as it does for non-synced slots, and disables
> > > timeout-based slot invalidation for synced slots.
> >
> > Yeah, I think the main question to help us decide is: do we want to invalidate
> > "inactive" synced slots locally (in addition to synchronizing the invalidation
> > from the primary)?
> 
> I think this approach looks way simpler than the other one. The other
> approach of linking inactive_since on the standby for synced slots to
> the actual LSNs (or other slot parameters) being updated or not looks
> more complicated, and might not go well with the end user.  However,
> we need to be able to say why we don't invalidate synced slots due to
> inactive timeout unlike the wal_removed invalidation that can happen
> right now on the standby for synced slots. This leads us to define
> actually what a slot being active means. Is syncing the data from the
> remote slot considered as the slot being active?
> 
> On the other hand, it may not sound great if we don't invalidate
> synced slots due to inactive timeout even though they hold resources
> such as WAL and XIDs.

Right and the "only" benefit then would be to give an idea as to when the last
sync did occur on the local slot.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  198. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-04-03T03:08:10Z

    On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Tue, Apr 02, 2024 at 12:07:54PM +0900, Masahiko Sawada wrote:
> > On Mon, Apr 1, 2024 at 12:18 PM Bharath Rupireddy
> >
> > FWIW, coming to this thread late, I think that the inactive_since
> > should not be synchronized from the primary. The wall clocks are
> > different on the primary and the standby so having the primary's
> > timestamp on the standby can confuse users, especially when there is a
> > big clock drift. Also, as Amit mentioned, inactive_since seems not to
> > be consistent with other parameters we copy. The
> > replication_slot_inactive_timeout feature should work on the standby
> > independent from the primary, like other slot invalidation mechanisms,
> > and it should be based on its own local clock.
>
> Thanks for sharing your thoughts! So, it looks like that most of us agree to not
> sync inactive_since from the primary, I'm fine with that.

+1 on not syncing slots from primary.

> > If we want to invalidate the synced slots due to the timeout, I think
> > we need to define what is "inactive" for synced slots.
> >
> > Suppose that the slotsync worker updates the local (synced) slot's
> > inactive_since whenever releasing the slot, irrespective of the actual
> > LSNs (or other slot parameters) having been updated. I think that this
> > idea cannot handle a slot that is not acquired on the primary. In this
> > case, the remote slot is inactive but the local slot is regarded as
> > active.  WAL files are piled up on the standby (and on the primary) as
> > the slot's LSNs don't move forward. I think we want to regard such a
> > slot as "inactive" also on the standby and invalidate it because of
> > the timeout.
>
> I think that makes sense to somehow link inactive_since on the standby to
> the actual LSNs (or other slot parameters) being updated or not.
>
> > > > Now, the other concern is that calling GetCurrentTimestamp()
> > > > could be costly when the values for the slot are not going to be
> > > > updated but if that happens we can optimize such that before acquiring
> > > > the slot we can have some minimal pre-checks to ensure whether we need
> > > > to update the slot or not.
> >
> > If we use such pre-checks, another problem might happen; it cannot
> > handle a case where the slot is acquired on the primary but its LSNs
> > don't move forward. Imagine a logical replication conflict happened on
> > the subscriber, and the logical replication enters the retry loop. In
> > this case, the remote slot's inactive_since gets updated for every
> > retry, but it looks inactive from the standby since the slot LSNs
> > don't change. Therefore, only the local slot could be invalidated due
> > to the timeout but probably we don't want to regard such a slot as
> > "inactive".
> >
> > Another idea I came up with is that the slotsync worker updates the
> > local slot's inactive_since to the local timestamp only when the
> > remote slot might have got inactive. If the remote slot is acquired by
> > someone, the local slot's inactive_since is also NULL. If the remote
> > slot gets inactive, the slotsync worker sets the local timestamp to
> > the local slot's inactive_since. Since the remote slot could be
> > acquired and released before the slotsync worker gets the remote slot
> > data again, if the remote slot's inactive_since > the local slot's
> > inactive_since, the slotsync worker updates the local one.
>
> Then I think we would need to be careful about time zone comparison.

Yes. Also we need to consider the case when a user is relying on
pg_sync_replication_slots() and has not enabled slot-sync worker. In
such a case if synced slot's inactive_since is derived from inactivity
of remote-slot, it might not be that frequently updated (based on when
the user actually runs the SQL function) and thus may be misleading.
OTOH, if inactivty_since of synced slots represents its own
inactivity, then it will give correct info even for the case when the
SQL function is run after a long time and slot-sync worker is
disabled.

> > IOW, we
> > detect whether the remote slot was acquired and released since the
> > last synchronization, by checking the remote slot's inactive_since.
> > This idea seems to handle these cases I mentioned unless I'm missing
> > something, but it requires for the slotsync worker to update
> > inactive_since in a different way than other parameters.
> >
> > Or a simple solution is that the slotsync worker updates
> > inactive_since as it does for non-synced slots, and disables
> > timeout-based slot invalidation for synced slots.

I like this idea better, it takes care of such a case too when the
user is relying on sync-function rather than worker and does not want
to get the slots invalidated in between 2 sync function calls.

> Yeah, I think the main question to help us decide is: do we want to invalidate
> "inactive" synced slots locally (in addition to synchronizing the invalidation
> from the primary)?


thanks
Shveta

  199. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-03T05:47:41Z

    On Wed, Apr 3, 2024 at 8:38 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> > > Or a simple solution is that the slotsync worker updates
> > > inactive_since as it does for non-synced slots, and disables
> > > timeout-based slot invalidation for synced slots.
>
> I like this idea better, it takes care of such a case too when the
> user is relying on sync-function rather than worker and does not want
> to get the slots invalidated in between 2 sync function calls.

Please find the attached v31 patches implementing the above idea:

- synced slots get their on inactive_since just like any other slot
- synced slots don't get invalidated due to inactive timeout because
such slots not considered active at all as they don't perform logical
decoding (of course, they will perform in fast_forward mode to fix the
other data loss issue, but they don't generate changes for them to be
called as *active* slots)
- synced slots inactive_since is set to current timestamp after the
standby gets promoted to help inactive_since interpret correctly just
like any other slot.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  200. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-03T06:50:19Z

    Hi,

On Wed, Apr 03, 2024 at 11:17:41AM +0530, Bharath Rupireddy wrote:
> On Wed, Apr 3, 2024 at 8:38 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > > Or a simple solution is that the slotsync worker updates
> > > > inactive_since as it does for non-synced slots, and disables
> > > > timeout-based slot invalidation for synced slots.
> >
> > I like this idea better, it takes care of such a case too when the
> > user is relying on sync-function rather than worker and does not want
> > to get the slots invalidated in between 2 sync function calls.
> 
> Please find the attached v31 patches implementing the above idea:

Thanks!

Some comments related to v31-0001:

=== testing the behavior

T1 ===

> - synced slots get their on inactive_since just like any other slot

It behaves as described.

T2 ===

> - synced slots inactive_since is set to current timestamp after the
> standby gets promoted to help inactive_since interpret correctly just
> like any other slot.
 
It behaves as described.

CR1 ===

+        <structfield>inactive_since</structfield> value will get updated
+        after every synchronization

indicates the last synchronization time? (I think that after every synchronization
could lead to confusion).

CR2 ===

+                       /*
+                        * Set the time since the slot has become inactive after shutting
+                        * down slot sync machinery. This helps correctly interpret the
+                        * time if the standby gets promoted without a restart.
+                        */

It looks to me that this comment is not at the right place because there is
nothing after the comment that indicates that we shutdown the "slot sync machinery".

Maybe a better place is before the function definition and mention that this is
currently called when we shutdown the "slot sync machinery"?

CR3 ===

+                        * We get the current time beforehand and only once to avoid
+                        * system calls overhead while holding the lock.

s/avoid system calls overhead while holding the lock/avoid system calls while holding the spinlock/?

CR4 ===

+        * Set the time since the slot has become inactive. We get the current
+        * time beforehand to avoid system call overhead while holding the lock

Same.

CR5 ===

+       # Check that the captured time is sane
+       if (defined $reference_time)
+       {

s/Check that the captured time is sane/Check that the inactive_since is sane/?

Sorry if some of those comments could have been done while I did review v29-0001.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  201. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-03T08:17:05Z

    Hi,

On Wed, Apr 03, 2024 at 11:17:41AM +0530, Bharath Rupireddy wrote:
> On Wed, Apr 3, 2024 at 8:38 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > > Or a simple solution is that the slotsync worker updates
> > > > inactive_since as it does for non-synced slots, and disables
> > > > timeout-based slot invalidation for synced slots.
> >
> > I like this idea better, it takes care of such a case too when the
> > user is relying on sync-function rather than worker and does not want
> > to get the slots invalidated in between 2 sync function calls.
> 
> Please find the attached v31 patches implementing the above idea:

Thanks!

Some comments regarding v31-0002:

=== testing the behavior

T1 ===

> - synced slots don't get invalidated due to inactive timeout because
> such slots not considered active at all as they don't perform logical
> decoding (of course, they will perform in fast_forward mode to fix the
> other data loss issue, but they don't generate changes for them to be
> called as *active* slots)

It behaves as described. OTOH non synced logical slots on the standby and
physical slots on the standby are invalidated which is what is expected.

T2 ===

In case the slot is invalidated on the primary,

primary:

postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
 slot_name |        inactive_since         | invalidation_reason
-----------+-------------------------------+---------------------
 s1        | 2024-04-03 06:56:28.075637+00 | inactive_timeout

then on the standby we get:

standby:

postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
 slot_name |        inactive_since        | invalidation_reason
-----------+------------------------------+---------------------
 s1        | 2024-04-03 07:06:43.37486+00 | inactive_timeout

shouldn't the slot be dropped/recreated instead of updating inactive_since?

=== code

CR1 ===

+        Invalidates replication slots that are inactive for longer the
+        specified amount of time

s/for longer the/for longer that/?

CR2 ===

+        <literal>true</literal>) as such synced slots don't actually perform
+        logical decoding.

We're switching in fast forward logical due to [1], so I'm not sure that's 100%
accurate here. I'm not sure we need to specify a reason.

CR3 ===

+ errdetail("This slot has been invalidated because it was inactive for more than the time specified by replication_slot_inactive_timeout parameter.")));

I think we can remove "parameter" (see for example the error message in
validate_remote_info()) and reduce it a bit, something like?

"This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout"?

CR4 ===

+ appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by replication_slot_inactive_timeout parameter."));

Same.

CR5 ===

+       /*
+        * This function isn't expected to be called for inactive timeout based
+        * invalidation. A separate function InvalidateInactiveReplicationSlot is
+        * to be used for that.

Do you think it's worth to explain why?

CR6 ===

+       if (replication_slot_inactive_timeout == 0)
+               return false;
+       else if (slot->inactive_since > 0)

"else" is not needed here.

CR7 ===

+               SpinLockAcquire(&slot->mutex);
+
+               /*
+                * Check if the slot needs to be invalidated due to
+                * replication_slot_inactive_timeout GUC. We do this with the spinlock
+                * held to avoid race conditions -- for example the inactive_since
+                * could change, or the slot could be dropped.
+                */
+               now = GetCurrentTimestamp();

We should not call GetCurrentTimestamp() while holding a spinlock.

CR8 ===

+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to inactive timeout GUC. Also, check the logical

s/inactive timeout GUC/replication_slot_inactive_timeout/?

CR9 ===

+# Start: Helper functions used for this test file
+# End: Helper functions used for this test file

I think that's the first TAP test with this comment. Not saying we should not but
why did you feel the need to add those?

[1]: https://www.postgresql.org/message-id/OS0PR01MB5716B3942AE49F3F725ACA92943B2@OS0PR01MB5716.jpnprd01.prod.outlook.com

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  202. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-04-03T09:27:55Z

    On Wed, Apr 3, 2024 at 11:17 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Apr 3, 2024 at 8:38 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > > Or a simple solution is that the slotsync worker updates
> > > > inactive_since as it does for non-synced slots, and disables
> > > > timeout-based slot invalidation for synced slots.
> >
> > I like this idea better, it takes care of such a case too when the
> > user is relying on sync-function rather than worker and does not want
> > to get the slots invalidated in between 2 sync function calls.
>
> Please find the attached v31 patches implementing the above idea:
>

Thanks for the patches, please find few comments:

v31-001:

1)
system-views.sgml:
value will get updated  after every synchronization from the
corresponding remote slot on the primary.

--This is confusing. It will be good to rephrase it.

2)
update_synced_slots_inactive_since()

--May be, we should mention in the header that this function is called
only during promotion.

3) 040_standby_failover_slots_sync.pl:
We capture inactive_since_on_primary when we do this for the first time at #175
ALTER SUBSCRIPTION regress_mysub1 DISABLE"

But we again recreate the sub and disable it at line #280.
Do you think we shall get inactive_since_on_primary again here, to be
compared with inactive_since_on_new_primary later?


v31-002:
(I had reviewed v29-002 but missed to post comments,  I think these
are still applicable)

1) I think replication_slot_inactivity_timeout was recommended here
(instead of replication_slot_inactive_timeout, so please give it a
thought):
https://www.postgresql.org/message-id/202403260739.udlp7lxixktx%40alvherre.pgsql

2) Commit msg:
a)
"It is often easy for developers to set a timeout of say 1
or 2 or 3 days at slot level, after which the inactive slots get
dropped."

Shall we say invalidated rather than dropped?

b)
"To achieve the above, postgres introduces a GUC allowing users
set inactive timeout and then a slot stays inactive for this much
amount of time it invalidates the slot."

Broken sentence.

<have not reviewed 002 patch in detail yet>

thanks
Shveta

  203. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-03T10:02:16Z

    On Wed, Apr 3, 2024 at 12:20 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> On Wed, Apr 03, 2024 at 11:17:41AM +0530, Bharath Rupireddy wrote:
> > On Wed, Apr 3, 2024 at 8:38 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > > > Or a simple solution is that the slotsync worker updates
> > > > > inactive_since as it does for non-synced slots, and disables
> > > > > timeout-based slot invalidation for synced slots.
> > >
> > > I like this idea better, it takes care of such a case too when the
> > > user is relying on sync-function rather than worker and does not want
> > > to get the slots invalidated in between 2 sync function calls.
> >
> > Please find the attached v31 patches implementing the above idea:
>
> Thanks!
>
> Some comments related to v31-0001:
>
> === testing the behavior
>
> T1 ===
>
> > - synced slots get their on inactive_since just like any other slot
>
> It behaves as described.
>
> T2 ===
>
> > - synced slots inactive_since is set to current timestamp after the
> > standby gets promoted to help inactive_since interpret correctly just
> > like any other slot.
>
> It behaves as described.
>
> CR1 ===
>
> +        <structfield>inactive_since</structfield> value will get updated
> +        after every synchronization
>
> indicates the last synchronization time? (I think that after every synchronization
> could lead to confusion).
>

+1.

> CR2 ===
>
> +                       /*
> +                        * Set the time since the slot has become inactive after shutting
> +                        * down slot sync machinery. This helps correctly interpret the
> +                        * time if the standby gets promoted without a restart.
> +                        */
>
> It looks to me that this comment is not at the right place because there is
> nothing after the comment that indicates that we shutdown the "slot sync machinery".
>
> Maybe a better place is before the function definition and mention that this is
> currently called when we shutdown the "slot sync machinery"?
>

Won't it be better to have an assert for SlotSyncCtx->pid? IIRC, we
have some existing issues where we don't ensure that no one is running
sync API before shutdown is complete but I think we can deal with that
separately and here we can still have an Assert.

> CR3 ===
>
> +                        * We get the current time beforehand and only once to avoid
> +                        * system calls overhead while holding the lock.
>
> s/avoid system calls overhead while holding the lock/avoid system calls while holding the spinlock/?
>

Is it valid to say that there is overhead of this call while holding
spinlock? Because I don't think at the time of promotion we expect any
other concurrent slot activity. The first reason seems good enough.

One other observation:
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -42,6 +42,7 @@
 #include "access/transam.h"
 #include "access/xlog_internal.h"
 #include "access/xlogrecovery.h"
+#include "access/xlogutils.h"

Is there a reason for this inclusion? I don't see any change which
should need this one.

-- 
With Regards,
Amit Kapila.

  204. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-03T10:49:46Z

    On Wed, Apr 3, 2024 at 2:58 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Apr 3, 2024 at 11:17 AM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Wed, Apr 3, 2024 at 8:38 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > > > Or a simple solution is that the slotsync worker updates
> > > > > inactive_since as it does for non-synced slots, and disables
> > > > > timeout-based slot invalidation for synced slots.
> > >
> > > I like this idea better, it takes care of such a case too when the
> > > user is relying on sync-function rather than worker and does not want
> > > to get the slots invalidated in between 2 sync function calls.
> >
> > Please find the attached v31 patches implementing the above idea:
> >
>
> Thanks for the patches, please find few comments:
>
> v31-001:
>
> 1)
> system-views.sgml:
> value will get updated  after every synchronization from the
> corresponding remote slot on the primary.
>
> --This is confusing. It will be good to rephrase it.
>
> 2)
> update_synced_slots_inactive_since()
>
> --May be, we should mention in the header that this function is called
> only during promotion.
>
> 3) 040_standby_failover_slots_sync.pl:
> We capture inactive_since_on_primary when we do this for the first time at #175
> ALTER SUBSCRIPTION regress_mysub1 DISABLE"
>
> But we again recreate the sub and disable it at line #280.
> Do you think we shall get inactive_since_on_primary again here, to be
> compared with inactive_since_on_new_primary later?
>

I think so.

Few additional comments on tests:
1.
+is( $standby1->safe_psql(
+ 'postgres',
+ "SELECT '$inactive_since_on_primary'::timestamptz <
'$inactive_since_on_standby'::timestamptz AND
+ '$inactive_since_on_standby'::timestamptz < '$slot_sync_time'::timestamptz;"

Shall we do <= check as we are doing in the main function
get_slot_inactive_since_value as the time duration is less so it can
be the same as well? Similarly, please check other tests.

2.
+=item $node->get_slot_inactive_since_value(self, slot_name, reference_time)
+
+Get inactive_since column value for a given replication slot validating it
+against optional reference time.
+
+=cut
+
+sub get_slot_inactive_since_value

I see that all callers validate against reference time. It is better
to name it validate_slot_inactive_since rather than using get_* as the
main purpose is to validate the passed value.

-- 
With Regards,
Amit Kapila.

  205. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-03T11:42:12Z

    On Wed, Apr 3, 2024 at 12:20 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > Please find the attached v31 patches implementing the above idea:
>
> Some comments related to v31-0001:
>
> === testing the behavior
>
> T1 ===
>
> > - synced slots get their on inactive_since just like any other slot
>
> It behaves as described.
>
> T2 ===
>
> > - synced slots inactive_since is set to current timestamp after the
> > standby gets promoted to help inactive_since interpret correctly just
> > like any other slot.
>
> It behaves as described.

Thanks for testing.

> CR1 ===
>
> +        <structfield>inactive_since</structfield> value will get updated
> +        after every synchronization
>
> indicates the last synchronization time? (I think that after every synchronization
> could lead to confusion).

Done.

> CR2 ===
>
> +                       /*
> +                        * Set the time since the slot has become inactive after shutting
> +                        * down slot sync machinery. This helps correctly interpret the
> +                        * time if the standby gets promoted without a restart.
> +                        */
>
> It looks to me that this comment is not at the right place because there is
> nothing after the comment that indicates that we shutdown the "slot sync machinery".
>
> Maybe a better place is before the function definition and mention that this is
> currently called when we shutdown the "slot sync machinery"?

Done.

> CR3 ===
>
> +                        * We get the current time beforehand and only once to avoid
> +                        * system calls overhead while holding the lock.
>
> s/avoid system calls overhead while holding the lock/avoid system calls while holding the spinlock/?

Done.

> CR4 ===
>
> +        * Set the time since the slot has become inactive. We get the current
> +        * time beforehand to avoid system call overhead while holding the lock
>
> Same.

Done.

> CR5 ===
>
> +       # Check that the captured time is sane
> +       if (defined $reference_time)
> +       {
>
> s/Check that the captured time is sane/Check that the inactive_since is sane/?
>
> Sorry if some of those comments could have been done while I did review v29-0001.

Done.

On Wed, Apr 3, 2024 at 2:58 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> Thanks for the patches, please find few comments:
>
> v31-001:
>
> 1)
> system-views.sgml:
> value will get updated  after every synchronization from the
> corresponding remote slot on the primary.
>
> --This is confusing. It will be good to rephrase it.

Done as per Bertrand's suggestion.

> 2)
> update_synced_slots_inactive_since()
>
> --May be, we should mention in the header that this function is called
> only during promotion.

Done as per Bertrand's suggestion.

> 3) 040_standby_failover_slots_sync.pl:
> We capture inactive_since_on_primary when we do this for the first time at #175
> ALTER SUBSCRIPTION regress_mysub1 DISABLE"
>
> But we again recreate the sub and disable it at line #280.
> Do you think we shall get inactive_since_on_primary again here, to be
> compared with inactive_since_on_new_primary later?

Hm. Done that. Recapturing both slot_creation_time_on_primary and
inactive_since_on_primary before and after CREATE SUBSCRIPTION creates
the slot again on the primary/publisher.

On Wed, Apr 3, 2024 at 3:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > CR2 ===
> >
> > +                       /*
> > +                        * Set the time since the slot has become inactive after shutting
> > +                        * down slot sync machinery. This helps correctly interpret the
> > +                        * time if the standby gets promoted without a restart.
> > +                        */
> >
> > It looks to me that this comment is not at the right place because there is
> > nothing after the comment that indicates that we shutdown the "slot sync machinery".
> >
> > Maybe a better place is before the function definition and mention that this is
> > currently called when we shutdown the "slot sync machinery"?
> >
> Won't it be better to have an assert for SlotSyncCtx->pid? IIRC, we
> have some existing issues where we don't ensure that no one is running
> sync API before shutdown is complete but I think we can deal with that
> separately and here we can still have an Assert.

That can work to ensure the slot sync worker isn't running as
SlotSyncCtx->pid gets updated only for the slot sync worker. I added
this assertion for now.

We need to ensure (in a separate patch and thread) there is no backend
acquiring it and performing sync while the slot sync worker is
shutting down. Otherwise, some of the slots can get resynced and some
are not while we are shutting down the slot sync worker as part of the
standby promotion which might leave the slots in an inconsistent
state.

> > CR3 ===
> >
> > +                        * We get the current time beforehand and only once to avoid
> > +                        * system calls overhead while holding the lock.
> >
> > s/avoid system calls overhead while holding the lock/avoid system calls while holding the spinlock/?
> >
> Is it valid to say that there is overhead of this call while holding
> spinlock? Because I don't think at the time of promotion we expect any
> other concurrent slot activity. The first reason seems good enough.

No slot activity but why GetCurrentTimestamp needs to be called every
time in a loop.

> One other observation:
> --- a/src/backend/replication/slot.c
> +++ b/src/backend/replication/slot.c
> @@ -42,6 +42,7 @@
>  #include "access/transam.h"
>  #include "access/xlog_internal.h"
>  #include "access/xlogrecovery.h"
> +#include "access/xlogutils.h"
>
> Is there a reason for this inclusion? I don't see any change which
> should need this one.

Not anymore. It was earlier needed for using the InRecovery flag in
the then approach.

On Wed, Apr 3, 2024 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > 3) 040_standby_failover_slots_sync.pl:
> > We capture inactive_since_on_primary when we do this for the first time at #175
> > ALTER SUBSCRIPTION regress_mysub1 DISABLE"
> >
> > But we again recreate the sub and disable it at line #280.
> > Do you think we shall get inactive_since_on_primary again here, to be
> > compared with inactive_since_on_new_primary later?
> >
>
> I think so.

Modified this to recapture the times before and after the slot gets recreated.

> Few additional comments on tests:
> 1.
> +is( $standby1->safe_psql(
> + 'postgres',
> + "SELECT '$inactive_since_on_primary'::timestamptz <
> '$inactive_since_on_standby'::timestamptz AND
> + '$inactive_since_on_standby'::timestamptz < '$slot_sync_time'::timestamptz;"
>
> Shall we do <= check as we are doing in the main function
> get_slot_inactive_since_value as the time duration is less so it can
> be the same as well? Similarly, please check other tests.

I get you. If the tests are so fast that losing a bit of precision
might cause tests to fail. So, I'v added equality check for all the
tests.

> 2.
> +=item $node->get_slot_inactive_since_value(self, slot_name, reference_time)
> +
> +Get inactive_since column value for a given replication slot validating it
> +against optional reference time.
> +
> +=cut
> +
> +sub get_slot_inactive_since_value
>
> I see that all callers validate against reference time. It is better
> to name it validate_slot_inactive_since rather than using get_* as the
> main purpose is to validate the passed value.

Existing callers yes. Also, I've removed the reference time as an
optional parameter.

Per an offlist chat with Amit, I've added the following note in
synchronize_one_slot:

@@ -584,6 +585,11 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
remote_dbid)
          * overwriting 'invalidated' flag to remote_slot's value. See
          * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly
          * if the slot is not acquired by other processes.
+         *
+         * XXX: If it ever turns out that slot acquire/release is costly for
+         * cases when none of the slot property is changed then we can do a
+         * pre-check to ensure that at least one of the slot property is
+         * changed before acquiring the slot.
          */
         ReplicationSlotAcquire(remote_slot->name, true);

Please find the attached v32-0001 patch with the above review comments
addressed. I'm working on review comments for 0002.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  206. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-03T13:16:25Z

    Hi,

On Wed, Apr 03, 2024 at 05:12:12PM +0530, Bharath Rupireddy wrote:
> On Wed, Apr 3, 2024 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > + 'postgres',
> > + "SELECT '$inactive_since_on_primary'::timestamptz <
> > '$inactive_since_on_standby'::timestamptz AND
> > + '$inactive_since_on_standby'::timestamptz < '$slot_sync_time'::timestamptz;"
> >
> > Shall we do <= check as we are doing in the main function
> > get_slot_inactive_since_value as the time duration is less so it can
> > be the same as well? Similarly, please check other tests.
> 
> I get you. If the tests are so fast that losing a bit of precision
> might cause tests to fail. So, I'v added equality check for all the
> tests.

> Please find the attached v32-0001 patch with the above review comments
> addressed.

Thanks!

Just one comment on v32-0001:

+# Synced slot on the standby must get its own inactive_since.
+is( $standby1->safe_psql(
+               'postgres',
+               "SELECT '$inactive_since_on_primary'::timestamptz <= '$inactive_since_on_standby'::timestamptz AND
+                       '$inactive_since_on_standby'::timestamptz <= '$slot_sync_time'::timestamptz;"
+       ),
+       "t",
+       'synchronized slot has got its own inactive_since');
+

By using <= we are not testing that it must get its own inactive_since (as we
allow them to be equal in the test). I think we should just add some usleep()
where appropriate and deny equality during the tests on inactive_since.

Except for the above, v32-0001 LGTM.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  207. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-03T14:58:04Z

    On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Just one comment on v32-0001:
>
> +# Synced slot on the standby must get its own inactive_since.
> +is( $standby1->safe_psql(
> +               'postgres',
> +               "SELECT '$inactive_since_on_primary'::timestamptz <= '$inactive_since_on_standby'::timestamptz AND
> +                       '$inactive_since_on_standby'::timestamptz <= '$slot_sync_time'::timestamptz;"
> +       ),
> +       "t",
> +       'synchronized slot has got its own inactive_since');
> +
>
> By using <= we are not testing that it must get its own inactive_since (as we
> allow them to be equal in the test). I think we should just add some usleep()
> where appropriate and deny equality during the tests on inactive_since.

Thanks. It looks like we can ignore the equality in all of the
inactive_since comparisons. IIUC, all the TAP tests do run with
primary and standbys on the single BF animals. And, it looks like
assigning the inactive_since timestamps to perl variables is giving
the microseconds precision level
(./tmp_check/log/regress_log_040_standby_failover_slots_sync:inactive_since
2024-04-03 14:30:09.691648+00). FWIW, we already have some TAP and SQL
tests relying on stats_reset timestamps without equality. So, I've
left the equality for the inactive_since tests.

> Except for the above, v32-0001 LGTM.

Thanks. Please see the attached v33-0001 patch after removing equality
on inactive_since TAP tests.

On Wed, Apr 3, 2024 at 1:47 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Some comments regarding v31-0002:
>
> === testing the behavior
>
> T1 ===
>
> > - synced slots don't get invalidated due to inactive timeout because
> > such slots not considered active at all as they don't perform logical
> > decoding (of course, they will perform in fast_forward mode to fix the
> > other data loss issue, but they don't generate changes for them to be
> > called as *active* slots)
>
> It behaves as described. OTOH non synced logical slots on the standby and
> physical slots on the standby are invalidated which is what is expected.

Right.

> T2 ===
>
> In case the slot is invalidated on the primary,
>
> primary:
>
> postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
>  slot_name |        inactive_since         | invalidation_reason
> -----------+-------------------------------+---------------------
>  s1        | 2024-04-03 06:56:28.075637+00 | inactive_timeout
>
> then on the standby we get:
>
> standby:
>
> postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
>  slot_name |        inactive_since        | invalidation_reason
> -----------+------------------------------+---------------------
>  s1        | 2024-04-03 07:06:43.37486+00 | inactive_timeout
>
> shouldn't the slot be dropped/recreated instead of updating inactive_since?

The sync slots that are invalidated on the primary aren't dropped and
recreated on the standby. There's no point in doing so because
invalidated slots on the primary can't be made useful. However, I
found that the synced slot is acquired and released unnecessarily
after the invalidation_reason is synced from the primary. I added a
skip check in synchronize_one_slot to skip acquiring and releasing the
slot if it's locally found inactive. With this, inactive_since won't
get updated for invalidated sync slots on the standby as we don't
acquire and release the slot.

> === code
>
> CR1 ===
>
> +        Invalidates replication slots that are inactive for longer the
> +        specified amount of time
>
> s/for longer the/for longer that/?

Fixed.

> CR2 ===
>
> +        <literal>true</literal>) as such synced slots don't actually perform
> +        logical decoding.
>
> We're switching in fast forward logical due to [1], so I'm not sure that's 100%
> accurate here. I'm not sure we need to specify a reason.

Fixed.

> CR3 ===
>
> + errdetail("This slot has been invalidated because it was inactive for more than the time specified by replication_slot_inactive_timeout parameter.")));
>
> I think we can remove "parameter" (see for example the error message in
> validate_remote_info()) and reduce it a bit, something like?
>
> "This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout"?

Done.

> CR4 ===
>
> + appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by replication_slot_inactive_timeout parameter."));
>
> Same.

Done. Changed it to "The slot has been inactive for more than
replication_slot_inactive_timeout."

> CR5 ===
>
> +       /*
> +        * This function isn't expected to be called for inactive timeout based
> +        * invalidation. A separate function InvalidateInactiveReplicationSlot is
> +        * to be used for that.
>
> Do you think it's worth to explain why?

Hm, I just wanted to point out the actual function here. I modified it
to something like the following, if others feel we don't need that, I
can remove it.

    /*
     * Use InvalidateInactiveReplicationSlot for inactive timeout based
     * invalidation.
     */

> CR6 ===
>
> +       if (replication_slot_inactive_timeout == 0)
> +               return false;
> +       else if (slot->inactive_since > 0)
>
> "else" is not needed here.

Nothing wrong there, but removed.

> CR7 ===
>
> +               SpinLockAcquire(&slot->mutex);
> +
> +               /*
> +                * Check if the slot needs to be invalidated due to
> +                * replication_slot_inactive_timeout GUC. We do this with the spinlock
> +                * held to avoid race conditions -- for example the inactive_since
> +                * could change, or the slot could be dropped.
> +                */
> +               now = GetCurrentTimestamp();
>
> We should not call GetCurrentTimestamp() while holding a spinlock.

I was thinking why to add up the wait time to acquire
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);. Now that I
moved it up before the spinlock but after the LWLockAcquire.

> CR8 ===
>
> +# Testcase start: Invalidate streaming standby's slot as well as logical
> +# failover slot on primary due to inactive timeout GUC. Also, check the logical
>
> s/inactive timeout GUC/replication_slot_inactive_timeout/?

Done.

> CR9 ===
>
> +# Start: Helper functions used for this test file
> +# End: Helper functions used for this test file
>
> I think that's the first TAP test with this comment. Not saying we should not but
> why did you feel the need to add those?

Hm. Removed.

> [1]: https://www.postgresql.org/message-id/OS0PR01MB5716B3942AE49F3F725ACA92943B2@OS0PR01MB5716.jpnprd01.prod.outlook.com


On Wed, Apr 3, 2024 at 2:58 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> v31-002:
> (I had reviewed v29-002 but missed to post comments,  I think these
> are still applicable)
>
> 1) I think replication_slot_inactivity_timeout was recommended here
> (instead of replication_slot_inactive_timeout, so please give it a
> thought):
> https://www.postgresql.org/message-id/202403260739.udlp7lxixktx%40alvherre.pgsql

Yeah. It's synonymous with inactive_since. If others have an opinion
to have replication_slot_inactivity_timeout, I'm fine with it.

> 2) Commit msg:
> a)
> "It is often easy for developers to set a timeout of say 1
> or 2 or 3 days at slot level, after which the inactive slots get
> dropped."
>
> Shall we say invalidated rather than dropped?

Right. Done that.

> b)
> "To achieve the above, postgres introduces a GUC allowing users
> set inactive timeout and then a slot stays inactive for this much
> amount of time it invalidates the slot."
>
> Broken sentence.

Reworded it a bit.

Please find the attached v33 patches.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  208. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-03T16:27:45Z

    Hi,

On Wed, Apr 03, 2024 at 08:28:04PM +0530, Bharath Rupireddy wrote:
> On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Just one comment on v32-0001:
> >
> > +# Synced slot on the standby must get its own inactive_since.
> > +is( $standby1->safe_psql(
> > +               'postgres',
> > +               "SELECT '$inactive_since_on_primary'::timestamptz <= '$inactive_since_on_standby'::timestamptz AND
> > +                       '$inactive_since_on_standby'::timestamptz <= '$slot_sync_time'::timestamptz;"
> > +       ),
> > +       "t",
> > +       'synchronized slot has got its own inactive_since');
> > +
> >
> > By using <= we are not testing that it must get its own inactive_since (as we
> > allow them to be equal in the test). I think we should just add some usleep()
> > where appropriate and deny equality during the tests on inactive_since.
> 
> > Except for the above, v32-0001 LGTM.
> 
> Thanks. Please see the attached v33-0001 patch after removing equality
> on inactive_since TAP tests.

Thanks! v33-0001 LGTM.

> On Wed, Apr 3, 2024 at 1:47 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> > Some comments regarding v31-0002:
> >
> > T2 ===
> >
> > In case the slot is invalidated on the primary,
> >
> > primary:
> >
> > postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
> >  slot_name |        inactive_since         | invalidation_reason
> > -----------+-------------------------------+---------------------
> >  s1        | 2024-04-03 06:56:28.075637+00 | inactive_timeout
> >
> > then on the standby we get:
> >
> > standby:
> >
> > postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
> >  slot_name |        inactive_since        | invalidation_reason
> > -----------+------------------------------+---------------------
> >  s1        | 2024-04-03 07:06:43.37486+00 | inactive_timeout
> >
> > shouldn't the slot be dropped/recreated instead of updating inactive_since?
> 
> The sync slots that are invalidated on the primary aren't dropped and
> recreated on the standby.

Yeah, right (I was confused with synced slots that are invalidated locally).

> However, I
> found that the synced slot is acquired and released unnecessarily
> after the invalidation_reason is synced from the primary. I added a
> skip check in synchronize_one_slot to skip acquiring and releasing the
> slot if it's locally found inactive. With this, inactive_since won't
> get updated for invalidated sync slots on the standby as we don't
> acquire and release the slot.

CR1 ===

Yeah, I can see:

@@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
                                                   " name slot \"%s\" already exists on the standby",
                                                   remote_slot->name));

+               /*
+                * Skip the sync if the local slot is already invalidated. We do this
+                * beforehand to save on slot acquire and release.
+                */
+               if (slot->data.invalidated != RS_INVAL_NONE)
+                       return false;

Thanks to the drop_local_obsolete_slots() call I think we are not missing the case
where the slot has been invalidated on the primary, invalidation reason has been
synced on the standby and later the slot is dropped/ recreated manually on the
primary (then it should be dropped/recreated on the standby too).

Also it seems we are not missing the case where a sync slot is invalidated
locally due to wal removal (it should be dropped/recreated).

> 
> > CR5 ===
> >
> > +       /*
> > +        * This function isn't expected to be called for inactive timeout based
> > +        * invalidation. A separate function InvalidateInactiveReplicationSlot is
> > +        * to be used for that.
> >
> > Do you think it's worth to explain why?
> 
> Hm, I just wanted to point out the actual function here. I modified it
> to something like the following, if others feel we don't need that, I
> can remove it.

Sorry If I was not clear but I meant to say "Do you think it's worth to explain
why we decided to create a dedicated function"? (currently we "just" explain that
we created one).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  209. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-04-04T04:11:31Z

    On Wed, Apr 3, 2024 at 11:58 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
>
> Please find the attached v33 patches.

@@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
    if (SlotSyncCtx->pid == InvalidPid)
    {
        SpinLockRelease(&SlotSyncCtx->mutex);
+       update_synced_slots_inactive_since();
        return;
    }
    SpinLockRelease(&SlotSyncCtx->mutex);
@@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
    }

    SpinLockRelease(&SlotSyncCtx->mutex);
+
+   update_synced_slots_inactive_since();
 }

Why do we want to update all synced slots' inactive_since values at
shutdown in spite of updating the value every time when releasing the
slot? It seems to contradict the fact that inactive_since is updated
when releasing or restoring the slot.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  210. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-04T04:34:02Z

    On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
>     if (SlotSyncCtx->pid == InvalidPid)
>     {
>         SpinLockRelease(&SlotSyncCtx->mutex);
> +       update_synced_slots_inactive_since();
>         return;
>     }
>     SpinLockRelease(&SlotSyncCtx->mutex);
> @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
>     }
>
>     SpinLockRelease(&SlotSyncCtx->mutex);
> +
> +   update_synced_slots_inactive_since();
>  }
>
> Why do we want to update all synced slots' inactive_since values at
> shutdown in spite of updating the value every time when releasing the
> slot? It seems to contradict the fact that inactive_since is updated
> when releasing or restoring the slot.

It is to get the inactive_since right for the cases where the standby
is promoted without a restart similar to when a standby is promoted
with restart in which case the inactive_since is set to current time
in RestoreSlotFromDisk.

Imagine the slot is synced last time at time t1 and then a few hours
passed, the standby is promoted without a restart. If we don't set
inactive_since to current time in this case in ShutDownSlotSync, the
inactive timeout invalidation mechanism can kick in immediately.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  211. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-04T05:18:11Z

    On Wed, Apr 3, 2024 at 8:28 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Just one comment on v32-0001:
> >
> > +# Synced slot on the standby must get its own inactive_since.
> > +is( $standby1->safe_psql(
> > +               'postgres',
> > +               "SELECT '$inactive_since_on_primary'::timestamptz <= '$inactive_since_on_standby'::timestamptz AND
> > +                       '$inactive_since_on_standby'::timestamptz <= '$slot_sync_time'::timestamptz;"
> > +       ),
> > +       "t",
> > +       'synchronized slot has got its own inactive_since');
> > +
> >
> > By using <= we are not testing that it must get its own inactive_since (as we
> > allow them to be equal in the test). I think we should just add some usleep()
> > where appropriate and deny equality during the tests on inactive_since.
>
> Thanks. It looks like we can ignore the equality in all of the
> inactive_since comparisons. IIUC, all the TAP tests do run with
> primary and standbys on the single BF animals. And, it looks like
> assigning the inactive_since timestamps to perl variables is giving
> the microseconds precision level
> (./tmp_check/log/regress_log_040_standby_failover_slots_sync:inactive_since
> 2024-04-03 14:30:09.691648+00). FWIW, we already have some TAP and SQL
> tests relying on stats_reset timestamps without equality. So, I've
> left the equality for the inactive_since tests.
>
> > Except for the above, v32-0001 LGTM.
>
> Thanks. Please see the attached v33-0001 patch after removing equality
> on inactive_since TAP tests.
>

The v33-0001 looks good to me. I have made minor changes in the
comments/commit message and removed one part of the test which was a
bit confusing and didn't seem to add much value. Let me know what you
think of the attached?

-- 
With Regards,
Amit Kapila.

  212. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-04T05:42:11Z

    On Thu, Apr 4, 2024 at 10:48 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> The v33-0001 looks good to me. I have made minor changes in the
> comments/commit message and removed one part of the test which was a
> bit confusing and didn't seem to add much value. Let me know what you
> think of the attached?

Thanks for the changes. v34-0001 LGTM.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  213. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-04-04T08:01:30Z

    On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
> >     if (SlotSyncCtx->pid == InvalidPid)
> >     {
> >         SpinLockRelease(&SlotSyncCtx->mutex);
> > +       update_synced_slots_inactive_since();
> >         return;
> >     }
> >     SpinLockRelease(&SlotSyncCtx->mutex);
> > @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
> >     }
> >
> >     SpinLockRelease(&SlotSyncCtx->mutex);
> > +
> > +   update_synced_slots_inactive_since();
> >  }
> >
> > Why do we want to update all synced slots' inactive_since values at
> > shutdown in spite of updating the value every time when releasing the
> > slot? It seems to contradict the fact that inactive_since is updated
> > when releasing or restoring the slot.
>
> It is to get the inactive_since right for the cases where the standby
> is promoted without a restart similar to when a standby is promoted
> with restart in which case the inactive_since is set to current time
> in RestoreSlotFromDisk.
>
> Imagine the slot is synced last time at time t1 and then a few hours
> passed, the standby is promoted without a restart. If we don't set
> inactive_since to current time in this case in ShutDownSlotSync, the
> inactive timeout invalidation mechanism can kick in immediately.
>

Thank you for the explanation! I understood the needs.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  214. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-04T08:36:42Z

    On Thu, Apr 4, 2024 at 1:32 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
> > >     if (SlotSyncCtx->pid == InvalidPid)
> > >     {
> > >         SpinLockRelease(&SlotSyncCtx->mutex);
> > > +       update_synced_slots_inactive_since();
> > >         return;
> > >     }
> > >     SpinLockRelease(&SlotSyncCtx->mutex);
> > > @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
> > >     }
> > >
> > >     SpinLockRelease(&SlotSyncCtx->mutex);
> > > +
> > > +   update_synced_slots_inactive_since();
> > >  }
> > >
> > > Why do we want to update all synced slots' inactive_since values at
> > > shutdown in spite of updating the value every time when releasing the
> > > slot? It seems to contradict the fact that inactive_since is updated
> > > when releasing or restoring the slot.
> >
> > It is to get the inactive_since right for the cases where the standby
> > is promoted without a restart similar to when a standby is promoted
> > with restart in which case the inactive_since is set to current time
> > in RestoreSlotFromDisk.
> >
> > Imagine the slot is synced last time at time t1 and then a few hours
> > passed, the standby is promoted without a restart. If we don't set
> > inactive_since to current time in this case in ShutDownSlotSync, the
> > inactive timeout invalidation mechanism can kick in immediately.
> >
>
> Thank you for the explanation! I understood the needs.
>

Do you want to review the v34_0001* further or shall I proceed with
the commit of the same?

-- 
With Regards,
Amit Kapila.

  215. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-04-04T09:05:02Z

    On Thu, Apr 4, 2024 at 5:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Apr 4, 2024 at 1:32 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy
> > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > >
> > > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
> > > >     if (SlotSyncCtx->pid == InvalidPid)
> > > >     {
> > > >         SpinLockRelease(&SlotSyncCtx->mutex);
> > > > +       update_synced_slots_inactive_since();
> > > >         return;
> > > >     }
> > > >     SpinLockRelease(&SlotSyncCtx->mutex);
> > > > @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
> > > >     }
> > > >
> > > >     SpinLockRelease(&SlotSyncCtx->mutex);
> > > > +
> > > > +   update_synced_slots_inactive_since();
> > > >  }
> > > >
> > > > Why do we want to update all synced slots' inactive_since values at
> > > > shutdown in spite of updating the value every time when releasing the
> > > > slot? It seems to contradict the fact that inactive_since is updated
> > > > when releasing or restoring the slot.
> > >
> > > It is to get the inactive_since right for the cases where the standby
> > > is promoted without a restart similar to when a standby is promoted
> > > with restart in which case the inactive_since is set to current time
> > > in RestoreSlotFromDisk.
> > >
> > > Imagine the slot is synced last time at time t1 and then a few hours
> > > passed, the standby is promoted without a restart. If we don't set
> > > inactive_since to current time in this case in ShutDownSlotSync, the
> > > inactive timeout invalidation mechanism can kick in immediately.
> > >
> >
> > Thank you for the explanation! I understood the needs.
> >
>
> Do you want to review the v34_0001* further or shall I proceed with
> the commit of the same?

Thanks for asking. The v34-0001 patch looks good to me.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  216. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-04T11:05:01Z

    On Thu, Apr 4, 2024 at 11:12 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Thu, Apr 4, 2024 at 10:48 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > The v33-0001 looks good to me. I have made minor changes in the
> > comments/commit message and removed one part of the test which was a
> > bit confusing and didn't seem to add much value. Let me know what you
> > think of the attached?
>
> Thanks for the changes. v34-0001 LGTM.
>

I was doing a final review before pushing 0001 and found that
'inactive_since' could be set twice during startup after promotion,
once while restoring slots and then via ShutDownSlotSync(). The reason
is that ShutDownSlotSync() will be invoked in normal startup on
primary though it won't do anything apart from setting inactive_since
if we have synced slots. I think you need to check 'StandbyMode' in
update_synced_slots_inactive_since() and return if the same is not
set. We can't use 'InRecovery' flag as that will be set even during
crash recovery.

Can you please test this once unless you don't agree with the above theory?

-- 
With Regards,
Amit Kapila.

  217. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-04T12:22:50Z

    On Thu, Apr 4, 2024 at 4:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > Thanks for the changes. v34-0001 LGTM.
>
> I was doing a final review before pushing 0001 and found that
> 'inactive_since' could be set twice during startup after promotion,
> once while restoring slots and then via ShutDownSlotSync(). The reason
> is that ShutDownSlotSync() will be invoked in normal startup on
> primary though it won't do anything apart from setting inactive_since
> if we have synced slots. I think you need to check 'StandbyMode' in
> update_synced_slots_inactive_since() and return if the same is not
> set. We can't use 'InRecovery' flag as that will be set even during
> crash recovery.
>
> Can you please test this once unless you don't agree with the above theory?

Nice catch. I've verified that update_synced_slots_inactive_since is
called even for normal server startups/crash recovery. I've added a
check to exit if the StandbyMode isn't set.

Please find the attached v35 patch.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  218. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-04-05T03:09:13Z

    On Thu, Apr 4, 2024 at 5:53 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Thu, Apr 4, 2024 at 4:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > Thanks for the changes. v34-0001 LGTM.
> >
> > I was doing a final review before pushing 0001 and found that
> > 'inactive_since' could be set twice during startup after promotion,
> > once while restoring slots and then via ShutDownSlotSync(). The reason
> > is that ShutDownSlotSync() will be invoked in normal startup on
> > primary though it won't do anything apart from setting inactive_since
> > if we have synced slots. I think you need to check 'StandbyMode' in
> > update_synced_slots_inactive_since() and return if the same is not
> > set. We can't use 'InRecovery' flag as that will be set even during
> > crash recovery.
> >
> > Can you please test this once unless you don't agree with the above theory?
>
> Nice catch. I've verified that update_synced_slots_inactive_since is
> called even for normal server startups/crash recovery. I've added a
> check to exit if the StandbyMode isn't set.
>
> Please find the attached v35 patch.

Thanks for the patch. Tested it , works well. Few cosmetic changes needed:

in 040 test file:
1)
# Capture the inactive_since of the slot from the primary. Note that the slot
# will be inactive since the corresponding subscription is disabled..

2 .. at the end. Replace with one.

2)
# Synced slot on the standby must get its own inactive_since.

. not needed in single line comment (to be consistent with
neighbouring comments)


3)
update_synced_slots_inactive_since():

if (!StandbyMode)
return;

It will be good to add comments here.

thanks
Shveta

  219. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-05T05:51:43Z

    On Wed, Apr 3, 2024 at 9:57 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > > shouldn't the slot be dropped/recreated instead of updating inactive_since?
> >
> > The sync slots that are invalidated on the primary aren't dropped and
> > recreated on the standby.
>
> Yeah, right (I was confused with synced slots that are invalidated locally).
>
> > However, I
> > found that the synced slot is acquired and released unnecessarily
> > after the invalidation_reason is synced from the primary. I added a
> > skip check in synchronize_one_slot to skip acquiring and releasing the
> > slot if it's locally found inactive. With this, inactive_since won't
> > get updated for invalidated sync slots on the standby as we don't
> > acquire and release the slot.
>
> CR1 ===
>
> Yeah, I can see:
>
> @@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
>                                                    " name slot \"%s\" already exists on the standby",
>                                                    remote_slot->name));
>
> +               /*
> +                * Skip the sync if the local slot is already invalidated. We do this
> +                * beforehand to save on slot acquire and release.
> +                */
> +               if (slot->data.invalidated != RS_INVAL_NONE)
> +                       return false;
>
> Thanks to the drop_local_obsolete_slots() call I think we are not missing the case
> where the slot has been invalidated on the primary, invalidation reason has been
> synced on the standby and later the slot is dropped/ recreated manually on the
> primary (then it should be dropped/recreated on the standby too).
>
> Also it seems we are not missing the case where a sync slot is invalidated
> locally due to wal removal (it should be dropped/recreated).

Right.

> > > CR5 ===
> > >
> > > +       /*
> > > +        * This function isn't expected to be called for inactive timeout based
> > > +        * invalidation. A separate function InvalidateInactiveReplicationSlot is
> > > +        * to be used for that.
> > >
> > > Do you think it's worth to explain why?
> >
> > Hm, I just wanted to point out the actual function here. I modified it
> > to something like the following, if others feel we don't need that, I
> > can remove it.
>
> Sorry If I was not clear but I meant to say "Do you think it's worth to explain
> why we decided to create a dedicated function"? (currently we "just" explain that
> we created one).

We added a new function (InvalidateInactiveReplicationSlot) to
invalidate slot based on inactive timeout because 1) we do the
inactive timeout invalidation at slot level as opposed to
InvalidateObsoleteReplicationSlots which does loop over all the slots,
2)
InvalidatePossiblyObsoleteSlot does release the lock in some cases,
has a lot of unneeded code for inactive timeout invalidation check, 3)
we want some control over saving the slot to disk because we hook the
inactive timeout invalidation into the loop that checkpoints the slot
info to the disk in CheckPointReplicationSlots.

I've added a comment atop InvalidateInactiveReplicationSlot.

Please find the attached v36 patch.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  220. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2024-04-05T07:43:58Z

    Hi,

On Fri, Apr 05, 2024 at 11:21:43AM +0530, Bharath Rupireddy wrote:
> On Wed, Apr 3, 2024 at 9:57 PM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> Please find the attached v36 patch.

Thanks!

A few comments:

1 ===

+       <para>
+        The timeout is measured from the time since the slot has become
+        inactive (known from its
+        <structfield>inactive_since</structfield> value) until it gets
+        used (i.e., its <structfield>active</structfield> is set to true).
+       </para>

That's right except when it's invalidated during the checkpoint (as the slot
is not acquired in CheckPointReplicationSlots()).

So, what about adding: "or a checkpoint occurs"? That would also explain that
the invalidation could occur during checkpoint.

2 ===

+       /* If the slot has been invalidated, recalculate the resource limits */
+       if (invalidated)
+       {

/If the slot/If a slot/?

3 ===

+ * NB - this function also runs as part of checkpoint, so avoid raising errors

s/NB - this/NB: This function/? (that looks more consistent with other comments
in the code)

4 ===

+ * Note that having a new function for RS_INVAL_INACTIVE_TIMEOUT cause instead

I understand it's "the RS_INVAL_INACTIVE_TIMEOUT cause" but reading "cause instead"
looks weird to me. Maybe it would make sense to reword this a bit.

5 ===

+        * considered not active as they don't actually perform logical decoding.

Not sure that's 100% accurate as we switched in fast forward logical
in 2ec005b4e2.

"as they perform only fast forward logical decoding (or not at all)", maybe?

6 ===

+       if (RecoveryInProgress() && slot->data.synced)
+               return false;
+
+       if (replication_slot_inactive_timeout == 0)
+               return false;

What about just using one if? It's more a matter of taste but it also probably
reduces the object file size a bit for non optimized build.

7 ===

+               /*
+                * Do not invalidate the slots which are currently being synced from
+                * the primary to the standby.
+                */
+               if (RecoveryInProgress() && slot->data.synced)
+                       return false;

I think we don't need this check as the exact same one is done just before.

8 ===

+sub check_for_slot_invalidation_in_server_log
+{
+       my ($node, $slot_name, $offset) = @_;
+       my $invalidated = 0;
+
+       for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+       {
+               $node->safe_psql('postgres', "CHECKPOINT");

Wouldn't be better to wait for the replication_slot_inactive_timeout time before
instead of triggering all those checkpoints? (it could be passed as an extra arg
to wait_for_slot_invalidation()).

9 ===

# Synced slot mustn't get invalidated on the standby, it must sync invalidation
# from the primary. So, we must not see the slot's invalidation message in server
# log.
ok( !$standby1->log_contains(
        "invalidating obsolete replication slot \"lsub1_sync_slot\"",
        $standby1_logstart),
    'check that syned slot has not been invalidated on the standby');

Would that make sense to trigger a checkpoint on the standby before this test?
I mean I think that without a checkpoint on the standby we should not see the
invalidation in the log anyway.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  221. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-06T06:25:38Z

    On Fri, Apr 5, 2024 at 1:14 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> > Please find the attached v36 patch.
>
> A few comments:
>
> 1 ===
>
> +       <para>
> +        The timeout is measured from the time since the slot has become
> +        inactive (known from its
> +        <structfield>inactive_since</structfield> value) until it gets
> +        used (i.e., its <structfield>active</structfield> is set to true).
> +       </para>
>
> That's right except when it's invalidated during the checkpoint (as the slot
> is not acquired in CheckPointReplicationSlots()).
>
> So, what about adding: "or a checkpoint occurs"? That would also explain that
> the invalidation could occur during checkpoint.

Reworded.

> 2 ===
>
> +       /* If the slot has been invalidated, recalculate the resource limits */
> +       if (invalidated)
> +       {
>
> /If the slot/If a slot/?

Modified it to be like elsewhere.

> 3 ===
>
> + * NB - this function also runs as part of checkpoint, so avoid raising errors
>
> s/NB - this/NB: This function/? (that looks more consistent with other comments
> in the code)

Done.

> 4 ===
>
> + * Note that having a new function for RS_INVAL_INACTIVE_TIMEOUT cause instead
>
> I understand it's "the RS_INVAL_INACTIVE_TIMEOUT cause" but reading "cause instead"
> looks weird to me. Maybe it would make sense to reword this a bit.

Reworded.

> 5 ===
>
> +        * considered not active as they don't actually perform logical decoding.
>
> Not sure that's 100% accurate as we switched in fast forward logical
> in 2ec005b4e2.
>
> "as they perform only fast forward logical decoding (or not at all)", maybe?

Changed it to "as they don't perform logical decoding to produce the
changes". In fast_forward mode no changes are produced.

> 6 ===
>
> +       if (RecoveryInProgress() && slot->data.synced)
> +               return false;
> +
> +       if (replication_slot_inactive_timeout == 0)
> +               return false;
>
> What about just using one if? It's more a matter of taste but it also probably
> reduces the object file size a bit for non optimized build.

Changed.

> 7 ===
>
> +               /*
> +                * Do not invalidate the slots which are currently being synced from
> +                * the primary to the standby.
> +                */
> +               if (RecoveryInProgress() && slot->data.synced)
> +                       return false;
>
> I think we don't need this check as the exact same one is done just before.

Right. Removed.

> 8 ===
>
> +sub check_for_slot_invalidation_in_server_log
> +{
> +       my ($node, $slot_name, $offset) = @_;
> +       my $invalidated = 0;
> +
> +       for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
> +       {
> +               $node->safe_psql('postgres', "CHECKPOINT");
>
> Wouldn't be better to wait for the replication_slot_inactive_timeout time before
> instead of triggering all those checkpoints? (it could be passed as an extra arg
> to wait_for_slot_invalidation()).

Done.

> 9 ===
>
> # Synced slot mustn't get invalidated on the standby, it must sync invalidation
> # from the primary. So, we must not see the slot's invalidation message in server
> # log.
> ok( !$standby1->log_contains(
>         "invalidating obsolete replication slot \"lsub1_sync_slot\"",
>         $standby1_logstart),
>     'check that syned slot has not been invalidated on the standby');
>
> Would that make sense to trigger a checkpoint on the standby before this test?
> I mean I think that without a checkpoint on the standby we should not see the
> invalidation in the log anyway.

Done.

Please find the attached v37 patch for further review.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  222. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-06T06:48:34Z

    On Sat, Apr 6, 2024 at 11:55 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>

Why the handling w.r.t active_pid in InvalidatePossiblyInactiveSlot()
is not similar to InvalidatePossiblyObsoleteSlot(). Won't we need to
ensure that there is no other active slot user? Is it sufficient to
check inactive_since for the same? If so, we need some comments to
explain the same.

Can we avoid introducing the new functions like
SaveGivenReplicationSlot() and MarkGivenReplicationSlotDirty(), if we
do the required work in the caller?

-- 
With Regards,
Amit Kapila.

  223. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-06T11:40:19Z

    On Sat, Apr 6, 2024 at 12:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Why the handling w.r.t active_pid in InvalidatePossiblyInactiveSlot()
> is not similar to InvalidatePossiblyObsoleteSlot(). Won't we need to
> ensure that there is no other active slot user? Is it sufficient to
> check inactive_since for the same? If so, we need some comments to
> explain the same.

I removed the separate functions and with minimal changes, I've now
placed the RS_INVAL_INACTIVE_TIMEOUT logic into
InvalidatePossiblyObsoleteSlot and use that even in
CheckPointReplicationSlots.

> Can we avoid introducing the new functions like
> SaveGivenReplicationSlot() and MarkGivenReplicationSlotDirty(), if we
> do the required work in the caller?

Hm. Removed them now.

Please see the attached v38 patch.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  224. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-13T04:06:25Z

    On Sat, Apr 6, 2024 at 5:10 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Please see the attached v38 patch.

Hi, thanks everyone for reviewing the design and patches so far. Here
I'm with the v39 patches implementing inactive timeout based (0001)
and XID age based (0002) invalidation mechanisms.

I'm quoting the hackers who are okay with inactive timeout based
invalidation mechanism:
Bertrand Drouvot -
https://www.postgresql.org/message-id/ZgL0N%2BxVJNkyqsKL%40ip-10-97-1-34.eu-west-3.compute.internal
and https://www.postgresql.org/message-id/ZgPHDAlM79iLtGIH%40ip-10-97-1-34.eu-west-3.compute.internal
Amit Kapila - https://www.postgresql.org/message-id/CAA4eK1L3awyzWMuymLJUm8SoFEQe%3DDa9KUwCcAfC31RNJ1xdJA%40mail.gmail.com
Nathan Bossart -
https://www.postgresql.org/message-id/20240325195443.GA2923888%40nathanxps13
Robert Haas - https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com

I'm quoting the hackers who are okay with XID age based invalidation mechanism:
Nathan Bossart -
https://www.postgresql.org/message-id/20240326150918.GB3181099%40nathanxps13
and https://www.postgresql.org/message-id/20240327150557.GA3994937%40nathanxps13
Alvaro Herrera -
https://www.postgresql.org/message-id/202403261539.xcjfle7sksz7%40alvherre.pgsql
Bertrand Drouvot -
https://www.postgresql.org/message-id/ZgPHDAlM79iLtGIH%40ip-10-97-1-34.eu-west-3.compute.internal
Amit Kapila - https://www.postgresql.org/message-id/CAA4eK1L3awyzWMuymLJUm8SoFEQe%3DDa9KUwCcAfC31RNJ1xdJA%40mail.gmail.com

There was a point raised by Robert
https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
for XID age based invalidation. An issue related to
vacuum_defer_cleanup_age
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=be504a3e974d75be6f95c8f9b7367126034f2d12
led to the removal of the GUC
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=1118cd37eb61e6a2428f457a8b2026a7bb3f801a.
The same issue may not happen for the XID age based invaliation. This
is because the XID age is not calculated using FullTransactionId but
using TransactionId as the slot's xmin and catalog_xmin are tracked as
TransactionId.

There was a point raised by Amit
https://www.postgresql.org/message-id/CAA4eK1K8wqLsMw6j0hE_SFoWAeo3Kw8UNnMfhsWaYDF1GWYQ%2Bg%40mail.gmail.com
on when to do the XID age based invalidation - whether in checkpointer
or when vacuum is being run or whenever ComputeXIDHorizons gets called
or in autovacuum process. For now, I've chosen the design to do these
new invalidation checks in two places - 1) whenever the slot is
acquired and the slot acquisition errors out if invalidated, 2) during
checkpoint. However, I'm open to suggestions on this.

I've also verified the case whether the replication_slot_xid_age
setting can help in case of server inching towards the XID wraparound.
I've created a primary and streaming standby setup with
hot_standby_feedback set to on (so that the slot gets an xmin). Then,
I've set replication_slot_xid_age to 2 billion on the primary, and
used xid_wraparound extension to reach XID wraparound on the primary.
Once I start receiving the WARNINGs about VACUUM, I did a checkpoint
after which the slot got invalidated enabling my VACUUM to freeze XIDs
saving my database from XID wraparound problem.

Thanks a lot Masahiko Sawada for an offlist chat about the XID age
calculation logic.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  225. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-04-22T13:50:37Z

    Hi,

On Thu, Apr 4, 2024 at 9:23 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Thu, Apr 4, 2024 at 4:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > Thanks for the changes. v34-0001 LGTM.
> >
> > I was doing a final review before pushing 0001 and found that
> > 'inactive_since' could be set twice during startup after promotion,
> > once while restoring slots and then via ShutDownSlotSync(). The reason
> > is that ShutDownSlotSync() will be invoked in normal startup on
> > primary though it won't do anything apart from setting inactive_since
> > if we have synced slots. I think you need to check 'StandbyMode' in
> > update_synced_slots_inactive_since() and return if the same is not
> > set. We can't use 'InRecovery' flag as that will be set even during
> > crash recovery.
> >
> > Can you please test this once unless you don't agree with the above theory?
>
> Nice catch. I've verified that update_synced_slots_inactive_since is
> called even for normal server startups/crash recovery. I've added a
> check to exit if the StandbyMode isn't set.
>
> Please find the attached v35 patch.
>

The documentation says about both 'active' and 'inactive_since'
columns of pg_replication_slots say:

---
active bool
True if this slot is currently actively being used

inactive_since timestamptz
The time since the slot has become inactive. NULL if the slot is
currently being used. Note that for slots on the standby that are
being synced from a primary server (whose synced field is true), the
inactive_since indicates the last synchronization (see Section 47.2.3)
time.
---

When reading the description I thought if 'active' is true,
'inactive_since' is NULL, but it doesn't seem to apply for temporary
slots. Since we don't reset the active_pid field of temporary slots
when the release, the 'active' is still true in the view but
'inactive_since' is not NULL. Do you think we need to mention it in
the documentation?

As for the timeout-based slot invalidation feature, we could end up
invalidating the temporary slots even if they are shown as active,
which could confuse users. Do we want to somehow deal with it?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  226. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-04-25T05:41:35Z

    On Mon, Apr 22, 2024 at 7:21 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > Please find the attached v35 patch.
>
> The documentation says about both 'active' and 'inactive_since'
> columns of pg_replication_slots say:
>
> ---
> active bool
> True if this slot is currently actively being used
>
> inactive_since timestamptz
> The time since the slot has become inactive. NULL if the slot is
> currently being used. Note that for slots on the standby that are
> being synced from a primary server (whose synced field is true), the
> inactive_since indicates the last synchronization (see Section 47.2.3)
> time.
> ---
>
> When reading the description I thought if 'active' is true,
> 'inactive_since' is NULL, but it doesn't seem to apply for temporary
> slots.

Right.

> Since we don't reset the active_pid field of temporary slots
> when the release, the 'active' is still true in the view but
> 'inactive_since' is not NULL.

Right. inactive_since is reset whenever the temporary slot is acquired
again within the same backend that created the temporary slot.

> Do you think we need to mention it in
> the documentation?

I think that's the reason we dropped "active" from the statement. It
was earlier "NULL if the slot is currently actively being used.". But,
per Bertrand's comment
https://www.postgresql.org/message-id/ZehE2IJcsetSJMHC%40ip-10-97-1-34.eu-west-3.compute.internal
changed it to ""NULL if the slot is currently being used.".

Temporary slots retain the active = true and active_pid = <pid of the
backend that created it> even when the slot is not being used until
the lifetime of the backend process. We haven't tied active or
active_pid flags to inactive_since, doing so now to represent the
temporary slot behaviour for active and active_pid will confuse users
more. As far as the inactive_since of a slot is concerned, it is set
to 0 when the slot is being used (acquired) and set to current
timestamp when the slot is not being used (released).

> As for the timeout-based slot invalidation feature, we could end up
> invalidating the temporary slots even if they are shown as active,
> which could confuse users. Do we want to somehow deal with it?

Yes. As long as the temporary slot is lying unused holding up
resources for more than the specified
replication_slot_inactive_timeout, it is bound to get invalidated.
This keeps behaviour consistent and less-confusing to the users.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  227. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-04-29T04:03:28Z

    On Thu, Apr 25, 2024 at 11:11 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Mon, Apr 22, 2024 at 7:21 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > > Please find the attached v35 patch.
> >
> > The documentation says about both 'active' and 'inactive_since'
> > columns of pg_replication_slots say:
> >
> > ---
> > active bool
> > True if this slot is currently actively being used
> >
> > inactive_since timestamptz
> > The time since the slot has become inactive. NULL if the slot is
> > currently being used. Note that for slots on the standby that are
> > being synced from a primary server (whose synced field is true), the
> > inactive_since indicates the last synchronization (see Section 47.2.3)
> > time.
> > ---
> >
> > When reading the description I thought if 'active' is true,
> > 'inactive_since' is NULL, but it doesn't seem to apply for temporary
> > slots.
>
> Right.
>
> > Since we don't reset the active_pid field of temporary slots
> > when the release, the 'active' is still true in the view but
> > 'inactive_since' is not NULL.
>
> Right. inactive_since is reset whenever the temporary slot is acquired
> again within the same backend that created the temporary slot.
>
> > Do you think we need to mention it in
> > the documentation?
>
> I think that's the reason we dropped "active" from the statement. It
> was earlier "NULL if the slot is currently actively being used.". But,
> per Bertrand's comment
> https://www.postgresql.org/message-id/ZehE2IJcsetSJMHC%40ip-10-97-1-34.eu-west-3.compute.internal
> changed it to ""NULL if the slot is currently being used.".
>
> Temporary slots retain the active = true and active_pid = <pid of the
> backend that created it> even when the slot is not being used until
> the lifetime of the backend process. We haven't tied active or
> active_pid flags to inactive_since, doing so now to represent the
> temporary slot behaviour for active and active_pid will confuse users
> more.
>

This is true and it's probably easy for us to understand as we
developed this feature but the same may not be true for others. I
wonder if we can be explicit about the difference of
active/inactive_since by adding something like the following for
inactive_since: Note that this field is not related to the active flag
as temporary slots can remain active till the session ends even when
they are not being used.

Sawada-San, do you have any suggestions on the wording?

>
 As far as the inactive_since of a slot is concerned, it is set
> to 0 when the slot is being used (acquired) and set to current
> timestamp when the slot is not being used (released).
>
> > As for the timeout-based slot invalidation feature, we could end up
> > invalidating the temporary slots even if they are shown as active,
> > which could confuse users. Do we want to somehow deal with it?
>
> Yes. As long as the temporary slot is lying unused holding up
> resources for more than the specified
> replication_slot_inactive_timeout, it is bound to get invalidated.
> This keeps behaviour consistent and less-confusing to the users.
>

Agreed. We may want to add something in the docs for this to avoid
confusion with the active flag.

-- 
With Regards,
Amit Kapila.

  228. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-06-17T12:25:04Z

    Hi,

On Sat, Apr 13, 2024 at 9:36 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> There was a point raised by Amit
> https://www.postgresql.org/message-id/CAA4eK1K8wqLsMw6j0hE_SFoWAeo3Kw8UNnMfhsWaYDF1GWYQ%2Bg%40mail.gmail.com
> on when to do the XID age based invalidation - whether in checkpointer
> or when vacuum is being run or whenever ComputeXIDHorizons gets called
> or in autovacuum process. For now, I've chosen the design to do these
> new invalidation checks in two places - 1) whenever the slot is
> acquired and the slot acquisition errors out if invalidated, 2) during
> checkpoint. However, I'm open to suggestions on this.

Here are my thoughts on when to do the XID age invalidation. In all
the patches sent so far, the XID age invalidation happens in two
places - one during the slot acquisition, and another during the
checkpoint. As the suggestion is to do it during the vacuum (manual
and auto), so that even if the checkpoint isn't happening in the
database for whatever reasons, a vacuum command or autovacuum can
invalidate the slots whose XID is aged.

An idea is to check for XID age based invalidation for all the slots
in ComputeXidHorizons() before it reads replication_slot_xmin and
replication_slot_catalog_xmin, and obviously before the proc array
lock is acquired. A potential problem with this approach is that the
invalidation check can become too aggressive as XID horizons are
computed from many places.

Another idea is to check for XID age based invalidation for all the
slots in higher levels than ComputeXidHorizons(), for example in
vacuum() which is an entry point for both vacuum command and
autovacuum. This approach seems similar to vacuum_failsafe_age GUC
which checks each relation for the failsafe age before vacuum gets
triggered on it.

Does anyone see any issues or risks with the above two approaches or
have any other ideas? Thoughts?

I attached v40 patches here. I reworded some of the ERROR messages,
and did some code clean-up. Note that I haven't implemented any of the
above approaches yet.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  229. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-06-17T15:09:53Z

    On Mon, Jun 17, 2024 at 05:55:04PM +0530, Bharath Rupireddy wrote:
> Here are my thoughts on when to do the XID age invalidation. In all
> the patches sent so far, the XID age invalidation happens in two
> places - one during the slot acquisition, and another during the
> checkpoint. As the suggestion is to do it during the vacuum (manual
> and auto), so that even if the checkpoint isn't happening in the
> database for whatever reasons, a vacuum command or autovacuum can
> invalidate the slots whose XID is aged.

+1.  IMHO this is a principled choice.  The similar max_slot_wal_keep_size
parameter is considered where it arguably matters most: when we are trying
to remove/recycle WAL segments.  Since this parameter is intended to
prevent the server from running out of space, it makes sense that we'd
apply it at the point where we are trying to free up space.  The proposed
max_slot_xid_age parameter is intended to prevent the server from running
out of transaction IDs, so it follows that we'd apply it at the point where
we reclaim them, which happens to be vacuum.

> An idea is to check for XID age based invalidation for all the slots
> in ComputeXidHorizons() before it reads replication_slot_xmin and
> replication_slot_catalog_xmin, and obviously before the proc array
> lock is acquired. A potential problem with this approach is that the
> invalidation check can become too aggressive as XID horizons are
> computed from many places.
>
> Another idea is to check for XID age based invalidation for all the
> slots in higher levels than ComputeXidHorizons(), for example in
> vacuum() which is an entry point for both vacuum command and
> autovacuum. This approach seems similar to vacuum_failsafe_age GUC
> which checks each relation for the failsafe age before vacuum gets
> triggered on it.

I don't presently have any strong opinion on where this logic should go,
but in general, I think we should only invalidate slots if invalidating
them would allow us to advance the vacuum cutoff.  If the cutoff is held
back by something else, I don't see a point in invalidating slots because
we'll just be breaking replication in return for no additional reclaimed
transaction IDs.

-- 
nathan

  230. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-06-24T06:00:00Z

    Hi,

On Mon, Jun 17, 2024 at 5:55 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Here are my thoughts on when to do the XID age invalidation. In all
> the patches sent so far, the XID age invalidation happens in two
> places - one during the slot acquisition, and another during the
> checkpoint. As the suggestion is to do it during the vacuum (manual
> and auto), so that even if the checkpoint isn't happening in the
> database for whatever reasons, a vacuum command or autovacuum can
> invalidate the slots whose XID is aged.
>
> An idea is to check for XID age based invalidation for all the slots
> in ComputeXidHorizons() before it reads replication_slot_xmin and
> replication_slot_catalog_xmin, and obviously before the proc array
> lock is acquired. A potential problem with this approach is that the
> invalidation check can become too aggressive as XID horizons are
> computed from many places.
>
> Another idea is to check for XID age based invalidation for all the
> slots in higher levels than ComputeXidHorizons(), for example in
> vacuum() which is an entry point for both vacuum command and
> autovacuum. This approach seems similar to vacuum_failsafe_age GUC
> which checks each relation for the failsafe age before vacuum gets
> triggered on it.

I am attaching the patches implementing the idea of invalidating
replication slots during vacuum when current slot xmin limits
(procArray->replication_slot_xmin and
procArray->replication_slot_catalog_xmin) are aged as per the new XID
age GUC. When either of these limits are aged, there must be at least
one replication slot that is aged, because the xmin limits, after all,
are the minimum of xmin or catalog_xmin of all replication slots. In
this approach, the new XID age GUC will help vacuum when needed,
because the current slot xmin limits are recalculated after
invalidating replication slots that are holding xmins for longer than
the age. The code is placed in vacuum() which is common for both
vacuum command and autovacuum, and gets executed only once every
vacuum cycle to not be too aggressive in invalidating.

However, there might be some concerns with this approach like the following:
1) Adding more code to vacuum might not be acceptable
2) What if invalidation of replication slots emits an error, will it
block vacuum forever? Currently, InvalidateObsoleteReplicationSlots()
is also called as part of the checkpoint, and emitting ERRORs from
within is avoided already. Therefore, there is no concern here for
now.
3) What if there are more replication slots to be invalidated, will it
delay the vacuum? If yes, by how much? <<TODO>>
4) Will the invalidation based on just current replication slot xmin
limits suffice irrespective of vacuum cutoffs? IOW, if the replication
slots are invalidated but vacuum isn't going to do any work because
vacuum cutoffs are not yet met? Is the invalidation work wasteful
here?
5) Is it okay to take just one more time the proc array lock to get
current replication slot xmin limits via
ProcArrayGetReplicationSlotXmin() once every vacuum cycle? <<TODO>>
6) Vacuum command can't be run on the standby in recovery. So, to help
invalidate replication slots on the standby, I have for now let the
checkpointer also do the XID age based invalidation. I know
invalidating both in checkpointer and vacuum may not be a great idea,
but I'm open to thoughts.

Following are some of the alternative approaches which IMHO don't help
vacuum when needed:
a) Let the checkpointer do the XID age based invalidation, and call it
out in the documentation that if the checkpoint doesn't happen, the
new GUC doesn't help even if the vacuum is run. This has been the
approach until v40 patch.
b) Checkpointer and/or other backends add an autovacuum work item via
AutoVacuumRequestWork(), and autovacuum when it gets to it will
invalidate the replication slots. But, what to do for the vacuum
command here?

Please find the attached v41 patches implementing the idea of vacuum
doing the invalidation.

Thoughts?

Thanks to Sawada-san for a detailed off-list discussion.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  231. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2024-07-09T22:01:23Z

    On Mon, Jun 24, 2024 at 11:30:00AM +0530, Bharath Rupireddy wrote:
> 6) Vacuum command can't be run on the standby in recovery. So, to help
> invalidate replication slots on the standby, I have for now let the
> checkpointer also do the XID age based invalidation. I know
> invalidating both in checkpointer and vacuum may not be a great idea,
> but I'm open to thoughts.

Hm.  I hadn't considered this angle.

> a) Let the checkpointer do the XID age based invalidation, and call it
> out in the documentation that if the checkpoint doesn't happen, the
> new GUC doesn't help even if the vacuum is run. This has been the
> approach until v40 patch.

My first reaction is that this is probably okay.  I guess you might run
into problems if you set max_slot_xid_age to 2B and checkpoint_timeout to 1
day, but even in that case your transaction ID usage rate would need to be
pretty high for wraparound to occur.

-- 
nathan

  232. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Ajin Cherian <itsajin@gmail.com> — 2024-08-12T12:17:55Z

    On Mon, Jun 24, 2024 at 4:01 PM Bharath Rupireddy <
bharath.rupireddyforpostgres@gmail.com> wrote:

> Hi,
>
> On Mon, Jun 17, 2024 at 5:55 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Here are my thoughts on when to do the XID age invalidation. In all
> > the patches sent so far, the XID age invalidation happens in two
> > places - one during the slot acquisition, and another during the
> > checkpoint. As the suggestion is to do it during the vacuum (manual
> > and auto), so that even if the checkpoint isn't happening in the
> > database for whatever reasons, a vacuum command or autovacuum can
> > invalidate the slots whose XID is aged.
> >
> > An idea is to check for XID age based invalidation for all the slots
> > in ComputeXidHorizons() before it reads replication_slot_xmin and
> > replication_slot_catalog_xmin, and obviously before the proc array
> > lock is acquired. A potential problem with this approach is that the
> > invalidation check can become too aggressive as XID horizons are
> > computed from many places.
> >
> > Another idea is to check for XID age based invalidation for all the
> > slots in higher levels than ComputeXidHorizons(), for example in
> > vacuum() which is an entry point for both vacuum command and
> > autovacuum. This approach seems similar to vacuum_failsafe_age GUC
> > which checks each relation for the failsafe age before vacuum gets
> > triggered on it.
>
> I am attaching the patches implementing the idea of invalidating
> replication slots during vacuum when current slot xmin limits
> (procArray->replication_slot_xmin and
> procArray->replication_slot_catalog_xmin) are aged as per the new XID
> age GUC. When either of these limits are aged, there must be at least
> one replication slot that is aged, because the xmin limits, after all,
> are the minimum of xmin or catalog_xmin of all replication slots. In
> this approach, the new XID age GUC will help vacuum when needed,
> because the current slot xmin limits are recalculated after
> invalidating replication slots that are holding xmins for longer than
> the age. The code is placed in vacuum() which is common for both
> vacuum command and autovacuum, and gets executed only once every
> vacuum cycle to not be too aggressive in invalidating.
>
> However, there might be some concerns with this approach like the
> following:
> 1) Adding more code to vacuum might not be acceptable
> 2) What if invalidation of replication slots emits an error, will it
> block vacuum forever? Currently, InvalidateObsoleteReplicationSlots()
> is also called as part of the checkpoint, and emitting ERRORs from
> within is avoided already. Therefore, there is no concern here for
> now.
> 3) What if there are more replication slots to be invalidated, will it
> delay the vacuum? If yes, by how much? <<TODO>>
> 4) Will the invalidation based on just current replication slot xmin
> limits suffice irrespective of vacuum cutoffs? IOW, if the replication
> slots are invalidated but vacuum isn't going to do any work because
> vacuum cutoffs are not yet met? Is the invalidation work wasteful
> here?
> 5) Is it okay to take just one more time the proc array lock to get
> current replication slot xmin limits via
> ProcArrayGetReplicationSlotXmin() once every vacuum cycle? <<TODO>>
> 6) Vacuum command can't be run on the standby in recovery. So, to help
> invalidate replication slots on the standby, I have for now let the
> checkpointer also do the XID age based invalidation. I know
> invalidating both in checkpointer and vacuum may not be a great idea,
> but I'm open to thoughts.
>
> Following are some of the alternative approaches which IMHO don't help
> vacuum when needed:
> a) Let the checkpointer do the XID age based invalidation, and call it
> out in the documentation that if the checkpoint doesn't happen, the
> new GUC doesn't help even if the vacuum is run. This has been the
> approach until v40 patch.
> b) Checkpointer and/or other backends add an autovacuum work item via
> AutoVacuumRequestWork(), and autovacuum when it gets to it will
> invalidate the replication slots. But, what to do for the vacuum
> command here?
>
> Please find the attached v41 patches implementing the idea of vacuum
> doing the invalidation.
>
> Thoughts?
>
> Thanks to Sawada-san for a detailed off-list discussion.
>

The patch no longer applies on HEAD, please rebase.

regards,
Ajin Cherian
Fujitsu Australia

  233. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-08-12T21:32:45Z

    On Tue, Jul 9, 2024 at 3:01 PM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Mon, Jun 24, 2024 at 11:30:00AM +0530, Bharath Rupireddy wrote:
> > 6) Vacuum command can't be run on the standby in recovery. So, to help
> > invalidate replication slots on the standby, I have for now let the
> > checkpointer also do the XID age based invalidation. I know
> > invalidating both in checkpointer and vacuum may not be a great idea,
> > but I'm open to thoughts.
>
> Hm.  I hadn't considered this angle.

Another idea would be to let the startup process do slot invalidation
when replaying a RUNNING_XACTS record. Since a RUNNING_XACTS record
has the latest XID on the primary, I think the startup process can
compare it to the slot-xmin, and invalidate slots which are older than
the age limit.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  234. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Ajin Cherian <itsajin@gmail.com> — 2024-08-14T03:50:38Z

    On Mon, Jun 24, 2024 at 4:01 PM Bharath Rupireddy <
bharath.rupireddyforpostgres@gmail.com> wrote:

> Hi,
>
> On Mon, Jun 17, 2024 at 5:55 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Please find the attached v41 patches implementing the idea of vacuum
> doing the invalidation.
>
> Thoughts?
>
>
>

Some minor comments on the patch:
1.
+ /*
+ * Release the lock if it's not yet to keep the cleanup path on
+ * error happy.
+ */

I suggest rephrasing to: " "Release the lock if it hasn't been already to
ensure smooth cleanup on error."


2.

elog(DEBUG1, "performing replication slot invalidation");

Probably change it to "performing replication slot invalidation checks" as
we might not actually invalidate any slot here.

3.

In CheckPointReplicationSlots()

+ invalidated =
InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
+ if (invalidated)
+ {
+ /*
+ * If any slots have been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }

Is this calculation of resource limits really required here when the same
is already done inside InvalidateObsoleteReplicationSlots()


regards,
Ajin Cherian
Fujitsu Australia

  235. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-08-26T06:14:05Z

    On Wed, Aug 14, 2024 at 9:20 AM Ajin Cherian <itsajin@gmail.com> wrote:
>
> Some minor comments on the patch:

Thanks for reviewing.

> 1.
> + /*
> + * Release the lock if it's not yet to keep the cleanup path on
> + * error happy.
> + */
>
> I suggest rephrasing to: " "Release the lock if it hasn't been already to ensure smooth cleanup on error."

Changed.

> 2.
>
> elog(DEBUG1, "performing replication slot invalidation");
>
> Probably change it to "performing replication slot invalidation checks" as we might not actually invalidate any slot here.

Changed.

> 3.
>
> + ReplicationSlotsComputeRequiredXmin(false);
> + ReplicationSlotsComputeRequiredLSN();
> + }
>
> Is this calculation of resource limits really required here when the same is already done inside InvalidateObsoleteReplicationSlots()

Nice catch. Removed.

Please find the attached v42 patches.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  236. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-08-26T11:05:02Z

    On Mon, Aug 26, 2024 at 11:44 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>

Few comments on 0001:
1.
@@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
remote_dbid)
     " name slot \"%s\" already exists on the standby",
     remote_slot->name));

+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
  /*
  * The slot has been synchronized before.

I was wondering why you have added this new check as part of this
patch. If you see the following comments in the related code, you will
know why we haven't done this previously.

/*
* The slot has been synchronized before.
*
* It is important to acquire the slot here before checking
* invalidation. If we don't acquire the slot first, there could be a
* race condition that the local slot could be invalidated just after
* checking the 'invalidated' flag here and we could end up
* overwriting 'invalidated' flag to remote_slot's value. See
* InvalidatePossiblyObsoleteSlot() where it invalidates slot directly
* if the slot is not acquired by other processes.
*
* XXX: If it ever turns out that slot acquire/release is costly for
* cases when none of the slot properties is changed then we can do a
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
ReplicationSlotAcquire(remote_slot->name, true);

We need some modifications in these comments if you want to add a
pre-check here.

2.
@@ -1907,6 +2033,31 @@ CheckPointReplicationSlots(bool is_shutdown)
  SaveSlotToPath(s, path, LOG);
  }
  LWLockRelease(ReplicationSlotAllocationLock);
+
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * Note that we will make another pass over replication slots for
+ * invalidations to keep the code simple. The assumption here is that the
+ * traversal over replication slots isn't that costly even with hundreds
+ * of replication slots. If it ever turns out that this assumption is
+ * wrong, we might have to put the invalidation check logic in the above
+ * loop, for that we might have to do the following:
+ *
+ * - Acqure ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated slot.
+ *
+ * XXX: Should we move inactive_timeout inavalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+    0, InvalidOid, InvalidTransactionId);

Why do we want to call this for shutdown case (when is_shutdown is
true)? I understand trying to invalidate slots during regular
checkpoint but not sure if we need it at the time of shutdown. The
other point is can we try to check the performance impact with 100s of
slots as mentioned in the code comments?

-- 
With Regards,
Amit Kapila.

  237. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-08-29T06:01:09Z

    Hi,

Thanks for looking into this.

On Mon, Aug 26, 2024 at 4:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Few comments on 0001:
> 1.
> @@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
>
> + /*
> + * Skip the sync if the local slot is already invalidated. We do this
> + * beforehand to avoid slot acquire and release.
> + */
>
> I was wondering why you have added this new check as part of this
> patch. If you see the following comments in the related code, you will
> know why we haven't done this previously.

Removed. Can deal with optimization separately.

> 2.
> + */
> + InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
> +    0, InvalidOid, InvalidTransactionId);
>
> Why do we want to call this for shutdown case (when is_shutdown is
> true)? I understand trying to invalidate slots during regular
> checkpoint but not sure if we need it at the time of shutdown.

Changed it to invalidate only for non-shutdown checkpoints.
inactive_timeout invalidation isn't critical for shutdown unlike
wal_removed which can help shutdown by freeing up some disk space.

> The
> other point is can we try to check the performance impact with 100s of
> slots as mentioned in the code comments?

I first checked how much does the wal_removed invalidation check add to the
checkpoint (see 2nd and 3rd column). I then checked how much
inactive_timeout invalidation check adds to the checkpoint (see 4th
column), it is not more than wal_remove invalidation check. I then checked
how much the wal_removed invalidation check adds for replication slots that
have already been invalidated due to inactive_timeout (see 5th column),
looks like not much.

| # of slots | HEAD (no invalidation) ms | HEAD (wal_removed) ms | PATCHED
(inactive_timeout) ms | PATCHED (inactive_timeout+wal_removed) ms |
|------------|----------------------------|-----------------------|-------------------------------|------------------------------------------|
| 100        | 18.591                     | 370.586               | 359.299
                      | 373.882                                  |
| 1000       | 15.722                     | 4834.901              |
5081.751                      | 5072.128                                 |
| 10000      | 19.261                     | 59801.062             |
61270.406                     | 60270.099                                |

Having said that, I'm okay to implement the optimization specified.
Thoughts?

+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.

Please see the attached v43 patches addressing the above review comments.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  238. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-08-30T02:43:10Z

    Hi, here are some review comments for patch v43-0001.

======
Commit message

1.
... introduces a GUC allowing users set inactive timeout.

~

1a. You should give the name of the new GUC in the commit message.

1b. /set/to set/

======
doc/src/sgml/config.sgml

GUC "replication_slot_inactive_timeout"

2.
Invalidates replication slots that are inactive for longer than
specified amount of time

nit - suggest use similar wording as the prior GUC (wal_sender_timeout):
Invalidate replication slots that are inactive for longer than this
amount of time.

~

3.
This invalidation check happens either when the slot is acquired for
use or during a checkpoint. The time since the slot has become
inactive is known from its inactive_since value using which the
timeout is measured.

nit - the wording is too complicated. suggest:
The timeout check occurs when the slot is next acquired for use, or
during a checkpoint. The slot's 'inactive_since' field value is when
the slot became inactive.

~

4.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby that are being synced from a
primary server (whose synced field is true).

nit - that word "whose" seems ambiguous. suggest:
(e.g. the standby slot has 'synced' field true).

======
doc/src/sgml/system-views.sgml

5.
inactive_timeout means that the slot has been inactive for the
duration specified by replication_slot_inactive_timeout parameter.

nit - suggestion ("longer than"):
... the slot has been inactive for longer than the duration specified
by the replication_slot_inactive_timeout parameter.

======
src/backend/replication/slot.c

6.
 /* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT

IMO this #define belongs in the slot.h, immediately below where the
enum is defined.

~~~

7. ReplicationSlotAcquire:

I had a fundamental question about this logic.

IIUC the purpose of the patch was to invalidate replication slots that
have been inactive for too long.

So, it makes sense to me that some periodic processing (e.g.
CheckPointReplicationSlots) might do a sweep over all the slots, and
invalidate the too-long-inactive ones that it finds.

OTOH, it seemed quite strange to me that the patch logic is also
detecting and invalidating inactive slots during the
ReplicationSlotAcquire function. This is kind of saying "ERROR -
sorry, because this was inactive for too long you can't have it" at
the very moment that you wanted to use it again! IIUC such a slot
would be invalidated by the function InvalidatePossiblyObsoleteSlot(),
but therein lies my doubt -- how can the slot be considered as
"obsolete" when we are in the very act of trying to acquire/use it?

I guess it might be argued this is not so different to the scenario of
attempting to acquire a slot that had been invalidated momentarily
before during checkpoint processing. But, somehow that scenario seems
more like bad luck to me, versus ReplicationSlotAcquire() deliberately
invalidating something we *know* is wanted.

~

8.
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive
since %s for more than %d seconds specified by
\"replication_slot_inactive_timeout\".",
+    timestamptz_to_str(s->inactive_since),
+    replication_slot_inactive_timeout)));

nit - IMO the info should be split into errdetail + errhint. Like this:
errdetail("The slot became invalid because it was inactive since %s,
which is more than %d seconds ago."...)
errhint("You might need to increase \"%s\".",
"replication_slot_inactive_timeout")

~~~

9. ReportSlotInvalidation

+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for more than %d seconds
specified by \"replication_slot_inactive_timeout\"."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+ break;

IMO this error in ReportSlotInvalidation() should be the same as the
other one from ReplicationSlotAcquire(), which I suggested above
(comment #8) should include a hint. Also, including a hint here will
make this new message consistent with the other errhint (for
"max_slot_wal_keep_size") that is already in this function.

~~~

10. InvalidatePossiblyObsoleteSlot

+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))

10a. Everything here is && so this has some redundant parentheses.

10b. Actually, IMO this complicated condition is overkill. Won't it be
better to just unconditionally assign
now = GetCurrentTimestamp(); here?

~

11.
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they don't
+ * perform logical decoding to produce the changes.

nit - tweaked punctuation

~

12.
+ * If the slot can be acquired, do so or if the slot is already ours,
+ * then mark it invalidated.  Otherwise we'll signal the owning
+ * process, below, and retry.

nit - tidied this comment. Suggestion:
If the slot can be acquired, do so and mark it as invalidated. If the
slot is already ours, mark it as invalidated. Otherwise, we'll signal
the owning process below and retry.

~

13.
+ if (active_pid == 0 ||
+ (MyReplicationSlot != NULL &&
+ MyReplicationSlot == s &&
+ active_pid == MyProcPid))

You are already checking MyReplicationSlot == s here, so that extra
check for MyReplicationSlot != NULL is redundant, isn't it?

~~~

14. CheckPointReplicationSlots

 /*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate slots during
+ * non-shutdown checkpoint.
  *
  * It is convenient to flush dirty replication slots at the time of checkpoint.
  * Additionally, in case of a shutdown checkpoint, we also identify the slots

nit - /Also, invalidate slots/Also, invalidate obsolete slots/

======
src/backend/utils/misc/guc_tables.c

15.
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),

nit - that is maybe a bit misleading because IIUC there is no real
"waiting" happening anywhere. Suggest:
Sets the amount of time a replication slot can remain inactive before
it will be invalidated.

======

Please take a look at the attached top-up patches. These include
changes for many of the nits above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  239. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-08-31T08:15:39Z

    Hi,

Thanks for looking into this.

On Fri, Aug 30, 2024 at 8:13 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> ======
> Commit message
>
> 1.
> ... introduces a GUC allowing users set inactive timeout.
>
> ~
>
> 1a. You should give the name of the new GUC in the commit message.

Modified.

> 1b. /set/to set/

Reworded the commit message.

> ======
> doc/src/sgml/config.sgml
>
> GUC "replication_slot_inactive_timeout"
>
> 2.
> Invalidates replication slots that are inactive for longer than
> specified amount of time
>
> nit - suggest use similar wording as the prior GUC (wal_sender_timeout):
> Invalidate replication slots that are inactive for longer than this
> amount of time.

Modified.

> 3.
> This invalidation check happens either when the slot is acquired for
> use or during a checkpoint. The time since the slot has become
> inactive is known from its inactive_since value using which the
> timeout is measured.
>
> nit - the wording is too complicated. suggest:
> The timeout check occurs when the slot is next acquired for use, or
> during a checkpoint. The slot's 'inactive_since' field value is when
> the slot became inactive.


> 4.
> Note that the inactive timeout invalidation mechanism is not
> applicable for slots on the standby that are being synced from a
> primary server (whose synced field is true).
>
> nit - that word "whose" seems ambiguous. suggest:
> (e.g. the standby slot has 'synced' field true).

Reworded.

> ======
> doc/src/sgml/system-views.sgml
>
> 5.
> inactive_timeout means that the slot has been inactive for the
> duration specified by replication_slot_inactive_timeout parameter.
>
> nit - suggestion ("longer than"):
> ... the slot has been inactive for longer than the duration specified
> by the replication_slot_inactive_timeout parameter.

Modified.

> ======
> src/backend/replication/slot.c
>
> 6.
>  /* Maximum number of invalidation causes */
> -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
> +#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
>
> IMO this #define belongs in the slot.h, immediately below where the
> enum is defined.

Please check the commit that introduced it -
https://www.postgresql.org/message-id/ZdU3CHqza9XJw4P-%40paquier.xyz.
It is kept in the file where it's used.

> 7. ReplicationSlotAcquire:
>
> I had a fundamental question about this logic.
>
> IIUC the purpose of the patch was to invalidate replication slots that
> have been inactive for too long.
>
> So, it makes sense to me that some periodic processing (e.g.
> CheckPointReplicationSlots) might do a sweep over all the slots, and
> invalidate the too-long-inactive ones that it finds.
>
> OTOH, it seemed quite strange to me that the patch logic is also
> detecting and invalidating inactive slots during the
> ReplicationSlotAcquire function. This is kind of saying "ERROR -
> sorry, because this was inactive for too long you can't have it" at
> the very moment that you wanted to use it again! IIUC such a slot
> would be invalidated by the function InvalidatePossiblyObsoleteSlot(),
> but therein lies my doubt -- how can the slot be considered as
> "obsolete" when we are in the very act of trying to acquire/use it?
>
> I guess it might be argued this is not so different to the scenario of
> attempting to acquire a slot that had been invalidated momentarily
> before during checkpoint processing. But, somehow that scenario seems
> more like bad luck to me, versus ReplicationSlotAcquire() deliberately
> invalidating something we *know* is wanted.

Hm. TBH, there's no real reason for invalidating the slot in
ReplicationSlotAcquire(). My thinking back then was to take this
opportunity to do some work. I agree to leave the invalidation work to
the checkpointer. However, I still think ReplicationSlotAcquire()
should error out if the slot has already been invalidated similar to
"can no longer get changes from replication slot \"%s\" for
wal_removed.

> 8.
> + ereport(ERROR,
> + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("can no longer get changes from replication slot \"%s\"",
> + NameStr(s->data.name)),
> + errdetail("This slot has been invalidated because it was inactive
> since %s for more than %d seconds specified by
> \"replication_slot_inactive_timeout\".",
> +    timestamptz_to_str(s->inactive_since),
> +    replication_slot_inactive_timeout)));
>
> nit - IMO the info should be split into errdetail + errhint. Like this:
> errdetail("The slot became invalid because it was inactive since %s,
> which is more than %d seconds ago."...)
> errhint("You might need to increase \"%s\".",
> "replication_slot_inactive_timeout")

"invalid" is being covered by errmsg "invalidating obsolete
replication slot", so no need to duplicate it in errdetail.

> 9. ReportSlotInvalidation
>
> + appendStringInfo(&err_detail,
> + _("The slot has been inactive since %s for more than %d seconds
> specified by \"replication_slot_inactive_timeout\"."),
> + timestamptz_to_str(inactive_since),
> + replication_slot_inactive_timeout);
> + break;
>
> IMO this error in ReportSlotInvalidation() should be the same as the
> other one from ReplicationSlotAcquire(), which I suggested above
> (comment #8) should include a hint. Also, including a hint here will
> make this new message consistent with the other errhint (for
> "max_slot_wal_keep_size") that is already in this function.

Not exactly the same but similar. Because ReportSlotInvalidation()
errmsg has an "invalidating" component, whereas errmsg in
ReplicationSlotAcquire doesn't. Please check latest wordings.

> 10. InvalidatePossiblyObsoleteSlot
>
> + if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
> + (replication_slot_inactive_timeout > 0 &&
> + s->inactive_since > 0 &&
> + !(RecoveryInProgress() && s->data.synced)))
>
> 10a. Everything here is && so this has some redundant parentheses.

Removed.

> 10b. Actually, IMO this complicated condition is overkill. Won't it be
> better to just unconditionally assign
> now = GetCurrentTimestamp(); here?

GetCurrentTimestamp() can get costlier on certain platforms. I think
the fields checking in the condition are pretty straight forward -
e.g. !RecoveryInProgress() server not in recovery, !s->data.synced
slot is not being synced and so on. Added a macro
IsInactiveTimeoutSlotInvalidationApplicable() for better readability
in two places.

> 11.
> + * Note that we don't invalidate synced slots because,
> + * they are typically considered not active as they don't
> + * perform logical decoding to produce the changes.
>
> nit - tweaked punctuation

Used the consistent wording in the commit message, docs and code comments.

> 12.
> + * If the slot can be acquired, do so or if the slot is already ours,
> + * then mark it invalidated.  Otherwise we'll signal the owning
> + * process, below, and retry.
>
> nit - tidied this comment. Suggestion:
> If the slot can be acquired, do so and mark it as invalidated. If the
> slot is already ours, mark it as invalidated. Otherwise, we'll signal
> the owning process below and retry.

Modified.

> 13.
> + if (active_pid == 0 ||
> + (MyReplicationSlot != NULL &&
> + MyReplicationSlot == s &&
> + active_pid == MyProcPid))
>
> You are already checking MyReplicationSlot == s here, so that extra
> check for MyReplicationSlot != NULL is redundant, isn't it?

Removed.

> 14. CheckPointReplicationSlots
>
>  /*
> - * Flush all replication slots to disk.
> + * Flush all replication slots to disk. Also, invalidate slots during
> + * non-shutdown checkpoint.
>   *
>   * It is convenient to flush dirty replication slots at the time of checkpoint.
>   * Additionally, in case of a shutdown checkpoint, we also identify the slots
>
> nit - /Also, invalidate slots/Also, invalidate obsolete slots/

Modified.

> 15.
> + {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
> + gettext_noop("Sets the amount of time to wait before invalidating an "
> + "inactive replication slot."),
>
> nit - that is maybe a bit misleading because IIUC there is no real
> "waiting" happening anywhere. Suggest:
> Sets the amount of time a replication slot can remain inactive before
> it will be invalidated.

Modified.

Please find the attached v44 patch with the above changes. I will
include the 0002 xid_age based invalidation patch later.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  240. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-02T06:55:35Z

    On Thu, Aug 29, 2024 at 11:31 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Thanks for looking into this.
>
> On Mon, Aug 26, 2024 at 4:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Few comments on 0001:
> > 1.
> > @@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
> >
> > + /*
> > + * Skip the sync if the local slot is already invalidated. We do this
> > + * beforehand to avoid slot acquire and release.
> > + */
> >
> > I was wondering why you have added this new check as part of this
> > patch. If you see the following comments in the related code, you will
> > know why we haven't done this previously.
>
> Removed. Can deal with optimization separately.
>
> > 2.
> > + */
> > + InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
> > +    0, InvalidOid, InvalidTransactionId);
> >
> > Why do we want to call this for shutdown case (when is_shutdown is
> > true)? I understand trying to invalidate slots during regular
> > checkpoint but not sure if we need it at the time of shutdown.
>
> Changed it to invalidate only for non-shutdown checkpoints. inactive_timeout invalidation isn't critical for shutdown unlike wal_removed which can help shutdown by freeing up some disk space.
>
> > The
> > other point is can we try to check the performance impact with 100s of
> > slots as mentioned in the code comments?
>
> I first checked how much does the wal_removed invalidation check add to the checkpoint (see 2nd and 3rd column). I then checked how much inactive_timeout invalidation check adds to the checkpoint (see 4th column), it is not more than wal_remove invalidation check. I then checked how much the wal_removed invalidation check adds for replication slots that have already been invalidated due to inactive_timeout (see 5th column), looks like not much.
>
> | # of slots | HEAD (no invalidation) ms | HEAD (wal_removed) ms | PATCHED (inactive_timeout) ms | PATCHED (inactive_timeout+wal_removed) ms |
> |------------|----------------------------|-----------------------|-------------------------------|------------------------------------------|
> | 100        | 18.591                     | 370.586               | 359.299                       | 373.882                                  |
> | 1000       | 15.722                     | 4834.901              | 5081.751                      | 5072.128                                 |
> | 10000      | 19.261                     | 59801.062             | 61270.406                     | 60270.099                                |
>
> Having said that, I'm okay to implement the optimization specified. Thoughts?
>

The other possibility is to try invalidating due to timeout along with
wal_removed case during checkpoint. The idea is that if the slot can
be invalidated due to WAL then fine, otherwise check if it can be
invalidated due to timeout. This can avoid looping the slots and doing
similar work multiple times during the checkpoint.

-- 
With Regards,
Amit Kapila.

  241. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-09-02T08:06:44Z

    Hi. Thanks for addressing my previous review comments.

Here are some review comments for v44-0001.

======
Commit message.

1.
Because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.

~

This sentence is bad grammar. The docs have the same wording, so
please see my doc review comment #4 suggestion below.

======
doc/src/sgml/config.sgml

2.
+       <para>
+        Invalidates replication slots that are inactive for longer than
+        specified amount of time. If this value is specified without units,
+        it is taken as seconds. A value of zero (which is default) disables
+        the timeout mechanism. This parameter can only be set in
+        the <filename>postgresql.conf</filename> file or on the server
+        command line.
+       </para>
+

nit - This is OK as-is, but OTOH why not make the wording consistent
with the previous GUC description? (e.g. see my v43 [1] #2 review
comment)

~~~

3.
+       <para>
+        This invalidation check happens either when the slot is acquired
+        for use or during checkpoint. The time since the slot has become
+        inactive is known from its
+        <structfield>inactive_since</structfield> value using which the
+        timeout is measured.
+       </para>
+

I felt this is slightly misleading because slot acquiring has nothing
to do with setting the slot invalidation anymore. Furthermore, the 2nd
sentence is bad grammar.

nit - IMO something simple like the following rewording can address
both of those points:

Slot invalidation due to inactivity timeout occurs during checkpoint.
The duration of slot inactivity is calculated using the slot's
<structfield>inactive_since</structfield> field value.

~

4.
+        Because such synced slots are typically considered not active
+        (for them to be later considered as inactive) as they don't perform
+        logical decoding to produce the changes.

That sentence has bad grammar.

nit – suggest a much simpler replacement:
Synced slots are always considered to be inactive because they don't
perform logical decoding to produce changes.

======
src/backend/replication/slot.c

5.
+#define IsInactiveTimeoutSlotInvalidationApplicable(s) \
+ (replication_slot_inactive_timeout > 0 && \
+ s->inactive_since > 0 && \
+ !RecoveryInProgress() && \
+ !s->data.synced)
+

5a.
I felt this would be better implemented as an inline function. Then it
can be commented on properly to explain the parts of the condition.
e.g. the large comment currently in InvalidatePossiblyObsoleteSlot()
would be more appropriate in this function.

~

5b.
The name is very long. Can't it be something shorter/simpler like:
'IsSlotATimeoutCandidate()'

~~~

6. ReplicationSlotAcquire

-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+    bool check_for_invalidation)

nit - Previously this new parameter really did mean to "check" for
[and set the slot] invalidation. But now I suggest renaming it to
'error_if_invalid' to properly reflect the new usage. And also in the
slot.h.

~

7.
+ /*
+ * Error out if the slot has been invalidated previously. Because there's
+ * no use in acquiring the invalidated slot.
+ */

nit - The comment is contrary to the code. If there was no reason to
skip this error, then you would not have the new parameter allowing
you to skip this error. I suggest just repeating the same comment as
in the function header.

~~~

8. ReportSlotInvalidation

nit - Added some blank lines for consistency.

~~~

9. InvalidatePossiblyObsoleteSlot

+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the
+ * server is in recovery mode or the slot on standby is
+ * currently being synced from the primary.
+ *
+ * Note that the inactive timeout invalidation mechanism
+ * is not applicable for slots on the standby server that
+ * are being synced from primary server. Because such
+ * synced slots are typically considered not active (for
+ * them to be later considered as inactive) as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (!IsInactiveTimeoutSlotInvalidationApplicable(s))
+ break;

9a.
Consistency is good (commit message, docs and code comments for this),
but the added sentence has bad grammar. Please see the docs review
comment #4 above for some alternate phrasing.

~

9b.
Now that this logic is moved into a macro (I suggested it should be an
inline function) IMO this comment does not belong here anymore because
it is commenting code that you cannot see. Instead, this comment (or
something like it) should be as comments within the new function.

======
src/include/replication/slot.h

10.
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+    bool check_for_invalidation);

Change the new param name as described in the earlier review comment.

======
src/test/recovery/t/050_invalidate_slots.pl

~~~

Please refer to the attached file which implements some of the nits
mentioned above.

======
[1] v43 review -
https://www.postgresql.org/message-id/CAHut%2BPuFzCHPCiZbpoQX59kgZbebuWT0gR0O7rOe4t_sdYu%3DOA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  242. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-02T09:50:06Z

    On Sat, Aug 31, 2024 at 1:45 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Please find the attached v44 patch with the above changes. I will
> include the 0002 xid_age based invalidation patch later.
>

It is better to get the 0001 reviewed and committed first. We can
discuss about 0002 afterwards as 0001 is in itself a complete and
separate patch that can be committed.

-- 
With Regards,
Amit Kapila.

  243. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-09-03T06:55:58Z

    Hi, my previous review posts did not cover the test code.

Here are my review comments for the v44-0001 test code

======
TEST CASE #1

1.
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+  or die
+  "Timed out while waiting for lsub1_sync_slot invalidation to be
synced on standby";
+

Is that comment correct? IIUC the synced slot should *already* be
invalidated from the primary, so here we are not really "waiting" for
it to be invalidated; Instead, we are just "confirming" that the
synchronized slot is already invalidated with the correct reason as
expected.

~~~

2.
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned lsub1_sync_slot has not been invalidated on the standby'
+);
+

This test case seemed bogus, for a couple of reasons:

2a. IIUC this 'lsub1_sync_slot' is the same one that is already
invalid (from the primary), so nobody should be surprised that an
already invalid slot doesn't get flagged as invalid again. i.e.
Shouldn't your test scenario here be done using a valid synced slot?

2b. AFAICT it was only moments above this CHECKPOINT where you
assigned the standby inactivity timeout to 2s. So even if there was
some bug invalidating synced slots I don't think you gave it enough
time to happen -- e.g. I doubt 2s has elapsed yet.

~

3.
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);

This seems a bit tricky. Both these (the stop and the wait) seem to
belong together, so I think maybe a single bigger explanatory comment
covering both parts would help for understanding.

======
TEST CASE #2

4.
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);

IIUC, this is just like comment #3 above. Both these (the stop and the
wait) seem to belong together, so I think maybe a single bigger
explanatory comment covering both parts would help for understanding.

~~~

5.
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================


IMO the rest of the comment after "Testcase end" isn't very useful.

======
sub wait_for_slot_invalidation

6.
+sub wait_for_slot_invalidation
+{

An explanatory header comment for this subroutine would be helpful.

~~~

7.
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+   or die
+   "Timed out while waiting for slot $slot_name to become inactive on
node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+   or die
+   "Timed out while waiting for info of slot $slot_name to be updated
on node $name";
+

Why are there are 2 separate poll_query_until's here? Can't those be
combined into just one?

~~~

8.
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+

Maybe this special sleep to prevent too many CHECKPOINTs should be
moved to be inside the other subroutine, which is actually doing those
CHECKPOINTs.

~~~

9.
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+   or die
+   "Timed out while waiting for inactive slot $slot_name to be
invalidated on node $name";
+

The comment seems misleading. IIUC you are not "waiting" for the
invalidation here, because it is the other subroutine doing the
waiting for the invalidation message in the logs. Instead, here I
think you are just confirming the 'invalidation_reason' got set
correctly. The comment should say what it is really doing.

======
sub check_for_slot_invalidation_in_server_log

10.
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{

I think the main function of this subroutine is the CHECKPOINT and the
waiting for the server log to say invalidation happened. It is doing a
loop of a) CHECKPOINT then b) inspecting the server log for the slot
invalidation, and c) waiting for a bit. Repeat 10 times.

A comment describing the logic for this subroutine would be helpful.

The most important side-effect of this function is the CHECKPOINT
because without that nothing will ever get invalidated due to
inactivity, but this key point is not obvious from the subroutine
name.

IMO it would be better to name this differently to reflect what it is
really doing:
e.g. "CHECKPOINT_and_wait_for_slot_invalidation_in_server_log"

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  244. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-03T09:31:06Z

    On Sat, Aug 31, 2024 at 1:45 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
>
> Please find the attached v44 patch with the above changes. I will
> include the 0002 xid_age based invalidation patch later.
>

Thanks for the patch Bharath. My review and testing is WIP, but please
find few comments and queries:

1)
I see that ReplicationSlotAlter() will error out if the slot is
invalidated due to timeout. I have not tested it myself, but do you
know if  slot-alter errors out for other invalidation causes as well?
Just wanted to confirm that the behaviour is consistent for all
invalidation causes.

2)
When a slot is invalidated, and we try to use that slot, it gives this msg:

ERROR:  can no longer get changes from replication slot "mysubnew1_2"
DETAIL:  The slot became invalid because it was inactive since
2024-09-03 14:23:34.094067+05:30, which is more than 600 seconds ago.
HINT:  You might need to increase "replication_slot_inactive_timeout.".

Isn't HINT misleading? Even if we increase it now, the slot can not be
reused again.


3)
When the slot is invalidated, the' inactive_since' still keeps on
changing when there is a subscriber trying to start replication
continuously. I think ReplicationSlotAcquire() keeps on failing and
thus Release keeps on setting it again and again. Shouldn't we stop
setting/chnaging  'inactive_since' once the slot is invalidated
already, otherwise it will be misleading.

postgres=# select failover,synced,inactive_since,invalidation_reason
from pg_replication_slots;

 failover | synced |          inactive_since          | invalidation_reason
----------+--------+----------------------------------+---------------------
 t        | f      | 2024-09-03 14:23:.. | inactive_timeout

after sometime:
 failover | synced |          inactive_since          | invalidation_reason
----------+--------+----------------------------------+---------------------
 t        | f      | 2024-09-03 14:26:..| inactive_timeout


4)
src/sgml/config.sgml:

4a)
+ A value of zero (which is default) disables the timeout mechanism.

Better will be:
A value of zero (which is default) disables the inactive timeout
invalidation mechanism .
or
A value of zero (which is default) disables the slot invalidation due
to the inactive timeout mechanism.

i.e. rephrase to indicate that invalidation is disabled.

4b)
'synced' and inactive_since should point to pg_replication_slots:

example:
<link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>

5)
src/sgml/system-views.sgml:
+ ..the slot has been inactive for longer than the duration specified
by replication_slot_inactive_timeout parameter.

Better to have:
..the slot has been inactive for a time longer than the duration
specified by the replication_slot_inactive_timeout parameter.

thanks
Shveta

  245. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-04T03:47:33Z

    On Tue, Sep 3, 2024 at 3:01 PM shveta malik <shveta.malik@gmail.com> wrote:
>
>
> 1)
> I see that ReplicationSlotAlter() will error out if the slot is
> invalidated due to timeout. I have not tested it myself, but do you
> know if  slot-alter errors out for other invalidation causes as well?
> Just wanted to confirm that the behaviour is consistent for all
> invalidation causes.

I was able to test this and as anticipated behavior is different. When
slot is invalidated due to say 'wal_removed', I am still able to do
'alter' of that slot.
Please see:

Pub:
  slot_name  | failover | synced |          inactive_since          |
invalidation_reason
-------------+----------+--------+----------------------------------+---------------------
 mysubnew1_1 | t        | f      | 2024-09-04 08:58:12.802278+05:30 |
wal_removed

Sub:
newdb1=# alter subscription mysubnew1_1 disable;
ALTER SUBSCRIPTION

newdb1=# alter subscription mysubnew1_1 set (failover=false);
ALTER SUBSCRIPTION

Pub: (failover altered)
  slot_name  | failover | synced |          inactive_since          |
invalidation_reason
-------------+----------+--------+----------------------------------+---------------------
 mysubnew1_1 | f        | f      | 2024-09-04 08:58:47.824471+05:30 |
wal_removed


while when invalidation_reason is 'inactive_timeout', it fails:

Pub:
  slot_name  | failover | synced |          inactive_since          |
invalidation_reason
-------------+----------+--------+----------------------------------+---------------------
 mysubnew1_1 | t        | f      | 2024-09-03 14:30:57.532206+05:30 |
inactive_timeout

Sub:
newdb1=# alter subscription mysubnew1_1 disable;
ALTER SUBSCRIPTION

newdb1=# alter subscription mysubnew1_1 set (failover=false);
ERROR:  could not alter replication slot "mysubnew1_1": ERROR:  can no
longer get changes from replication slot "mysubnew1_1"
DETAIL:  The slot became invalid because it was inactive since
2024-09-04 08:54:20.308996+05:30, which is more than 0 seconds ago.
HINT:  You might need to increase "replication_slot_inactive_timeout.".

I think the behavior should be same.

thanks
Shveta

  246. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-04T09:18:51Z

    On Wed, Sep 4, 2024 at 9:17 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Tue, Sep 3, 2024 at 3:01 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> >


1)
It is related to one of my previous comments (pt 3 in [1]) where I
stated that inactive_since should not keep on changing once a slot is
invalidated.
Below is one side effect if inactive_since keeps on changing:

postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
pg_current_wal_lsn());
ERROR:  can no longer get changes from replication slot "mysubnew1_1"
DETAIL:  The slot became invalid because it was inactive since
2024-09-04 10:03:56.68053+05:30, which is more than 10 seconds ago.
HINT:  You might need to increase "replication_slot_inactive_timeout.".

postgres=# select now();
               now
---------------------------------
 2024-09-04 10:04:00.26564+05:30

'DETAIL' gives wrong information, we are not past 10-seconds. This is
because inactive_since got updated even in ERROR scenario.


2)
One more issue in this message is, once I set
replication_slot_inactive_timeout to a bigger value, it becomes more
misleading. This is because invalidation was done in the past using
previous value while message starts showing new value:

ALTER SYSTEM SET replication_slot_inactive_timeout TO '36h';

--see 129600 secs in DETAIL and the current time.
postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
pg_current_wal_lsn());
ERROR:  can no longer get changes from replication slot "mysubnew1_1"
DETAIL:  The slot became invalid because it was inactive since
2024-09-04 10:06:38.980939+05:30, which is more than 129600 seconds
ago.
postgres=# select now();
               now
----------------------------------
 2024-09-04 10:07:35.201894+05:30

I feel we should change this message itself.

~~~~~

When invalidation is due to wal_removed, we get a way simpler message:

newdb1=# SELECT * FROM pg_replication_slot_advance('mysubnew1_2',
pg_current_wal_lsn());
ERROR:  replication slot "mysubnew1_2" cannot be advanced
DETAIL:  This slot has never previously reserved WAL, or it has been
invalidated.

This message does not mention 'max_slot_wal_keep_size'. We should have
a similar message for our case. Thoughts?

[1]:  https://www.postgresql.org/message-id/CAJpy0uC8Dg-0JS3NRUwVUemgz5Ar2v3_EQQFXyAigWSEQ8U47Q%40mail.gmail.com

thanks
Shveta

  247. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-05T03:55:51Z

    On Wed, Sep 4, 2024 at 2:49 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Sep 4, 2024 at 9:17 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Tue, Sep 3, 2024 at 3:01 PM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > >
>
>
> 1)
> It is related to one of my previous comments (pt 3 in [1]) where I
> stated that inactive_since should not keep on changing once a slot is
> invalidated.
>

Agreed. Updating the inactive_since for a slot that is already invalid
is misleading.

>
>
> 2)
> One more issue in this message is, once I set
> replication_slot_inactive_timeout to a bigger value, it becomes more
> misleading. This is because invalidation was done in the past using
> previous value while message starts showing new value:
>
> ALTER SYSTEM SET replication_slot_inactive_timeout TO '36h';
>
> --see 129600 secs in DETAIL and the current time.
> postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
> pg_current_wal_lsn());
> ERROR:  can no longer get changes from replication slot "mysubnew1_1"
> DETAIL:  The slot became invalid because it was inactive since
> 2024-09-04 10:06:38.980939+05:30, which is more than 129600 seconds
> ago.
> postgres=# select now();
>                now
> ----------------------------------
>  2024-09-04 10:07:35.201894+05:30
>
> I feel we should change this message itself.
>

+1.

-- 
With Regards,
Amit Kapila.

  248. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-05T04:00:16Z

    On Wed, Sep 4, 2024 at 9:17 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Tue, Sep 3, 2024 at 3:01 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> >
> > 1)
> > I see that ReplicationSlotAlter() will error out if the slot is
> > invalidated due to timeout. I have not tested it myself, but do you
> > know if  slot-alter errors out for other invalidation causes as well?
> > Just wanted to confirm that the behaviour is consistent for all
> > invalidation causes.
>
> I was able to test this and as anticipated behavior is different. When
> slot is invalidated due to say 'wal_removed', I am still able to do
> 'alter' of that slot.
> Please see:
>
> Pub:
>   slot_name  | failover | synced |          inactive_since          |
> invalidation_reason
> -------------+----------+--------+----------------------------------+---------------------
>  mysubnew1_1 | t        | f      | 2024-09-04 08:58:12.802278+05:30 |
> wal_removed
>
> Sub:
> newdb1=# alter subscription mysubnew1_1 disable;
> ALTER SUBSCRIPTION
>
> newdb1=# alter subscription mysubnew1_1 set (failover=false);
> ALTER SUBSCRIPTION
>
> Pub: (failover altered)
>   slot_name  | failover | synced |          inactive_since          |
> invalidation_reason
> -------------+----------+--------+----------------------------------+---------------------
>  mysubnew1_1 | f        | f      | 2024-09-04 08:58:47.824471+05:30 |
> wal_removed
>
>
> while when invalidation_reason is 'inactive_timeout', it fails:
>
> Pub:
>   slot_name  | failover | synced |          inactive_since          |
> invalidation_reason
> -------------+----------+--------+----------------------------------+---------------------
>  mysubnew1_1 | t        | f      | 2024-09-03 14:30:57.532206+05:30 |
> inactive_timeout
>
> Sub:
> newdb1=# alter subscription mysubnew1_1 disable;
> ALTER SUBSCRIPTION
>
> newdb1=# alter subscription mysubnew1_1 set (failover=false);
> ERROR:  could not alter replication slot "mysubnew1_1": ERROR:  can no
> longer get changes from replication slot "mysubnew1_1"
> DETAIL:  The slot became invalid because it was inactive since
> 2024-09-04 08:54:20.308996+05:30, which is more than 0 seconds ago.
> HINT:  You might need to increase "replication_slot_inactive_timeout.".
>
> I think the behavior should be same.
>

We should not allow the invalid replication slot to be altered
irrespective of the reason unless there is any benefit.

-- 
With Regards,
Amit Kapila.

  249. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-09-08T11:54:47Z

    Hi,

Thanks for reviewing.

On Mon, Sep 2, 2024 at 1:37 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Commit message.
>
> 1.
> Because such synced slots are typically considered not
> active (for them to be later considered as inactive) as they don't
> perform logical decoding to produce the changes.
>
> This sentence is bad grammar. The docs have the same wording, so
> please see my doc review comment #4 suggestion below.

+1

> 2.
> +       <para>
> +        Invalidates replication slots that are inactive for longer than
> +        specified amount of time. If this value is specified without units,
> +        it is taken as seconds. A value of zero (which is default) disables
> +        the timeout mechanism. This parameter can only be set in
> +        the <filename>postgresql.conf</filename> file or on the server
> +        command line.
> +       </para>
> +
>
> nit - This is OK as-is, but OTOH why not make the wording consistent
> with the previous GUC description? (e.g. see my v43 [1] #2 review
> comment)

+1.

> 3.
> +       <para>
> +        This invalidation check happens either when the slot is acquired
> +        for use or during checkpoint. The time since the slot has become
> +        inactive is known from its
> +        <structfield>inactive_since</structfield> value using which the
> +        timeout is measured.
> +       </para>
> +
>
> I felt this is slightly misleading because slot acquiring has nothing
> to do with setting the slot invalidation anymore. Furthermore, the 2nd
> sentence is bad grammar.
>
> nit - IMO something simple like the following rewording can address
> both of those points:
>
> Slot invalidation due to inactivity timeout occurs during checkpoint.
> The duration of slot inactivity is calculated using the slot's
> <structfield>inactive_since</structfield> field value.

+1.

> 4.
> +        Because such synced slots are typically considered not active
> +        (for them to be later considered as inactive) as they don't perform
> +        logical decoding to produce the changes.
>
> That sentence has bad grammar.
>
> nit – suggest a much simpler replacement:
> Synced slots are always considered to be inactive because they don't
> perform logical decoding to produce changes.

+1.

> 5.
> +#define IsInactiveTimeoutSlotInvalidationApplicable(s) \
>
> 5a.
> I felt this would be better implemented as an inline function. Then it
> can be commented on properly to explain the parts of the condition.
> e.g. the large comment currently in InvalidatePossiblyObsoleteSlot()
> would be more appropriate in this function.

+1.

> 5b.
> The name is very long. Can't it be something shorter/simpler like:
> 'IsSlotATimeoutCandidate()'
>
> ~~~

Missing inactive in the above suggested name. Used
SlotInactiveTimeoutCheckAllowed, similar to XLogInsertAllowed.

> 6. ReplicationSlotAcquire
>
> -ReplicationSlotAcquire(const char *name, bool nowait)
> +ReplicationSlotAcquire(const char *name, bool nowait,
> +    bool check_for_invalidation)
>
> nit - Previously this new parameter really did mean to "check" for
> [and set the slot] invalidation. But now I suggest renaming it to
> 'error_if_invalid' to properly reflect the new usage. And also in the
> slot.h.

+1.

> 7.
> + /*
> + * Error out if the slot has been invalidated previously. Because there's
> + * no use in acquiring the invalidated slot.
> + */
>
> nit - The comment is contrary to the code. If there was no reason to
> skip this error, then you would not have the new parameter allowing
> you to skip this error. I suggest just repeating the same comment as
> in the function header.

+1.

> 8. ReportSlotInvalidation
>
> nit - Added some blank lines for consistency.

+1.

> 9. InvalidatePossiblyObsoleteSlot
>
> 9a.
> Consistency is good (commit message, docs and code comments for this),
> but the added sentence has bad grammar. Please see the docs review
> comment #4 above for some alternate phrasing.

+1.

> 9b.
> Now that this logic is moved into a macro (I suggested it should be an
> inline function) IMO this comment does not belong here anymore because
> it is commenting code that you cannot see. Instead, this comment (or
> something like it) should be as comments within the new function.
>
> ======
> src/include/replication/slot.h

+1.

> 10.
> +extern void ReplicationSlotAcquire(const char *name, bool nowait,
> +    bool check_for_invalidation);
>
> Change the new param name as described in the earlier review comment.

+1.

> Please refer to the attached file which implements some of the nits
> mentioned above.

Merged the diff into v45. Thanks.

On Tue, Sep 3, 2024 at 12:26 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> TEST CASE #1
>
> 1.
> +# Wait for the inactive replication slot to be invalidated.
>
> Is that comment correct? IIUC the synced slot should *already* be
> invalidated from the primary, so here we are not really "waiting" for
> it to be invalidated; Instead, we are just "confirming" that the
> synchronized slot is already invalidated with the correct reason as
> expected.

Modified the comment.

> 2.
> +# Synced slot mustn't get invalidated on the standby even after a checkpoint,
> +# it must sync invalidation from the primary. So, we must not see the slot's
> +# invalidation message in server log.
>
> This test case seemed bogus, for a couple of reasons:
>
> 2a. IIUC this 'lsub1_sync_slot' is the same one that is already
> invalid (from the primary), so nobody should be surprised that an
> already invalid slot doesn't get flagged as invalid again. i.e.
> Shouldn't your test scenario here be done using a valid synced slot?

+1. Added another test case for checking the synced slot not getting
invalidated despite inactive timeout being set.

> 2b. AFAICT it was only moments above this CHECKPOINT where you
> assigned the standby inactivity timeout to 2s. So even if there was
> some bug invalidating synced slots I don't think you gave it enough
> time to happen -- e.g. I doubt 2s has elapsed yet.

Added sleep(timeout+1) before the checkpoint.

> 3.
> +# Stop standby to make the standby's replication slot on the primary inactive
> +$standby1->stop;
> +
> +# Wait for the standby's replication slot to become inactive
>
> TEST CASE #2
>
> 4.
> +# Stop subscriber to make the replication slot on publisher inactive
> +$subscriber->stop;
> +
> +# Wait for the replication slot to become inactive and then invalidated due to
> +# timeout.
> +wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
> + $inactive_timeout);
>
> IIUC, this is just like comment #3 above. Both these (the stop and the
> wait) seem to belong together, so I think maybe a single bigger
> explanatory comment covering both parts would help for understanding.

Done.

> 5.
> +# Testcase end: Invalidate logical subscriber's slot due to
> +# replication_slot_inactive_timeout.
> +# =============================================================================
>
> IMO the rest of the comment after "Testcase end" isn't very useful.

Removed.

> ======
> sub wait_for_slot_invalidation
>
> 6.
> +sub wait_for_slot_invalidation
> +{
>
> An explanatory header comment for this subroutine would be helpful.

Done.

> 7.
> + # Wait for the replication slot to become inactive
> + $node->poll_query_until(
>
> Why are there are 2 separate poll_query_until's here? Can't those be
> combined into just one?

Ah. My bad. Removed.

> ~~~
>
> 8.
> + # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
> + # for the slot to get invalidated.
> + sleep($inactive_timeout);
> +
>
> Maybe this special sleep to prevent too many CHECKPOINTs should be
> moved to be inside the other subroutine, which is actually doing those
> CHECKPOINTs.

Done.

> 9.
> + # Wait for the inactive replication slot to be invalidated
> +   "Timed out while waiting for inactive slot $slot_name to be
> invalidated on node $name";
> +
>
> The comment seems misleading. IIUC you are not "waiting" for the
> invalidation here, because it is the other subroutine doing the
> waiting for the invalidation message in the logs. Instead, here I
> think you are just confirming the 'invalidation_reason' got set
> correctly. The comment should say what it is really doing.

Modified.

> sub check_for_slot_invalidation_in_server_log
>
> 10.
> +# Check for invalidation of slot in server log
> +sub check_for_slot_invalidation_in_server_log
> +{
>
> I think the main function of this subroutine is the CHECKPOINT and the
> waiting for the server log to say invalidation happened. It is doing a
> loop of a) CHECKPOINT then b) inspecting the server log for the slot
> invalidation, and c) waiting for a bit. Repeat 10 times.
>
> A comment describing the logic for this subroutine would be helpful.
>
> The most important side-effect of this function is the CHECKPOINT
> because without that nothing will ever get invalidated due to
> inactivity, but this key point is not obvious from the subroutine
> name.
>
> IMO it would be better to name this differently to reflect what it is
> really doing:
> e.g. "CHECKPOINT_and_wait_for_slot_invalidation_in_server_log"

That would be too long. Changed the function name to
trigger_slot_invalidation() which is appropriate.

Please find the v45 patch. Addressed above and Shveta's review comments [1].

Amit's comments [2] and [3] are still pending.

[1] https://www.postgresql.org/message-id/CAJpy0uC8Dg-0JS3NRUwVUemgz5Ar2v3_EQQFXyAigWSEQ8U47Q%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAA4eK1K7DdT_5HnOWs5tVPYC%3D-h%2Bm85wu7k-7RVJaJ7zMxprWQ%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CAA4eK1%2Bkt-QRr1RP%3DD%3D4_tp%2BS%2BCErQ6rNe7KVYEyZ3f6PYXpvw%40mail.gmail.com

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  250. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-09-08T11:55:42Z

    Hi,

Thanks for reviewing.

On Tue, Sep 3, 2024 at 3:01 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> 1)
> I see that ReplicationSlotAlter() will error out if the slot is
> invalidated due to timeout. I have not tested it myself, but do you
> know if  slot-alter errors out for other invalidation causes as well?
> Just wanted to confirm that the behaviour is consistent for all
> invalidation causes.

Will respond to Amit's comment soon.

> 2)
> When a slot is invalidated, and we try to use that slot, it gives this msg:
>
> ERROR:  can no longer get changes from replication slot "mysubnew1_2"
> DETAIL:  The slot became invalid because it was inactive since
> 2024-09-03 14:23:34.094067+05:30, which is more than 600 seconds ago.
> HINT:  You might need to increase "replication_slot_inactive_timeout.".
>
> Isn't HINT misleading? Even if we increase it now, the slot can not be
> reused again.
>
> Below is one side effect if inactive_since keeps on changing:
>
> postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
> pg_current_wal_lsn());
> ERROR:  can no longer get changes from replication slot "mysubnew1_1"
> DETAIL:  The slot became invalid because it was inactive since
> 2024-09-04 10:03:56.68053+05:30, which is more than 10 seconds ago.
> HINT:  You might need to increase "replication_slot_inactive_timeout.".
>
> postgres=# select now();
>                now
> ---------------------------------
>  2024-09-04 10:04:00.26564+05:30
>
> 'DETAIL' gives wrong information, we are not past 10-seconds. This is
> because inactive_since got updated even in ERROR scenario.
>
> ERROR:  can no longer get changes from replication slot "mysubnew1_1"
> DETAIL:  The slot became invalid because it was inactive since
> 2024-09-04 10:06:38.980939+05:30, which is more than 129600 seconds
> ago.
> postgres=# select now();
>                now
> ----------------------------------
>  2024-09-04 10:07:35.201894+05:30
>
> I feel we should change this message itself.

Removed the hint and corrected the detail message as following:

errmsg("can no longer get changes from replication slot \"%s\"",
NameStr(s->data.name)),
errdetail("This slot has been invalidated because it was inactive for
longer than the amount of time specified by \"%s\".",
"replication_slot_inactive_timeout.")));

> 3)
> When the slot is invalidated, the' inactive_since' still keeps on
> changing when there is a subscriber trying to start replication
> continuously. I think ReplicationSlotAcquire() keeps on failing and
> thus Release keeps on setting it again and again. Shouldn't we stop
> setting/chnaging  'inactive_since' once the slot is invalidated
> already, otherwise it will be misleading.
>
> postgres=# select failover,synced,inactive_since,invalidation_reason
> from pg_replication_slots;
>
>  failover | synced |          inactive_since          | invalidation_reason
> ----------+--------+----------------------------------+---------------------
>  t        | f      | 2024-09-03 14:23:.. | inactive_timeout
>
> after sometime:
>  failover | synced |          inactive_since          | invalidation_reason
> ----------+--------+----------------------------------+---------------------
>  t        | f      | 2024-09-03 14:26:..| inactive_timeout

Changed it to not update inactive_since for slots invalidated due to
inactive timeout.

> 4)
> src/sgml/config.sgml:
>
> 4a)
> + A value of zero (which is default) disables the timeout mechanism.
>
> Better will be:
> A value of zero (which is default) disables the inactive timeout
> invalidation mechanism .

Changed.

> 4b)
> 'synced' and inactive_since should point to pg_replication_slots:
>
> example:
> <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>

Modified.

> 5)
> src/sgml/system-views.sgml:
> + ..the slot has been inactive for longer than the duration specified
> by replication_slot_inactive_timeout parameter.
>
> Better to have:
> ..the slot has been inactive for a time longer than the duration
> specified by the replication_slot_inactive_timeout parameter.

Changed it to the following to be consistent with the config.sgml.

          <literal>inactive_timeout</literal> means that the slot has been
          inactive for longer than the amount of time specified by the
          <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.

Please find the v45 patch posted upthread at
https://www.postgresql.org/message-id/CALj2ACWXQT3_HY40ceqKf1DadjLQP6b1r%3D0sZRh-xhAOd-b0pA%40mail.gmail.com
for the changes.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  251. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-09T03:47:30Z

    On Thu, Sep 5, 2024 at 9:30 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Sep 4, 2024 at 9:17 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Tue, Sep 3, 2024 at 3:01 PM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > >
> > > 1)
> > > I see that ReplicationSlotAlter() will error out if the slot is
> > > invalidated due to timeout. I have not tested it myself, but do you
> > > know if  slot-alter errors out for other invalidation causes as well?
> > > Just wanted to confirm that the behaviour is consistent for all
> > > invalidation causes.
> >
> > I was able to test this and as anticipated behavior is different. When
> > slot is invalidated due to say 'wal_removed', I am still able to do
> > 'alter' of that slot.
> > Please see:
> >
> > Pub:
> >   slot_name  | failover | synced |          inactive_since          |
> > invalidation_reason
> > -------------+----------+--------+----------------------------------+---------------------
> >  mysubnew1_1 | t        | f      | 2024-09-04 08:58:12.802278+05:30 |
> > wal_removed
> >
> > Sub:
> > newdb1=# alter subscription mysubnew1_1 disable;
> > ALTER SUBSCRIPTION
> >
> > newdb1=# alter subscription mysubnew1_1 set (failover=false);
> > ALTER SUBSCRIPTION
> >
> > Pub: (failover altered)
> >   slot_name  | failover | synced |          inactive_since          |
> > invalidation_reason
> > -------------+----------+--------+----------------------------------+---------------------
> >  mysubnew1_1 | f        | f      | 2024-09-04 08:58:47.824471+05:30 |
> > wal_removed
> >
> >
> > while when invalidation_reason is 'inactive_timeout', it fails:
> >
> > Pub:
> >   slot_name  | failover | synced |          inactive_since          |
> > invalidation_reason
> > -------------+----------+--------+----------------------------------+---------------------
> >  mysubnew1_1 | t        | f      | 2024-09-03 14:30:57.532206+05:30 |
> > inactive_timeout
> >
> > Sub:
> > newdb1=# alter subscription mysubnew1_1 disable;
> > ALTER SUBSCRIPTION
> >
> > newdb1=# alter subscription mysubnew1_1 set (failover=false);
> > ERROR:  could not alter replication slot "mysubnew1_1": ERROR:  can no
> > longer get changes from replication slot "mysubnew1_1"
> > DETAIL:  The slot became invalid because it was inactive since
> > 2024-09-04 08:54:20.308996+05:30, which is more than 0 seconds ago.
> > HINT:  You might need to increase "replication_slot_inactive_timeout.".
> >
> > I think the behavior should be same.
> >
>
> We should not allow the invalid replication slot to be altered
> irrespective of the reason unless there is any benefit.
>

Okay, then I think we need to change the existing behaviour of the
other invalidation causes which still allow alter-slot.

thanks
Shveta

  252. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-09-09T04:56:17Z

    Hi,

On Mon, Sep 9, 2024 at 9:17 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> > We should not allow the invalid replication slot to be altered
> > irrespective of the reason unless there is any benefit.
>
> Okay, then I think we need to change the existing behaviour of the
> other invalidation causes which still allow alter-slot.

+1. Perhaps, track it in a separate thread?

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  253. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-09T04:58:42Z

    On Mon, Sep 9, 2024 at 10:26 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Mon, Sep 9, 2024 at 9:17 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > We should not allow the invalid replication slot to be altered
> > > irrespective of the reason unless there is any benefit.
> >
> > Okay, then I think we need to change the existing behaviour of the
> > other invalidation causes which still allow alter-slot.
>
> +1. Perhaps, track it in a separate thread?

I think so. It does not come under the scope of this thread.

thanks
Shveta

  254. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-09T05:23:50Z

    On Sun, Sep 8, 2024 at 5:25 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
>
> Please find the v45 patch. Addressed above and Shveta's review comments [1].
>

Thanks for the patch. Please find my comments:

1)
src/sgml/config.sgml:

+  Synced slots are always considered to be inactive because they
don't perform logical decoding to produce changes.

It is better we avoid such a statement, as internally we use logical
decoding to advance restart-lsn, see
'LogicalSlotAdvanceAndCheckSnapState' called form slotsync.c.
<Also see related comment 6 below>

2)
src/sgml/config.sgml:

+ disables the inactive timeout invalidation mechanism

+ Slot invalidation due to inactivity timeout occurs during checkpoint.

Either have 'inactive' at both the places or 'inactivity'.


3)
slot.c:
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause
cause,
+    ReplicationSlot *s,
+    XLogRecPtr oldestLSN,
+    Oid dboid,
+    TransactionId snapshotConflictHorizon,
+    bool *invalidated);
+static inline bool SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s);

I think, we do not need above 2 declarations. The code compile fine
without these as the usage is later than the definition.


4)
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
+ */
+ if (error_if_invalid && s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)

The comment is generic while the 'if condition' is specific to one
invalidation cause. Even though I feel it can be made generic test for
all invalidation causes but that is not under scope of this thread and
needs more testing/analysis. For the time being, we can make comment
specific to the concerned invalidation cause. The header of function
will also need the same change.

5)
SlotInactiveTimeoutCheckAllowed():

+ * Check if inactive timeout invalidation mechanism is disabled or slot is
+ * currently being used or server is in recovery mode or slot on standby is
+ * currently being synced from the primary.
+ *

These comments say exact opposite of what we are checking in code.
Since the function name has 'Allowed' in it, we should be putting
comments which say what allows it instead of what disallows it.


6)

+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)

Perhaps we should avoid mentioning logical decoding here. When slots
are synced, they are performing decoding and their inactive_since is
changing continuously. A better way to make this statement will be:

We want to ensure that the slots being synchronized are not
invalidated, as they need to be preserved for future use when the
standby server is promoted to the primary. This is necessary for
resuming logical replication from the new primary server.
<Rephrase if needed>

7)

InvalidatePossiblyObsoleteSlot()

we are calling SlotInactiveTimeoutCheckAllowed() twice in this
function. We shall optimize.

At the first usage place, shall we simply get timestamp when cause is
RS_INVAL_INACTIVE_TIMEOUT without checking
SlotInactiveTimeoutCheckAllowed() as IMO it does not seem a
performance critical section. Or if we retain check at first place,
then at the second place we can avoid calling it again based on
whether 'now' is NULL or not.

thanks
Shveta

  255. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-09-09T07:40:50Z

    Hi, here are some review comments for v45-0001 (excluding the test code)

======
doc/src/sgml/config.sgml

1.
+        Note that the inactive timeout invalidation mechanism is not
+        applicable for slots on the standby server that are being synced
+        from primary server (i.e., standby slots having

nit - /from primary server/from the primary server/

======
src/backend/replication/slot.c

2. ReplicationSlotAcquire

+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive
for longer than the amount of time specified by \"%s\".",
+    "replication_slot_inactive_timeout.")));

nit - "replication_slot_inactive_timeout." - should be no period
inside that GUC name literal

~~~

3. ReportSlotInvalidation

I didn't understand why there was a hint for:
"You might need to increase \"%s\".", "max_slot_wal_keep_size"

But you don't have an equivalent hint for timeout invalidation:
"You might need to increase \"%s\".", "replication_slot_inactive_timeout"

Why aren't these similar cases consistent?

~~~

4. RestoreSlotFromDisk

+ /* Use the same inactive_since time for all the slots. */
+ if (now == 0)
+ now = GetCurrentTimestamp();
+

Is the deferred assignment really necessary? Why not just
unconditionally assign the 'now' just before the for-loop? Or even at
the declaration? e.g. The 'replication_slot_inactive_timeout' is
measured in seconds so I don't think 'inactive_since' being wrong by a
millisecond here will make any difference.

======
src/include/replication/slot.h

5. ReplicationSlotSetInactiveSince

+/*
+ * Set slot's inactive_since property unless it was previously invalidated due
+ * to inactive timeout.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+ bool acquire_lock)
+{
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ if (s->data.invalidated != RS_INVAL_INACTIVE_TIMEOUT)
+ s->inactive_since = *now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}

Is the logic correct? What if the slot was already invalid due to some
reason other than RS_INVAL_INACTIVE_TIMEOUT? Is an Assert needed?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  256. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-09T09:34:44Z

    On Mon, Sep 9, 2024 at 10:28 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Sep 9, 2024 at 10:26 AM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Hi,
> >
> > On Mon, Sep 9, 2024 at 9:17 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > > We should not allow the invalid replication slot to be altered
> > > > irrespective of the reason unless there is any benefit.
> > >
> > > Okay, then I think we need to change the existing behaviour of the
> > > other invalidation causes which still allow alter-slot.
> >
> > +1. Perhaps, track it in a separate thread?
>
> I think so. It does not come under the scope of this thread.
>

It makes sense to me as well. But let's go ahead and get that sorted out first.

-- 
With Regards,
Amit Kapila.

  257. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-09-09T18:42:50Z

    Hi,

On Mon, Sep 9, 2024 at 3:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > > > > We should not allow the invalid replication slot to be altered
> > > > > irrespective of the reason unless there is any benefit.
> > > >
> > > > Okay, then I think we need to change the existing behaviour of the
> > > > other invalidation causes which still allow alter-slot.
> > >
> > > +1. Perhaps, track it in a separate thread?
> >
> > I think so. It does not come under the scope of this thread.
>
> It makes sense to me as well. But let's go ahead and get that sorted out first.

Moved the discussion to new thread -
https://www.postgresql.org/message-id/CALj2ACW4fSOMiKjQ3%3D2NVBMTZRTG8Ujg6jsK9z3EvOtvA4vzKQ%40mail.gmail.com.
Please have a look.

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  258. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-09-10T01:34:24Z

    Hi, here is the remainder of my v45-0001 review. These comments are
for the test code only.

======
Testcase #1

1.
+# Testcase start
+#
+# Invalidate streaming standby slot and logical failover slot on primary due to
+# inactive timeout. Also, check logical failover slot synced to standby from
+# primary doesn't invalidate on its own, but gets the invalidated
state from the
+# primary.

nit - s/primary/the primary/ (in a couple of places)
nit - s/standby/the standby/
nit - other trivial tweaks.

~~~

2.
+# Create sync slot on primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1',
'test_decoding', false, false, true);}
+);

nit - s/primary/the primary/

~~~

3.
+$primary->safe_psql(
+ 'postgres', qq[
+    SELECT pg_create_physical_replication_slot(slot_name :=
'sb_slot1', immediately_reserve := true);
+]);

Should this have a comment?

~~~

4.
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);

nit - s/standby/the standby/

~~~

5.
+# Sync primary slot to standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");

nit - /Sync primary slot to standby/Sync the primary slots to the standby/

~~~

6.
+# Confirm that logical failover slot is created on standby

nit - s/Confirm that logical failover slot is created on
standby/Confirm that the logical failover slot is created on the
standby/

~~~

7.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+   WHERE slot_name = 'sync_slot1' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot sync_slot1 has synced as true on standby');

IMO here you should also be checking that the sync slot state is NOT
invalidated, just as a counterpoint for the test part later that
checks that it IS invalidated.

~~~

8.
+my $inactive_timeout = 1;
+
+# Set timeout so that next checkpoint will invalidate inactive slot
+$primary->safe_psql(
+ 'postgres', qq[
+    ALTER SYSTEM SET replication_slot_inactive_timeout TO
'${inactive_timeout}s';
+]);
+$primary->reload;

8a.
nit - I think that $inactive_timeout assignment belongs below your comment.

~

8b.
nit - s/Set timeout so that next checkpoint will invalidate inactive
slot/Set timeout GUC so that the next checkpoint will invalidate
inactive slots/

~~~

9.
+# Check for logical failover slot to become inactive on primary. Note that
+# nobody has acquired slot yet, so it must get invalidated due to
+# inactive timeout.

nit - /Check for logical failover slot to become inactive on
primary./Wait for logical failover slot to become inactive on the
primary./
nit - /has acquired slot/has acquired the slot/

~~~

10.
+# Sync primary slot to standby. Note that primary slot has already been
+# invalidated due to inactive timeout. Standby must just sync inavalidated
+# state.

nit - minor, add "the". fix typo "inavalidated", etc. suggestion:

Re-sync the primary slots to the standby. Note that the primary slot was already
invalidated (above) due to inactive timeout. The standby must just
sync the invalidated
state.

~~~

11.
+# Make standby slot on primary inactive and check for invalidation
+$standby1->stop;

nit - /standby slot/the standby slot/
nit - /on primary/on the primary/

======
Testcase #2

12.
I'm not sure it is necessary to do all this extra work. IIUC, there
was already almost everything you needed in the previous Testcase #1.
So, I thought you could just combine this extra standby timeout test
in Testcase #1.

Indeed, your Testcase #1 comment still says it is doing this: ("Also,
check logical failover slot synced to standby from primary doesn't
invalidate on its own,...")

e.g.
- NEW: set the GUC timeout on the standby
- sync the sync_slot (already doing in test #1)
- ensure the synced slot is NOT invalid (already suggested above for test #1)
- NEW: then do a standby sleep > timeout duration
- NEW: then do a standby CHECKPOINT...
- NEW: then ensure the sync slot invalidation did NOT happen
- then proceed with the rest of test #1...

======
Testcase #3

13.
nit - remove a few blank lines to group associated statements together.

~~~

14.
+$publisher->safe_psql(
+ 'postgres', qq[
+    ALTER SYSTEM SET replication_slot_inactive_timeout TO '
${inactive_timeout}s';
+]);
+$publisher->reload;

nit - this deserves a comment, the same as in Testcase #1

======
sub wait_for_slot_invalidation

15.
+# Check for slot to first become inactive and then get invalidated
+sub check_for_slot_invalidation

nit - IMO the previous name was better (e.g. "wait_for.." instead of
"check_for...") because that describes exactly what the subroutine is
doing.

suggestion:
# Wait for the slot to first become inactive and then get invalidated
sub wait_for_slot_invalidation

~~~

16.
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;

The variable $name seems too vague. How about $node_name?

~~~

17.
+ # Wait for invalidation reason to be set
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+   or die
+   "Timed out while waiting for invalidation reason of slot $slot to
be set on node $name";

17a.
nit - /# Wait for invalidation reason to be set/# Check that the
invalidation reason is 'inactive_timeout'/

IIUC, the 'trigger_slot_invalidation' function has already invalidated
the slot at this point, so we are not really "Waiting..."; we are
"Checking..." that the reason was correctly set.

~

17b.
I think this code fragment maybe would be better put inside the
'trigger_slot_invalidation' function. (I've done this in the nitpicks
attachment)

~~~

18.
+ # Check that invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);

18a.
s/Check that invalidated slot/Check that an invalidated slot/

~

18b.
nit - Remove some blank lines, because the comment applies to all below it.

======
sub trigger_slot_invalidation

19.
+# Trigger slot invalidation and confirm it in server log
+sub trigger_slot_invalidation

nit - s/confirm it in server log/confirm it in the server log/

~

20.
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;

(same as the other subroutine)
nit - The variable $name seems too vague. How about $node_name?

======

Please refer to the attached nitpicks top-up patch which implements
most of the above nits.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  259. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-16T03:25:10Z

    On Tue, Sep 10, 2024 at 12:13 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Mon, Sep 9, 2024 at 3:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > > > > We should not allow the invalid replication slot to be altered
> > > > > > irrespective of the reason unless there is any benefit.
> > > > >
> > > > > Okay, then I think we need to change the existing behaviour of the
> > > > > other invalidation causes which still allow alter-slot.
> > > >
> > > > +1. Perhaps, track it in a separate thread?
> > >
> > > I think so. It does not come under the scope of this thread.
> >
> > It makes sense to me as well. But let's go ahead and get that sorted out first.
>
> Moved the discussion to new thread -
> https://www.postgresql.org/message-id/CALj2ACW4fSOMiKjQ3%3D2NVBMTZRTG8Ujg6jsK9z3EvOtvA4vzKQ%40mail.gmail.com.
> Please have a look.
>

That is pushed now. Please send the rebased patch after addressing the
pending comments.

-- 
With Regards,
Amit Kapila.

  260. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-09-16T09:47:47Z

    Hi,

Thanks for reviewing.

On Mon, Sep 9, 2024 at 10:54 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> 2)
> src/sgml/config.sgml:
>
> + disables the inactive timeout invalidation mechanism
>
> + Slot invalidation due to inactivity timeout occurs during checkpoint.
>
> Either have 'inactive' at both the places or 'inactivity'.

Used "inactive timeout".

> 3)
> slot.c:
> +static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause
> cause,
> +    ReplicationSlot *s,
> +    XLogRecPtr oldestLSN,
> +    Oid dboid,
> +    TransactionId snapshotConflictHorizon,
> +    bool *invalidated);
> +static inline bool SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s);
>
> I think, we do not need above 2 declarations. The code compile fine
> without these as the usage is later than the definition.

Hm, it's a usual practice that I follow irrespective of the placement
of function declarations. Since it was brought up, I removed the
declarations.

> 4)
> + /*
> + * An error is raised if error_if_invalid is true and the slot has been
> + * invalidated previously.
> + */
> + if (error_if_invalid && s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
>
> The comment is generic while the 'if condition' is specific to one
> invalidation cause. Even though I feel it can be made generic test for
> all invalidation causes but that is not under scope of this thread and
> needs more testing/analysis.

Right.

> For the time being, we can make comment
> specific to the concerned invalidation cause. The header of function
> will also need the same change.

Adjusted the comment, but left the variable name error_if_invalid as
is. Didn't want to make it long, one can look at the code to
understand what it is used for.

> 5)
> SlotInactiveTimeoutCheckAllowed():
>
> + * Check if inactive timeout invalidation mechanism is disabled or slot is
> + * currently being used or server is in recovery mode or slot on standby is
> + * currently being synced from the primary.
> + *
>
> These comments say exact opposite of what we are checking in code.
> Since the function name has 'Allowed' in it, we should be putting
> comments which say what allows it instead of what disallows it.

Modified.

> 1)
> src/sgml/config.sgml:
>
> +  Synced slots are always considered to be inactive because they
> don't perform logical decoding to produce changes.
>
> It is better we avoid such a statement, as internally we use logical
> decoding to advance restart-lsn, see
> 'LogicalSlotAdvanceAndCheckSnapState' called form slotsync.c.
> <Also see related comment 6 below>
>
> 6)
>
> + * Synced slots are always considered to be inactive because they don't
> + * perform logical decoding to produce changes.
> + */
> +static inline bool
> +SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
>
> Perhaps we should avoid mentioning logical decoding here. When slots
> are synced, they are performing decoding and their inactive_since is
> changing continuously. A better way to make this statement will be:
>
> We want to ensure that the slots being synchronized are not
> invalidated, as they need to be preserved for future use when the
> standby server is promoted to the primary. This is necessary for
> resuming logical replication from the new primary server.
> <Rephrase if needed>

They are performing logical decoding, but not producing the changes
for the clients to consume. So, IMO, the accompanying "to produce
changes" next to the "logical decoding" is good here.

> 7)
>
> InvalidatePossiblyObsoleteSlot()
>
> we are calling SlotInactiveTimeoutCheckAllowed() twice in this
> function. We shall optimize.
>
> At the first usage place, shall we simply get timestamp when cause is
> RS_INVAL_INACTIVE_TIMEOUT without checking
> SlotInactiveTimeoutCheckAllowed() as IMO it does not seem a
> performance critical section. Or if we retain check at first place,
> then at the second place we can avoid calling it again based on
> whether 'now' is NULL or not.

Getting a current timestamp can get costlier on platforms that use
various clock sources, so assigning 'now' unconditionally isn't the
way IMO. Using the inline function in two places improves the
readability. Can optimize it if there's any performance impact of
calling the inline function in two places.

Will post the new patch version soon.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  261. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-09-16T10:01:11Z

    Hi,

Thanks for reviewing.

On Mon, Sep 9, 2024 at 1:11 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> 1.
> +        Note that the inactive timeout invalidation mechanism is not
> +        applicable for slots on the standby server that are being synced
> +        from primary server (i.e., standby slots having
>
> nit - /from primary server/from the primary server/

+1

> 2. ReplicationSlotAcquire
>
> + errmsg("can no longer get changes from replication slot \"%s\"",
> + NameStr(s->data.name)),
> + errdetail("This slot has been invalidated because it was inactive
> for longer than the amount of time specified by \"%s\".",
> +    "replication_slot_inactive_timeout.")));
>
> nit - "replication_slot_inactive_timeout." - should be no period
> inside that GUC name literal

Typo. Fixed.

> 3. ReportSlotInvalidation
>
> I didn't understand why there was a hint for:
> "You might need to increase \"%s\".", "max_slot_wal_keep_size"
>
> Why aren't these similar cases consistent?

It looks misleading and not very useful. What happens if the removed
WAL (that's needed for the slot) is put back into pg_wal somehow (by
manually copying from archive or by some tool/script)? Can the slot
invalidated due to wal_removed start sending WAL to its clients?

> But you don't have an equivalent hint for timeout invalidation:
> "You might need to increase \"%s\".", "replication_slot_inactive_timeout"

I removed this per review comments upthread.

> 4. RestoreSlotFromDisk
>
> + /* Use the same inactive_since time for all the slots. */
> + if (now == 0)
> + now = GetCurrentTimestamp();
> +
>
> Is the deferred assignment really necessary? Why not just
> unconditionally assign the 'now' just before the for-loop? Or even at
> the declaration? e.g. The 'replication_slot_inactive_timeout' is
> measured in seconds so I don't think 'inactive_since' being wrong by a
> millisecond here will make any difference.

Moved it before the for-loop.

> 5. ReplicationSlotSetInactiveSince
>
> +/*
> + * Set slot's inactive_since property unless it was previously invalidated due
> + * to inactive timeout.
> + */
> +static inline void
> +ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
> + bool acquire_lock)
> +{
> + if (acquire_lock)
> + SpinLockAcquire(&s->mutex);
> +
> + if (s->data.invalidated != RS_INVAL_INACTIVE_TIMEOUT)
> + s->inactive_since = *now;
> +
> + if (acquire_lock)
> + SpinLockRelease(&s->mutex);
> +}
>
> Is the logic correct? What if the slot was already invalid due to some
> reason other than RS_INVAL_INACTIVE_TIMEOUT? Is an Assert needed?

Hm. Since invalidated slots can't be acquired and made active, not
modifying inactive_since irrespective of invalidation reason looks
good to me.

Please find the attached v46 patch having changes for the above review
comments and your test review comments and Shveta's review comments.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  262. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-16T11:24:40Z

    On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Please find the attached v46 patch having changes for the above review
> comments and your test review comments and Shveta's review comments.
>

-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
 {
  ReplicationSlot *s;
  int active_pid;
@@ -615,6 +620,22 @@ retry:
  /* We made this slot active, so it's ours now. */
  MyReplicationSlot = s;

+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive
for longer than the amount of time specified by \"%s\".",
+    "replication_slot_inactive_timeout")));
+ }

Why raise the ERROR just for timeout invalidation here and why not if
the slot is invalidated for other reasons? This raises the question of
what happens before this patch if the invalid slot is used from places
where we call ReplicationSlotAcquire(). I did a brief code analysis
and found that for StartLogicalReplication(), even if the error won't
occur in ReplicationSlotAcquire(), it would have been caught in
CreateDecodingContext(). I think that is where we should also add this
new error. Similarly, pg_logical_slot_get_changes_guts() and other
logical replication functions should be calling
CreateDecodingContext() which can raise the new ERROR. I am not sure
about how the invalid slots are handled during physical replication,
please check the behavior of that before this patch.

-- 
With Regards,
Amit Kapila.

  263. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2024-09-16T17:10:52Z

    Hi,

Thanks for looking into this.

On Mon, Sep 16, 2024 at 4:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Why raise the ERROR just for timeout invalidation here and why not if
> the slot is invalidated for other reasons? This raises the question of
> what happens before this patch if the invalid slot is used from places
> where we call ReplicationSlotAcquire(). I did a brief code analysis
> and found that for StartLogicalReplication(), even if the error won't
> occur in ReplicationSlotAcquire(), it would have been caught in
> CreateDecodingContext(). I think that is where we should also add this
> new error. Similarly, pg_logical_slot_get_changes_guts() and other
> logical replication functions should be calling
> CreateDecodingContext() which can raise the new ERROR. I am not sure
> about how the invalid slots are handled during physical replication,
> please check the behavior of that before this patch.

When physical slots are invalidated due to wal_removed reason, the failure
happens at a much later point for the streaming standbys while reading the
requested WAL files like the following:

2024-09-16 16:29:52.416 UTC [876059] FATAL:  could not receive data from
WAL stream: ERROR:  requested WAL segment 000000010000000000000005 has
already been removed
2024-09-16 16:29:52.416 UTC [872418] LOG:  waiting for WAL to become
available at 0/5002000

At this point, despite the slot being invalidated, its wal_status can still
come back to 'unreserved' even from 'lost', and the standby can catch up if
removed WAL files are copied either by manually or by a tool/script to the
primary's pg_wal directory. IOW, the physical slots invalidated due to
wal_removed are *somehow* recoverable unlike the logical slots.

IIUC, the invalidation of a slot implies that it is not guaranteed to hold
any resources like WAL and XMINs. Does it also imply that the slot must be
unusable?

-- 
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  264. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-09-17T01:27:24Z

    Here are a few comments for the patch v46-0001.

======
src/backend/replication/slot.c

1. ReportSlotInvalidation

On Mon, Sep 16, 2024 at 8:01 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Mon, Sep 9, 2024 at 1:11 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > 3. ReportSlotInvalidation
> >
> > I didn't understand why there was a hint for:
> > "You might need to increase \"%s\".", "max_slot_wal_keep_size"
> >
> > Why aren't these similar cases consistent?
>
> It looks misleading and not very useful. What happens if the removed
> WAL (that's needed for the slot) is put back into pg_wal somehow (by
> manually copying from archive or by some tool/script)? Can the slot
> invalidated due to wal_removed start sending WAL to its clients?
>
> > But you don't have an equivalent hint for timeout invalidation:
> > "You might need to increase \"%s\".", "replication_slot_inactive_timeout"
>
> I removed this per review comments upthread.

IIUC the errors are quite similar, so my previous review comment was
mostly about the unexpected inconsistency of why one of them has a
hint and the other one does not. I don't have a strong opinion about
whether they should both *have* or *not have* hints, so long as they
are treated the same.

If you think the current code hint is not useful then maybe we need a
new thread to address that existing issue. For example, maybe it
should be removed or reworded.

~~~

2. InvalidatePossiblyObsoleteSlot:

+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ if (!SlotInactiveTimeoutCheckAllowed(s))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+    replication_slot_inactive_timeout * 1000))

nit - it might be tidier to avoid multiple breaks by just combining
these conditions. See the nitpick attachment.

~~~

3.
  * - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs

nit - use comment wording "inactive slot timeout has occurred", to
make it identical to the comment in slot.h

======
src/test/recovery/t/050_invalidate_slots.pl

4.
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby. So, we must not see invalidation message in server
+# log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"sync_slot1\"",
+ $logstart),
+ 'check that synced slot sync_slot1 has not been invalidated on standby'
+);
+

It seems kind of brittle to check the logs for something that is NOT
there because any change to the message will make this accidentally
pass. Apart from that, it might anyway be more efficient just to check
the pg_replication_slots again to make sure the 'invalidation_reason
remains' still NULL.

======

Please see the attachment which implements some of the nit changes
mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  265. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-18T06:51:56Z

    On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
>
> Please find the attached v46 patch having changes for the above review
> comments and your test review comments and Shveta's review comments.
>

Thanks for addressing comments.

Is there a reason that we don't support this invalidation on hot
standby for non-synced slots? Shouldn't we support this time-based
invalidation there too just like other invalidations?

thanks
Shveta

  266. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-18T09:19:00Z

    On Wed, Sep 18, 2024 at 12:21 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Hi,
> >
> >
> > Please find the attached v46 patch having changes for the above review
> > comments and your test review comments and Shveta's review comments.
> >
>
> Thanks for addressing comments.
>
> Is there a reason that we don't support this invalidation on hot
> standby for non-synced slots? Shouldn't we support this time-based
> invalidation there too just like other invalidations?
>

Now since we are not changing inactive_since once it is invalidated,
we are not even initializing it during restart; and thus later when
someone tries to use slot, it leads to assert in
ReplicationSlotAcquire()  ( Assert(s->inactive_since > 0);

Steps:
--Disable logical subscriber and let the slot on publisher gets
invalidated due to inactive_timeout.
--Enable the logical subscriber again.
--Restart publisher.

a) We should initialize inactive_since when
ReplicationSlotSetInactiveSince() is called from RestoreSlotFromDisk()
even though it is invalidated.
b) And shall we mention in the doc of 'active_since', that once the
slot is invalidated, this value will remain unchanged until we
shutdown the server. On server restart, it is initialized to start
time. Thought?


thanks
Shveta

  267. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-18T10:01:16Z

    On Wed, Sep 18, 2024 at 2:49 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> > > Please find the attached v46 patch having changes for the above review
> > > comments and your test review comments and Shveta's review comments.
> > >

When the synced slot is marked as 'inactive_timeout' invalidated on
hot standby due to invalidation of publisher 's failover slot, the
former starts showing NULL' inactive_since'. Is this intentional
behaviour? I feel inactive_since should be non-NULL here too?
Thoughts?

physical standby:
postgres=# select slot_name, inactive_since, invalidation_reason,
failover, synced from pg_replication_slots;
slot_name  |          inactive_since                              |
invalidation_reason | failover | synced
-------------+----------------------------------+---------------------+----------+--------
sub2 | 2024-09-18 15:20:04.364998+05:30 |           | t        | t
sub3 | 2024-09-18 15:20:04.364953+05:30 |           | t        | t

After sync of invalidation_reason:

slot_name  |          inactive_since          | invalidation_reason |
failover | synced
-------------+----------------------------------+---------------------+----------+--------
 sub2 |                               | inactive_timeout    | t        | t
 sub3 |                               | inactive_timeout    | t        | t


thanks
shveta

  268. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-18T12:10:10Z

    On Mon, Sep 16, 2024 at 10:41 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Thanks for looking into this.
>
> On Mon, Sep 16, 2024 at 4:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Why raise the ERROR just for timeout invalidation here and why not if
> > the slot is invalidated for other reasons? This raises the question of
> > what happens before this patch if the invalid slot is used from places
> > where we call ReplicationSlotAcquire(). I did a brief code analysis
> > and found that for StartLogicalReplication(), even if the error won't
> > occur in ReplicationSlotAcquire(), it would have been caught in
> > CreateDecodingContext(). I think that is where we should also add this
> > new error. Similarly, pg_logical_slot_get_changes_guts() and other
> > logical replication functions should be calling
> > CreateDecodingContext() which can raise the new ERROR. I am not sure
> > about how the invalid slots are handled during physical replication,
> > please check the behavior of that before this patch.
>
> When physical slots are invalidated due to wal_removed reason, the failure happens at a much later point for the streaming standbys while reading the requested WAL files like the following:
>
> 2024-09-16 16:29:52.416 UTC [876059] FATAL:  could not receive data from WAL stream: ERROR:  requested WAL segment 000000010000000000000005 has already been removed
> 2024-09-16 16:29:52.416 UTC [872418] LOG:  waiting for WAL to become available at 0/5002000
>
> At this point, despite the slot being invalidated, its wal_status can still come back to 'unreserved' even from 'lost', and the standby can catch up if removed WAL files are copied either by manually or by a tool/script to the primary's pg_wal directory. IOW, the physical slots invalidated due to wal_removed are *somehow* recoverable unlike the logical slots.
>
> IIUC, the invalidation of a slot implies that it is not guaranteed to hold any resources like WAL and XMINs. Does it also imply that the slot must be unusable?
>

If we can't hold the dead rows against xmin of the invalid slot, then
how can we make it usable even after copying the required WAL?

-- 
With Regards,
Amit Kapila.

  269. Re: Introduce XID age and inactive timeout based replication slot invalidation

    shveta malik <shveta.malik@gmail.com> — 2024-09-19T04:10:12Z

    On Wed, Sep 18, 2024 at 3:31 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Sep 18, 2024 at 2:49 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > > Please find the attached v46 patch having changes for the above review
> > > > comments and your test review comments and Shveta's review comments.
> > > >
>

When we promote hot standby with synced logical slots to become new
primary, the logical slots are never invalidated with
'inactive_timeout' on new primary.  It seems the check in
SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
invalidation of slots on primary even if they are marked as 'synced'.
Please see [4].
I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
Once all these are addressed, I can continue reviewing further.

[1]: https://www.postgresql.org/message-id/CAJpy0uAwxc49Dz6t%3D-y_-z-MU%2BA4RWX4BR3Zri_jj2qgGMq_8g%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/CAJpy0uC6nN3SLbEuCvz7-CpaPdNdXxH%3DfeW5MhYQch-JWV0tLg%40mail.gmail.com
[3]: https://www.postgresql.org/message-id/CAJpy0uBXXJC6f04%2BFU1axKaU%2Bp78wN0SEhUNE9XoqbjXj%3Dhhgw%40mail.gmail.com

[4]:
--------------------
postgres=#  select pg_is_in_recovery();
--------
 f

postgres=# show replication_slot_inactive_timeout;
 replication_slot_inactive_timeout
-----------------------------------
 10s

postgres=# select slot_name, inactive_since, invalidation_reason,
synced from pg_replication_slots;
  slot_name  |          inactive_since          | invalidation_reason | synced
-------------+----------------------------------+---------------------+----------+--------
 mysubnew1_1 | 2024-09-19 09:04:09.714283+05:30 |                     | t

postgres=# select now();
               now
----------------------------------
 2024-09-19 09:06:28.871354+05:30

postgres=# checkpoint;
CHECKPOINT

postgres=# select slot_name, inactive_since, invalidation_reason,
synced from pg_replication_slots;
  slot_name  |          inactive_since          | invalidation_reason | synced
-------------+----------------------------------+---------------------+----------+--------
 mysubnew1_1 | 2024-09-19 09:04:09.714283+05:30 |                     | t
--------------------

thanks
Shveta

  270. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-07T10:03:33Z

    On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Please find the attached v46 patch having changes for the above review
> comments and your test review comments and Shveta's review comments.
>
Hi,

I’ve reviewed this thread and am interested in working on the
remaining tasks and comments, as well as the future review comments.
However, Bharath, please let me know if you'd prefer to continue with
it.

Attached the rebased v47 patch, which also addresses Peter’s comments
#2, #3, and #4 at [1]. I will try addressing other comments as well in
next versions.

[1] https://www.postgresql.org/message-id/CAHut%2BPs%3Dx%2B2Hq5ue0YppOeDZqgHTnyw%3Du%2Bvs-qy0JRjKaeJtew%40mail.gmail.com

--
Thanks,
Nisha

  271. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-11T12:42:28Z

    On Wed, Sep 18, 2024 at 3:31 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Sep 18, 2024 at 2:49 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > > Please find the attached v46 patch having changes for the above review
> > > > comments and your test review comments and Shveta's review comments.
> > > >
>
> When the synced slot is marked as 'inactive_timeout' invalidated on
> hot standby due to invalidation of publisher 's failover slot, the
> former starts showing NULL' inactive_since'. Is this intentional
> behaviour? I feel inactive_since should be non-NULL here too?
> Thoughts?
>
> physical standby:
> postgres=# select slot_name, inactive_since, invalidation_reason,
> failover, synced from pg_replication_slots;
> slot_name  |          inactive_since                              |
> invalidation_reason | failover | synced
> -------------+----------------------------------+---------------------+----------+--------
> sub2 | 2024-09-18 15:20:04.364998+05:30 |           | t        | t
> sub3 | 2024-09-18 15:20:04.364953+05:30 |           | t        | t
>
> After sync of invalidation_reason:
>
> slot_name  |          inactive_since          | invalidation_reason |
> failover | synced
> -------------+----------------------------------+---------------------+----------+--------
>  sub2 |                               | inactive_timeout    | t        | t
>  sub3 |                               | inactive_timeout    | t        | t
>
>

For synced slots on the standby, inactive_since indicates the last
synchronization time rather than the time the slot became inactive
(see doc - https://www.postgresql.org/docs/devel/view-pg-replication-slots.html).

In the reported case above, once a synced slot is invalidated we don't
even keep the last synchronization time for it. This is because when a
synced slot on the standby is marked invalid, inactive_since is reset
to NULL each time the slot-sync worker acquires a lock on it. This
lock acquisition before checking invalidation is done to avoid certain
race conditions and will activate the slot temporarily, resetting
inactive_since. Later, the slot-sync worker updates inactive_since for
all synced slots to the current synchronization time. However, for
invalid slots, this update is skipped, as per the patch’s design.

If we want to preserve the inactive_since value for the invalid synced
slots on standby, we need to clarify the time it should display. Here
are three possible approaches:

1) Copy the primary's inactive_since upon invalidation: When a slot
becomes invalid on the primary, the slot-sync worker could copy the
primary slot’s inactive_since to the standby slot and retain it, by
preventing future updates on the standby.

2) Use the current time of standby when the synced slot is marked
invalid for the first time and do not update it in subsequent sync
cycles if the slot is invalid.

Approach (2) seems more reasonable to me, however, Both 1) & 2)
approaches contradicts the purpose of inactive_since, as it no longer
represents either the true "last sync time" or the "time slot became
inactive" because the slot-sync worker acquires locks periodically for
syncing, and keeps activating the slot.

3) Continuously update inactive_since for invalid synced slots as
well: Treat invalid synced slots like valid ones by updating
inactive_since with each sync cycle. This way, we can keep the "last
sync time" in the inactive_since. However, this could confuse users
when "invalidation_reason=inactive_timeout" is set for a synced slot
on standby but inactive_since would reflect sync time rather than the
time slot became inactive. IIUC, on the primary, when
invalidation_reason=inactive_timeout for a slot, the inactive_since
represents the actual time the slot became inactive before getting
invalidated, unless the primary is restarted.

Thoughts?

--
Thanks,
Nisha

  272. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-13T09:29:25Z

    On Thu, 7 Nov 2024 at 15:33, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Please find the attached v46 patch having changes for the above review
> > comments and your test review comments and Shveta's review comments.
> >
> Hi,
>
> I’ve reviewed this thread and am interested in working on the
> remaining tasks and comments, as well as the future review comments.
> However, Bharath, please let me know if you'd prefer to continue with
> it.
>
> Attached the rebased v47 patch, which also addresses Peter’s comments
> #2, #3, and #4 at [1]. I will try addressing other comments as well in
> next versions.

The following crash occurs while upgrading:
2024-11-13 14:19:45.955 IST [44539] LOG:  checkpoint starting: time
TRAP: failed Assert("!(*invalidated && SlotIsLogical(s) &&
IsBinaryUpgrade)"), File: "slot.c", Line: 1793, PID: 44539
postgres: checkpointer (ExceptionalCondition+0xbb)[0x555555e305bd]
postgres: checkpointer (+0x63ab04)[0x555555b8eb04]
postgres: checkpointer
(InvalidateObsoleteReplicationSlots+0x149)[0x555555b8ee5f]
postgres: checkpointer (CheckPointReplicationSlots+0x267)[0x555555b8f125]
postgres: checkpointer (+0x1f3ee8)[0x555555747ee8]
postgres: checkpointer (CreateCheckPoint+0x78f)[0x5555557475ee]
postgres: checkpointer (CheckpointerMain+0x632)[0x555555b2f1e7]
postgres: checkpointer (postmaster_child_launch+0x119)[0x555555b30892]
postgres: checkpointer (+0x5e2dc8)[0x555555b36dc8]
postgres: checkpointer (PostmasterMain+0x14bd)[0x555555b33647]
postgres: checkpointer (+0x487f2e)[0x5555559dbf2e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ffff6c29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ffff6c29e40]
postgres: checkpointer (_start+0x25)[0x555555634c25]
2024-11-13 14:19:45.967 IST [44538] LOG:  checkpointer process (PID
44539) was terminated by signal 6: Aborted

This can happen in the following case:
1) Setup a logical replication cluster with enough data so that it
will take at least few minutes to upgrade
2) Stop the publisher node
3) Configure replication_slot_inactive_timeout and checkpoint_timeout
to 30 seconds
4) Upgrade the publisher node.

This is happening because logical replication slots are getting
invalidated during upgrade and there is an assertion which checks that
the slots are not invalidated.
I feel this can be fixed by having a function similar to
check_max_slot_wal_keep_size which will make sure that
replication_slot_inactive_timeout is 0 during upgrade.

Regards,
Vignesh

  273. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-13T09:30:28Z

    Please find the v48 patch attached.

On Thu, Sep 19, 2024 at 9:40 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> When we promote hot standby with synced logical slots to become new
> primary, the logical slots are never invalidated with
> 'inactive_timeout' on new primary.  It seems the check in
> SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
> invalidation of slots on primary even if they are marked as 'synced'.

fixed.

> I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
> Once all these are addressed, I can continue reviewing further.
>

Fixed issues reported in [1], [2].

[1]: https://www.postgresql.org/message-id/CAJpy0uAwxc49Dz6t%3D-y_-z-MU%2BA4RWX4BR3Zri_jj2qgGMq_8g%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/CAJpy0uC6nN3SLbEuCvz7-CpaPdNdXxH%3DfeW5MhYQch-JWV0tLg%40mail.gmail.com

  274. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-13T09:35:01Z

    On Wed, Sep 18, 2024 at 12:22 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > Hi,
> >
> >
> > Please find the attached v46 patch having changes for the above review
> > comments and your test review comments and Shveta's review comments.
> >
>
> Thanks for addressing comments.
>
> Is there a reason that we don't support this invalidation on hot
> standby for non-synced slots? Shouldn't we support this time-based
> invalidation there too just like other invalidations?
>

I don’t see any reason to *not* support this invalidation on hot
standby for non-synced slots. Therefore, I’ve added the same in v48.

--
Thanks,
Nisha

  275. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-11-13T23:58:45Z

    Hi Nisha.

Thanks for the recent patch updates. Here are my review comments for
the latest patch v48-0001.

======
Commit message

1.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage for instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.

~

What do the words "for instance" mean here? Did it mean "per instance"
or "(for example)" or something else?

======
doc/src/sgml/system-views.sgml

2.
       <para>
         The time since the slot has become inactive.
-        <literal>NULL</literal> if the slot is currently being used.
-        Note that for slots on the standby that are being synced from a
+        <literal>NULL</literal> if the slot is currently being used. Once the
+        slot is invalidated, this value will remain unchanged until we shutdown
+        the server. Note that for slots on the standby that are being
synced from a
         primary server (whose <structfield>synced</structfield> field is
         <literal>true</literal>), the

Is this change related to the new inactivity timeout feature or are
you just clarifying the existing behaviour of the 'active_since'
field.

Note there is already another thread [1] created to patch/clarify this
same field. So if you are just clarifying existing behavior then IMO
it would be better if you can to try and get your desired changes
included there quickly before that other patch gets pushed.

~~~

3.
+         <para>
+          <literal>inactive_timeout</literal> means that the slot has been
+          inactive for longer than the amount of time specified by the
+          <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+         </para>

Maybe there is a slightly shorter/simpler way to express this. For example,

BEFORE
inactive_timeout means that the slot has been inactive for longer than
the amount of time specified by the replication_slot_inactive_timeout
parameter.

SUGGESTION
inactive_timeout means that the slot has remained inactive beyond the
duration specified by the replication_slot_inactive_timeout parameter.

======
src/backend/replication/slot.c

4.
+int replication_slot_inactive_timeout = 0;

IMO it would be more informative to give the units in the variable
name (but not in the GUC name). e.g.
'replication_slot_inactive_timeout_secs'.

~~~

ReplicationSlotAcquire:

5.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
  */
 void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)

This function comment makes it seem like "invalidated previously"
might mean *any* kind of invalidation, but later in the body of the
function we find the logic is really only used for inactive timeout.

+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */

So, I think a better name for that parameter might be
'error_if_inactive_timeout'

OTOH, if it really is supposed to erro for *any* kind of invalidation
then there needs to be more ereports.

~~~

6.
+ errdetail("This slot has been invalidated because it was inactive
for longer than the amount of time specified by \"%s\".",

This errdetail message seems quite long. I think it can be shortened
like below and still retain exactly the same meaning:

BEFORE:
This slot has been invalidated because it was inactive for longer than
the amount of time specified by \"%s\".

SUGGESTION:
This slot has been invalidated due to inactivity exceeding the time
limit set by "%s".

~~~

ReportSlotInvalidation:

7.
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for longer than the amount of
time specified by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;

Here also as in the above review comment #6 I think the message can be
shorter and still say the same thing

BEFORE:
_("The slot has been inactive since %s for longer than the amount of
time specified by \"%s\"."),

SUGGESTION:
_("The slot has been inactive since %s, exceeding the time limit set
by \"%s\"."),

~~~

SlotInactiveTimeoutCheckAllowed:

8.
+/*
+ * Is this replication slot allowed for inactive timeout invalidation check?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. Server is in recovery and slot is not being synced from the primary
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */

8a.
Somehow that first sentence seems strange. Would it be better to write it like:

SUGGESTION
Can this replication slot timeout due to inactivity?

~

8b.
AFAICT that reason 3 ("Server is in recovery and slot is not being
synced from the primary") seems not quite worded right...

Should it say more like:
The slot is not being synced from the primary while the server is in recovery

or maybe like:
The slot is not currently being synced from the primary (e.g. not
'synced' is true when server is in recovery)

~

8c.
Similarly, I think something about that "Note that the inactive
timeout invalidation mechanism is not applicable..." paragraph needs
tweaking because IMO that should also now be saying something about
'RecoveryInProgress'.

~~~

9.
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)

Maybe the function name should be 'IsSlotInactiveTimeoutPossible' or
something better.

~~~

InvalidatePossiblyObsoleteSlot:

10.
  break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */

Since there are no other blank lines anywhere in this switch, the
introduction of this one in v48 looks out of place to me. IMO it would
be more readable if a blank line followed each/every of the breaks,
but then that is not a necessary change for this patch so...

~~~

11.
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);

Given this assertion, does it mean that "(s->active_pid == 0)" should
have been another condition done up-front in the function
'SlotInactiveTimeoutCheckAllowed'?

~~~

12.
  /*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately.  Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
  */
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))

I wasn't sure how this change belongs to this patch, because the logic
of the previous review comment said for the case of invalidation due
to inactivity that active_id must be 0. e.g. Assert(s->active_pid ==
0);

~~~

RestoreSlotFromDisk:

13.
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;

In v47 this assignment used to call the function
'ReplicationSlotSetInactiveSince'. I recognise there is a very subtle
difference between direct assignment and the function, because the
function will skip assignment if the slot is already invalidated.
Anyway, if you are *deliberately* not wanting to call
ReplicationSlotSetInactiveSince here then I think this assignment
should be commented to explain the reason why not, otherwise someone
in the future might be tempted to think it was just an oversight and
add the call back in that you don't want.

======
src/test/recovery/t/050_invalidate_slots.pl

14.
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby. So, we must not see invalidation message in server
+# log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+   WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+

But, now, we are confirming this by another way -- not checking the
logs here, so the comment "So, we must not see invalidation message in
server log." is no longer appropriate here.

======
[1] https://www.postgresql.org/message-id/flat/CAA4eK1JQFdssaBBh-oQskpKM-UpG8jPyUdtmGWa_0qCDy%2BK7_A%40mail.gmail.com#ab98379f220288ed40d34f8c2a21cf96

Kind Regards,
Peter Smith.
Fujitsu Australia

  276. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-14T03:44:40Z

    On Wed, 13 Nov 2024 at 15:00, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Please find the v48 patch attached.
>
> On Thu, Sep 19, 2024 at 9:40 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > When we promote hot standby with synced logical slots to become new
> > primary, the logical slots are never invalidated with
> > 'inactive_timeout' on new primary.  It seems the check in
> > SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
> > invalidation of slots on primary even if they are marked as 'synced'.
>
> fixed.
>
> > I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
> > Once all these are addressed, I can continue reviewing further.
> >
>
> Fixed issues reported in [1], [2].

Few comments:
1) Since we don't change the value of now in
ReplicationSlotSetInactiveSince, the function parameter can be passed
by value:
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+                                                               bool
acquire_lock)
+{
+       if (s->data.invalidated != RS_INVAL_NONE)
+               return;
+
+       if (acquire_lock)
+               SpinLockAcquire(&s->mutex);
+
+       s->inactive_since = *now;

2) Currently it allows a minimum value of less than 1 second like in
milliseconds, I feel we can have some minimum value at least something
like checkpoint_timeout:
diff --git a/src/backend/utils/misc/guc_tables.c
b/src/backend/utils/misc/guc_tables.c
index 8a67f01200..367f510118 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
                NULL, NULL, NULL
        },

+       {
+               {"replication_slot_inactive_timeout", PGC_SIGHUP,
REPLICATION_SENDING,
+                       gettext_noop("Sets the amount of time a
replication slot can remain inactive before "
+                                                "it will be invalidated."),
+                       NULL,
+                       GUC_UNIT_S
+               },
+               &replication_slot_inactive_timeout,
+               0, 0, INT_MAX,
+               NULL, NULL, NULL
+       },

3) Since SlotInactiveTimeoutCheckAllowed check is just done above and
the current time has been retrieved can we used "now" variable instead
of SlotInactiveTimeoutCheckAllowed again second time:
@@ -1651,6 +1713,26 @@
InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
                                        if (SlotIsLogical(s))
                                                invalidation_cause = cause;
                                        break;
+                               case RS_INVAL_INACTIVE_TIMEOUT:
+
+                                       /*
+                                        * Check if the slot needs to
be invalidated due to
+                                        *
replication_slot_inactive_timeout GUC.
+                                        */
+                                       if
(SlotInactiveTimeoutCheckAllowed(s) &&
+
TimestampDifferenceExceeds(s->inactive_since, now,
+
                            replication_slot_inactive_timeout * 1000))
+                                       {
+                                               invalidation_cause = cause;
+                                               inactive_since =
s->inactive_since;

4) I'm not sure if this change required by this patch or is it a
general optimization, if it is required for this patch we can detail
the comments:
@@ -2208,6 +2328,7 @@ RestoreSlotFromDisk(const char *name)
        bool            restored = false;
        int                     readBytes;
        pg_crc32c       checksum;
+       TimestampTz now;

        /* no need to lock here, no concurrent access allowed yet */

@@ -2368,6 +2489,9 @@ RestoreSlotFromDisk(const char *name)
                                                NameStr(cp.slotdata.name)),
                                 errhint("Change \"wal_level\" to be
\"replica\" or higher.")));

+       /* Use same inactive_since time for all slots */
+       now = GetCurrentTimestamp();
+
        /* nothing can be active yet, don't lock anything */
        for (i = 0; i < max_replication_slots; i++)
        {
@@ -2400,7 +2524,7 @@ RestoreSlotFromDisk(const char *name)
                 * slot from the disk into memory. Whoever acquires
the slot i.e.
                 * makes the slot active will reset it.
                 */
-               slot->inactive_since = GetCurrentTimestamp();
+               slot->inactive_since = now;

5) Why should the slot invalidation be updated during shutdown,
shouldn't the inactive_since value be intact during shutdown?
-        <literal>NULL</literal> if the slot is currently being used.
-        Note that for slots on the standby that are being synced from a
+        <literal>NULL</literal> if the slot is currently being used. Once the
+        slot is invalidated, this value will remain unchanged until we shutdown
+        the server. Note that for slots on the standby that are being
synced from a

6) New Style of ereport does not need braces around errcode, it can be
changed similarly:
+       if (error_if_invalid &&
+               s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+       {
+               Assert(s->inactive_since > 0);
+               ereport(ERROR,
+
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+                                errmsg("can no longer get changes
from replication slot \"%s\"",
+                                               NameStr(s->data.name)),
+                                errdetail("This slot has been
invalidated because it was inactive for longer than the amount of time
specified by \"%s\".",
+
"replication_slot_inactive_timeout")));

Regards,
Vignesh

  277. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-19T07:12:55Z

    Attached is the v49 patch set:
- Fixed the bug reported in [1].
- Addressed comments in [2] and [3].

I've split the patch into two, implementing the suggested idea in
comment #5 of [2] separately in 001:

Patch-001: Adds additional error reports (for all invalidation types)
in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
true.
Patch-002: The original patch with comments addressed.

~~~~

[1] https://www.postgresql.org/message-id/CALDaNm2mwkVFLfe8pLcU1W5Oy1vRr1Wzp53XGV08kr4Z2%3DSJpA%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPt6s-qNPdxH5%3D-fr2QKLEv0h16sQ8EvLiGJ-SdQNS6pbw%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CALDaNm2VQW_gpOJ-QWkEA_h18DN31ELEz2_7QmwWCAg9%3DZew4A%40mail.gmail.com

--
Thanks,
Nisha

  278. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-19T07:17:37Z

    On Thu, Nov 14, 2024 at 5:29 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha.
>
> Thanks for the recent patch updates. Here are my review comments for
> the latest patch v48-0001.
>

Thank you for the review. Comments are addressed in v49 version.
Below is my response to comments that may require further discussion.

> ======
> doc/src/sgml/system-views.sgml
>
> 2.
>        <para>
>          The time since the slot has become inactive.
> -        <literal>NULL</literal> if the slot is currently being used.
> -        Note that for slots on the standby that are being synced from a
> +        <literal>NULL</literal> if the slot is currently being used. Once the
> +        slot is invalidated, this value will remain unchanged until we shutdown
> +        the server. Note that for slots on the standby that are being
> synced from a
>          primary server (whose <structfield>synced</structfield> field is
>          <literal>true</literal>), the
>
> Is this change related to the new inactivity timeout feature or are
> you just clarifying the existing behaviour of the 'active_since'
> field.
>

Yes, this patch introduces inactive_timeout invalidation and prevents
updates to inactive_since for invalid slots. Only a node restart can
modify it,  so, I believe we should retain these lines in this patch.

> Note there is already another thread [1] created to patch/clarify this
> same field. So if you are just clarifying existing behavior then IMO
> it would be better if you can to try and get your desired changes
> included there quickly before that other patch gets pushed.
>

Thanks for the reference, I have posted my suggestion on the thread.

>
> ReplicationSlotAcquire:
>
> 5.
> + *
> + * An error is raised if error_if_invalid is true and the slot has been
> + * invalidated previously.
>   */
>  void
> -ReplicationSlotAcquire(const char *name, bool nowait)
> +ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
>
> This function comment makes it seem like "invalidated previously"
> might mean *any* kind of invalidation, but later in the body of the
> function we find the logic is really only used for inactive timeout.
>
> + /*
> + * An error is raised if error_if_invalid is true and the slot has been
> + * previously invalidated due to inactive timeout.
> + */
>
> So, I think a better name for that parameter might be
> 'error_if_inactive_timeout'
>
> OTOH, if it really is supposed to erro for *any* kind of invalidation
> then there needs to be more ereports.
>

+1 to the idea.
I have created a separate patch v49-0001 adding more ereports for all
kinds of invalidations.

> ~~~
> SlotInactiveTimeoutCheckAllowed:
>
> 8.
> +/*
> + * Is this replication slot allowed for inactive timeout invalidation check?
> + *
> + * Inactive timeout invalidation is allowed only when:
> + *
> + * 1. Inactive timeout is set
> + * 2. Slot is inactive
> + * 3. Server is in recovery and slot is not being synced from the primary
> + *
> + * Note that the inactive timeout invalidation mechanism is not
> + * applicable for slots on the standby server that are being synced
> + * from the primary server (i.e., standby slots having 'synced' field 'true').
> + * Synced slots are always considered to be inactive because they don't
> + * perform logical decoding to produce changes.
> + */
>
> 8a.
> Somehow that first sentence seems strange. Would it be better to write it like:
>
> SUGGESTION
> Can this replication slot timeout due to inactivity?
>

I feel the suggestion is not very clear on the purpose of the
function, This function doesn't check inactivity or decide slot
timeout invalidation. It only pre-checks if the slot qualifies for an
inactivity check, which the caller will perform.
As I have changed function name too as per commnet#9, I used the following  -
"Is inactive timeout invalidation possible for this replication slot?"
Thoughts?

> ~
> 8c.
> Similarly, I think something about that "Note that the inactive
> timeout invalidation mechanism is not applicable..." paragraph needs
> tweaking because IMO that should also now be saying something about
> 'RecoveryInProgress'.
>

'RecoveryInProgress' check indicates that the server is a standby, and
the mentioned paragraph uses the term "standby" to describe the
condition. It seems unnecessary to mention RecoveryInProgress
separately.

> ~~~
>
> InvalidatePossiblyObsoleteSlot:
>
> 10.
>   break;
> + case RS_INVAL_INACTIVE_TIMEOUT:
> +
> + /*
> + * Check if the slot needs to be invalidated due to
> + * replication_slot_inactive_timeout GUC.
> + */
>
> Since there are no other blank lines anywhere in this switch, the
> introduction of this one in v48 looks out of place to me.

pgindent automatically added this blank line after 'case
RS_INVAL_INACTIVE_TIMEOUT'.

> IMO it would
> be more readable if a blank line followed each/every of the breaks,
> but then that is not a necessary change for this patch so...
>

Since it's not directly related to the patch, I feel it might be best
to leave it as is for now.

> ~~~
>
> 11.
> + /*
> + * Invalidation due to inactive timeout implies that
> + * no one is using the slot.
> + */
> + Assert(s->active_pid == 0);
>
> Given this assertion, does it mean that "(s->active_pid == 0)" should
> have been another condition done up-front in the function
> 'SlotInactiveTimeoutCheckAllowed'?
>

I don't think it's a good idea to check (s->active_pid == 0) upfront,
before the timeout-invalidation check. AFAIU, this assertion is meant
to ensure active_pid = 0 only if the slot is going to be invalidated,
i.e., when the following condition is true:

TimestampDifferenceExceeds(s->inactive_since, now,

replication_slot_inactive_timeout_sec * 1000)

Thoughts? Open to others' opinions too.

> ~~~
>
> 12.
>   /*
> - * If the slot can be acquired, do so and mark it invalidated
> - * immediately.  Otherwise we'll signal the owning process, below, and
> - * retry.
> + * If the slot can be acquired, do so and mark it as invalidated. If
> + * the slot is already ours, mark it as invalidated. Otherwise, we'll
> + * signal the owning process below and retry.
>   */
> - if (active_pid == 0)
> + if (active_pid == 0 ||
> + (MyReplicationSlot == s &&
> + active_pid == MyProcPid))
>
> I wasn't sure how this change belongs to this patch, because the logic
> of the previous review comment said for the case of invalidation due
> to inactivity that active_id must be 0. e.g. Assert(s->active_pid ==
> 0);
>

I don't fully understand the purpose of this change yet. I'll look
into it further and get back.

> ~~~
>
> RestoreSlotFromDisk:
>
> 13.
> - slot->inactive_since = GetCurrentTimestamp();
> + slot->inactive_since = now;
>
> In v47 this assignment used to call the function
> 'ReplicationSlotSetInactiveSince'. I recognise there is a very subtle
> difference between direct assignment and the function, because the
> function will skip assignment if the slot is already invalidated.
> Anyway, if you are *deliberately* not wanting to call
> ReplicationSlotSetInactiveSince here then I think this assignment
> should be commented to explain the reason why not, otherwise someone
> in the future might be tempted to think it was just an oversight and
> add the call back in that you don't want.
>

Added comment saying avoid using ReplicationSlotSetInactiveSince()
here as it will skip the invalid slots.

~~~~

--
Thanks,
Nisha

  279. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-19T07:20:51Z

    On Thu, Nov 14, 2024 at 9:14 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 13 Nov 2024 at 15:00, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Please find the v48 patch attached.
> >
> > On Thu, Sep 19, 2024 at 9:40 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > When we promote hot standby with synced logical slots to become new
> > > primary, the logical slots are never invalidated with
> > > 'inactive_timeout' on new primary.  It seems the check in
> > > SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
> > > invalidation of slots on primary even if they are marked as 'synced'.
> >
> > fixed.
> >
> > > I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
> > > Once all these are addressed, I can continue reviewing further.
> > >
> >
> > Fixed issues reported in [1], [2].
>
> Few comments:

Thanks for the review.

>
> 2) Currently it allows a minimum value of less than 1 second like in
> milliseconds, I feel we can have some minimum value at least something
> like checkpoint_timeout:
> diff --git a/src/backend/utils/misc/guc_tables.c
> b/src/backend/utils/misc/guc_tables.c
> index 8a67f01200..367f510118 100644
> --- a/src/backend/utils/misc/guc_tables.c
> +++ b/src/backend/utils/misc/guc_tables.c
> @@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
>                 NULL, NULL, NULL
>         },
>
> +       {
> +               {"replication_slot_inactive_timeout", PGC_SIGHUP,
> REPLICATION_SENDING,
> +                       gettext_noop("Sets the amount of time a
> replication slot can remain inactive before "
> +                                                "it will be invalidated."),
> +                       NULL,
> +                       GUC_UNIT_S
> +               },
> +               &replication_slot_inactive_timeout,
> +               0, 0, INT_MAX,
> +               NULL, NULL, NULL
> +       },
>

Currently, the feature is disabled by default when
replication_slot_inactive_timeout = 0. However, if we set a minimum
value, the default_val cannot be less than min_val, making it
impossible to use 0 to disable the feature.
Thoughts or any suggestions?

>
> 4) I'm not sure if this change required by this patch or is it a
> general optimization, if it is required for this patch we can detail
> the comments:
> @@ -2208,6 +2328,7 @@ RestoreSlotFromDisk(const char *name)
>         bool            restored = false;
>         int                     readBytes;
>         pg_crc32c       checksum;
> +       TimestampTz now;
>
>         /* no need to lock here, no concurrent access allowed yet */
>
> @@ -2368,6 +2489,9 @@ RestoreSlotFromDisk(const char *name)
>                                                 NameStr(cp.slotdata.name)),
>                                  errhint("Change \"wal_level\" to be
> \"replica\" or higher.")));
>
> +       /* Use same inactive_since time for all slots */
> +       now = GetCurrentTimestamp();
> +
>         /* nothing can be active yet, don't lock anything */
>         for (i = 0; i < max_replication_slots; i++)
>         {
> @@ -2400,7 +2524,7 @@ RestoreSlotFromDisk(const char *name)
>                  * slot from the disk into memory. Whoever acquires
> the slot i.e.
>                  * makes the slot active will reset it.
>                  */
> -               slot->inactive_since = GetCurrentTimestamp();
> +               slot->inactive_since = now;
>

After removing the "ReplicationSlotSetInactiveSince" from here, it
became irrelevant to this patch. Now, it is a general optimization to
set the same timestamp for all slots while restoring from disk. I have
added a few comments as per Peter's suggestion.

> 5) Why should the slot invalidation be updated during shutdown,
> shouldn't the inactive_since value be intact during shutdown?
> -        <literal>NULL</literal> if the slot is currently being used.
> -        Note that for slots on the standby that are being synced from a
> +        <literal>NULL</literal> if the slot is currently being used. Once the
> +        slot is invalidated, this value will remain unchanged until we shutdown
> +        the server. Note that for slots on the standby that are being
> synced from a
>

The "inactive_since" data of a slot is not stored on disk, so the
older value cannot be restored after a restart.

--
Thanks,
Nisha

  280. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-19T10:23:35Z

    On Tue, 19 Nov 2024 at 12:43, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Attached is the v49 patch set:
> - Fixed the bug reported in [1].
> - Addressed comments in [2] and [3].
>
> I've split the patch into two, implementing the suggested idea in
> comment #5 of [2] separately in 001:
>
> Patch-001: Adds additional error reports (for all invalidation types)
> in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> true.
> Patch-002: The original patch with comments addressed.

Few comments:
1) I felt this check in wait_for_slot_invalidation is not required as
there is a call to trigger_slot_invalidation which sleeps for
inactive_timeout seconds and ensures checkpoint is triggered, also the
test passes without this:
+       # Wait for slot to become inactive
+       $node->poll_query_until(
+               'postgres', qq[
+               SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+                       WHERE slot_name = '$slot' AND active = 'f' AND
+                                 inactive_since IS NOT NULL;
+       ])
+         or die
+         "Timed out while waiting for slot $slot to become inactive
on node $node_name";

2) Instead of calling this in a loop, won't it be enough to call
checkpoint only once explicitly:
+       for (my $i = 0; $i < 10 *
$PostgreSQL::Test::Utils::timeout_default; $i++)
+       {
+               $node->safe_psql('postgres', "CHECKPOINT");
+               if ($node->log_contains(
+                               "invalidating obsolete replication
slot \"$slot\"", $offset))
+               {
+                       $invalidated = 1;
+                       last;
+               }
+               usleep(100_000);
+       }
+       ok($invalidated,
+               "check that slot $slot invalidation has been logged on
node $node_name"
+       );

3) Since pg_sync_replication_slots is a sync call, we can directly use
"is( $standby1->safe_psql('postgres', SELECT COUNT(slot_name) = 1 FROM
pg_replication_slots..." instead of poll_query_until:
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+$standby1->poll_query_until(
+       'postgres', qq[
+       SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+               WHERE slot_name = 'sync_slot1' AND
+               invalidation_reason = 'inactive_timeout';
+])
+  or die
+  "Timed out while waiting for sync_slot1 invalidation to be synced
on standby";

4) Since this variable is being referred to at many places, how about
changing it to inactive_timeout_1s so that it is easier while
reviewing across many places:
# Set timeout GUC on the standby to verify that the next checkpoint will not
# invalidate synced slots.
my $inactive_timeout = 1;

5) Since we have already tested invalidation of logical replication
slot 'sync_slot1' above, this test might not be required:
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber slot due to inactive timeout.
+
+my $publisher = $primary;
+
+# Prepare for test
+$publisher->safe_psql(
+       'postgres', qq[
+    ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;

Regards,
Vignesh

  281. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-20T07:59:29Z

    On Tue, 19 Nov 2024 at 12:43, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Attached is the v49 patch set:
> - Fixed the bug reported in [1].
> - Addressed comments in [2] and [3].
>
> I've split the patch into two, implementing the suggested idea in
> comment #5 of [2] separately in 001:
>
> Patch-001: Adds additional error reports (for all invalidation types)
> in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> true.
> Patch-002: The original patch with comments addressed.

This Assert can fail:
+                                       /*
+                                        * Check if the slot needs to
be invalidated due to
+                                        *
replication_slot_inactive_timeout GUC.
+                                        */
+                                       if (now &&
+
TimestampDifferenceExceeds(s->inactive_since, now,
+
                            replication_slot_inactive_timeout_sec *
1000))
+                                       {
+                                               invalidation_cause = cause;
+                                               inactive_since =
s->inactive_since;
+
+                                               /*
+                                                * Invalidation due to
inactive timeout implies that
+                                                * no one is using the slot.
+                                                */
+                                               Assert(s->active_pid == 0);

With the following scenario:
Set replication_slot_inactive_timeout to 10 seconds
-- Create a slot
postgres=# select pg_create_logical_replication_slot ('test',
'pgoutput', true, true);
 pg_create_logical_replication_slot
------------------------------------
 (test,0/1748068)
(1 row)

-- Wait for 10 seconds and execute checkpoint
postgres=# checkpoint;
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process
exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and
repeat your command.
server closed the connection unexpectedly

The assert fails:
#5  0x00005b074f0c922f in ExceptionalCondition
(conditionName=0x5b074f2f0b4c "s->active_pid == 0",
fileName=0x5b074f2f0010 "slot.c", lineNumber=1762) at assert.c:66
#6  0x00005b074ee26ead in InvalidatePossiblyObsoleteSlot
(cause=RS_INVAL_INACTIVE_TIMEOUT, s=0x740925361780, oldestLSN=0,
dboid=0, snapshotConflictHorizon=0, invalidated=0x7fffaee87e63) at
slot.c:1762
#7  0x00005b074ee273b2 in InvalidateObsoleteReplicationSlots
(cause=RS_INVAL_INACTIVE_TIMEOUT, oldestSegno=0, dboid=0,
snapshotConflictHorizon=0) at slot.c:1952
#8  0x00005b074ee27678 in CheckPointReplicationSlots
(is_shutdown=false) at slot.c:2061
#9  0x00005b074e9dfda7 in CheckPointGuts (checkPointRedo=24412528,
flags=108) at xlog.c:7513
#10 0x00005b074e9df4ad in CreateCheckPoint (flags=108) at xlog.c:7179
#11 0x00005b074edc6bfc in CheckpointerMain (startup_data=0x0,
startup_data_len=0) at checkpointer.c:463

Regards,
Vignesh

  282. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-21T05:06:32Z

    On Tue, 19 Nov 2024 at 12:51, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Thu, Nov 14, 2024 at 9:14 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 13 Nov 2024 at 15:00, Nisha Moond <nisha.moond412@gmail.com> wrote:
> > >
> > > Please find the v48 patch attached.
> > >
> > 2) Currently it allows a minimum value of less than 1 second like in
> > milliseconds, I feel we can have some minimum value at least something
> > like checkpoint_timeout:
> > diff --git a/src/backend/utils/misc/guc_tables.c
> > b/src/backend/utils/misc/guc_tables.c
> > index 8a67f01200..367f510118 100644
> > --- a/src/backend/utils/misc/guc_tables.c
> > +++ b/src/backend/utils/misc/guc_tables.c
> > @@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
> >                 NULL, NULL, NULL
> >         },
> >
> > +       {
> > +               {"replication_slot_inactive_timeout", PGC_SIGHUP,
> > REPLICATION_SENDING,
> > +                       gettext_noop("Sets the amount of time a
> > replication slot can remain inactive before "
> > +                                                "it will be invalidated."),
> > +                       NULL,
> > +                       GUC_UNIT_S
> > +               },
> > +               &replication_slot_inactive_timeout,
> > +               0, 0, INT_MAX,
> > +               NULL, NULL, NULL
> > +       },
> >
>
> Currently, the feature is disabled by default when
> replication_slot_inactive_timeout = 0. However, if we set a minimum
> value, the default_val cannot be less than min_val, making it
> impossible to use 0 to disable the feature.
> Thoughts or any suggestions?

We could implement this similarly to how the vacuum_buffer_usage_limit
GUC is handled. Setting the value to 0 would allow the operation to
use any amount of shared_buffers. Otherwise, valid sizes would range
from 128 kB to 16 GB. Similarly, we can modify
check_replication_slot_inactive_timeout to behave in the same way as
check_vacuum_buffer_usage_limit function.

Regards,
Vignesh

  283. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-21T12:05:01Z

    On Wed, Nov 20, 2024 at 1:29 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 19 Nov 2024 at 12:43, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Attached is the v49 patch set:
> > - Fixed the bug reported in [1].
> > - Addressed comments in [2] and [3].
> >
> > I've split the patch into two, implementing the suggested idea in
> > comment #5 of [2] separately in 001:
> >
> > Patch-001: Adds additional error reports (for all invalidation types)
> > in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> > true.
> > Patch-002: The original patch with comments addressed.
>
> This Assert can fail:
>

Attached v50 patch-set addressing review comments in [1] and [2].

Regarding the assert issue reported in [2]:
- For temporary replication slots, the current session's pid serves as
the active_pid for the slot, which is expected behavior.
- Therefore, the ASSERT has been removed in v50. Now, if a temporary
slot qualifies for a timeout invalidation, the holding process will be
terminated, and the slot will be invalidated.

[1] https://www.postgresql.org/message-id/CALDaNm2UUTfJczjR-rEQwKgmx%3DiFnuMnR1cXv7ccB%2BO9P15mYg%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CALDaNm0g86wD2%3DbQdFOy0smsP0MZWyz0CUqXej%3DQi-hCEeqkag%40mail.gmail.com

--
Thanks,
Nisha

  284. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-22T12:13:24Z

    On Thu, 21 Nov 2024 at 17:35, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Wed, Nov 20, 2024 at 1:29 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, 19 Nov 2024 at 12:43, Nisha Moond <nisha.moond412@gmail.com> wrote:
> > >
> > > Attached is the v49 patch set:
> > > - Fixed the bug reported in [1].
> > > - Addressed comments in [2] and [3].
> > >
> > > I've split the patch into two, implementing the suggested idea in
> > > comment #5 of [2] separately in 001:
> > >
> > > Patch-001: Adds additional error reports (for all invalidation types)
> > > in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> > > true.
> > > Patch-002: The original patch with comments addressed.
> >
> > This Assert can fail:
> >
>
> Attached v50 patch-set addressing review comments in [1] and [2].

We are setting inactive_since when the replication slot is released.
We are marking the slot as inactive only if it has been released.
However, there's a scenario where the network connection between the
publisher and subscriber may be lost where the replication slot is not
released, but no changes are replicated due to the network problem. In
this case, no updates would occur in the replication slot for a period
exceeding the replication_slot_inactive_timeout.
Should we invalidate these replication slots as well, or is it
intentionally left out?

Regards,
Vignesh

  285. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-11-25T06:06:03Z

    Hi Nisha,

Here are my review comments for the patch v50-0001.

======
Commit message

1.
In ReplicationSlotAcquire(), raise an error for invalid slots if caller
specify error_if_invalid=true.

/caller/the caller/
/specify/specifies/

======
src/backend/replication/slot.c

ReplicationSlotAcquire:

2.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
  */
 void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)

The "has been invalidated previously." sounds a bit tricky. Do you just mean:

"An error is raised if error_if_invalid is true and the slot is found
to be invalid."

~

3.
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated.
+ */

(ditto previous comment)

~

4.
+ appendStringInfo(&err_detail, _("This slot has been invalidated because "));
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ appendStringInfo(&err_detail, _("wal_level is insufficient for slot."));
+ break;

4a.
I suspect that building the errdetail in 2 parts like this will be
troublesome for the translators of some languages. Probably it is
safer to have the entire errdetail for each case.

~

4b.
By convention, I think the GUC "wal_level" should be double-quoted in
the message.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  286. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-11-27T03:09:15Z

    Hi Nisha,

Here are some review comments for the patch v50-0002.

======
src/backend/replication/slot.c

InvalidatePossiblyObsoleteSlot:

1.
+ if (now &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+    replication_slot_inactive_timeout_sec * 1000))

Previously this was using an additional call to SlotInactiveTimeoutCheckAllowed:

+ if (SlotInactiveTimeoutCheckAllowed(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+    replication_slot_inactive_timeout * 1000))

Is it OK to skip that call? e.g. can the slot fields possibly change
between assigning the 'now' and acquiring the mutex? If not, then the
current code is fine. The only reason for asking is because it is
slightly suspicious that it was not done this "easy" way in the first
place.

~~~

check_replication_slot_inactive_timeout:

2.
+/*
+ * GUC check_hook for replication_slot_inactive_timeout
+ *
+ * We don't allow the value of replication_slot_inactive_timeout other than 0
+ * during the binary upgrade.
+ */

The "We don't allow..." sentence seems like a backward way of saying:
The value of replication_slot_inactive_timeout must be set to 0 during
the binary upgrade.

======
src/test/recovery/t/050_invalidate_slots.pl

3.
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby.

What does "on its own" mean here? Do you mean it won't get invalidated
unless the invalidation state is propagated from the primary? Maybe
the comment can be clearer.

~

4.
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout_1s) = @_;
+ my $node_name = $node->name;
+

It was OK to change the variable name to 'inactive_timeout_1s' outside
of here, but within the subroutine, I don't think it is appropriate
because this is a parameter that potentially could have any value.

~

5.
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout_1s) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;

It was OK to change the variable name to 'inactive_timeout_1s' outside
of here, but within the subroutine, I don't think it is appropriate
because this is a parameter that potentially could have any value.

~

6.
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout_1s + 1);
+
+ # Run a checkpoint
+ $node->safe_psql('postgres', "CHECKPOINT");

Since you are not doing multiple checkpoints anymore, it looks like
that "Give enough time..." comment needs updating.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  287. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-27T10:54:49Z

    On Wed, Nov 27, 2024 at 8:39 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha,
>
> Here are some review comments for the patch v50-0002.
>
> ======
> src/backend/replication/slot.c
>
> InvalidatePossiblyObsoleteSlot:
>
> 1.
> + if (now &&
> + TimestampDifferenceExceeds(s->inactive_since, now,
> +    replication_slot_inactive_timeout_sec * 1000))
>
> Previously this was using an additional call to SlotInactiveTimeoutCheckAllowed:
>
> + if (SlotInactiveTimeoutCheckAllowed(s) &&
> + TimestampDifferenceExceeds(s->inactive_since, now,
> +    replication_slot_inactive_timeout * 1000))
>
> Is it OK to skip that call? e.g. can the slot fields possibly change
> between assigning the 'now' and acquiring the mutex? If not, then the
> current code is fine. The only reason for asking is because it is
> slightly suspicious that it was not done this "easy" way in the first
> place.
>
Good catch! While the mutex was being acquired right after the now
assignment, there was a rare chance of another process modifying the
slot in the meantime. So, I reverted the change in v51. To optimize
the SlotInactiveTimeoutCheckAllowed() call, it's sufficient to check
it here instead of during the 'now' assignment.

Attached v51 patch-set addressing all comments in [1] and [2].

[1] https://www.postgresql.org/message-id/CAHut%2BPtuiQj1hwm%3D73xJ8hWuw-9cXbN4dHJHpM6EXxubDJgmFA%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPvi-g%2B9%2Bhjmjg44OzTN9L3YGQiCXBDAVaTVWvSn5SSwmw%40mail.gmail.com

--
Thanks,
Nisha

  288. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-11-27T21:37:47Z

    Hi Nisha, here are my review comments for the patch v51-0001.

======
src/backend/replication/slot.c

ReplicationSlotAcquire:

1.
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+    NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+
+ pfree(err_detail.data);
+ }
+

Won't the 'pfree' be unreachable due to the prior ereport ERROR?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  289. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-11-27T23:50:05Z

    Hi Nisha. Here are some review comments for patch v51-0002.

======
doc/src/sgml/system-views.sgml

1.
         The time when the slot became inactive. <literal>NULL</literal> if the
-        slot is currently being streamed.
+        slot is currently being streamed. Once the slot is invalidated, this
+        value will remain unchanged until we shutdown the server.
.

I think "Once the ..." kind of makes it sound like invalidation is
inevitable. Also maybe it's better to remove the "we".

SUGGESTION:
If the slot becomes invalidated, this value will remain unchanged
until server shutdown.

======
src/backend/replication/slot.c

ReplicationSlotAcquire:

2.
GENERAL.

This just is a question/idea. It may not be feasible to change. It
seems like there is a lot of overlap between the error messages in
'ReplicationSlotAcquire' which are saying "This slot has been
invalidated because...", and with the other function
'ReportSlotInvalidation' which is kind of the same but called in
different circumstances and with slightly different message text. I
wondered if there is a way to use common code to unify these messages
instead of having a nearly duplicate set of messages for all the
invalidation causes?

~~~

3.
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfo(&err_detail, _("inactivity exceeded the time limit
set by \"%s\"."),
+ "replication_slot_inactive_timeout");
+ break;

Should this err_detail also say "This slot has been invalidated
because ..." like all the others?

~~~

InvalidatePossiblyObsoleteSlot:

4.
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (IsSlotInactiveTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+    replication_slot_inactive_timeout_sec * 1000))
+ {

Maybe this code should have Assert(now > 0); before the condition just
as a way to 'document' that it is assumed 'now' was already set this
outside the mutex.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  290. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-11-28T07:59:21Z

    Dear Nisha,

> 
> Attached v51 patch-set addressing all comments in [1] and [2].
>

Thanks for working on the feature! I've stated to review the patch.
Here are my comments - sorry if there are something which have already been discussed.
The thread is too long to follow correctly.

Comments for 0001
=============

01. binary_upgrade_logical_slot_has_caught_up

ISTM that error_if_invalid is set to true when the slot can be moved forward, otherwise
it is set to false. Regarding the binary_upgrade_logical_slot_has_caught_up, however,
only valid slots will be passed to the funciton (see pg_upgrade/info.c) so I feel
it is OK to set to true. Thought?

02. ReplicationSlotAcquire

According to other functions, we are adding to a note to the translator when
parameters represent some common nouns, GUC names. I feel we should add a comment
for RS_INVAL_WAL_LEVEL part based on it.


Comments for 0002
=============

03. check_replication_slot_inactive_timeout

Can we overwrite replication_slot_inactive_timeout to zero when pg_uprade (and also
pg_createsubscriber?) starts a server process? Several parameters have already been
specified via -c option at that time. This can avoid an error while the upgrading.
Note that this part is still needed even if you accept the comment. Users can
manually boot with upgrade mode.

04. ReplicationSlotAcquire

Same comment as 02.

05. ReportSlotInvalidation

Same comment as 02.

06. found bug

While testing the patch, I found that slots can be invalidated too early when when
the GUC is quite large. I think because an overflow is caused in InvalidatePossiblyObsoleteSlot().

- Reproducer

I set the replication_slot_inactive_timeout to INT_MAX and executed below commands,
and found that the slot is invalidated.

```
postgres=# SHOW replication_slot_inactive_timeout;
 replication_slot_inactive_timeout 
-----------------------------------
 2147483647s
(1 row)
postgres=# SELECT * FROM pg_create_logical_replication_slot('test', 'test_decoding');
 slot_name |    lsn    
-----------+-----------
 test      | 0/18B7F38
(1 row)
postgres=# CHECKPOINT ;
CHECKPOINT
postgres=# SELECT slot_name, inactive_since, invalidation_reason FROM pg_replication_slots ;
 slot_name |        inactive_since         | invalidation_reason 
-----------+-------------------------------+---------------------
 test      | 2024-11-28 07:50:25.927594+00 | inactive_timeout
(1 row)
```

- analysis

In InvalidatePossiblyObsoleteSlot(), replication_slot_inactive_timeout_sec * 1000
is passed to the third argument of TimestampDifferenceExceeds(), which is also the
integer datatype. This causes an overflow and parameter is handled as the small
value.

- solution

I think there are two possible solutions. You can choose one of them:

a. Make the maximum INT_MAX/1000, or
b. Change the unit to millisecond.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  291. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-28T09:13:50Z

    On Fri, 22 Nov 2024 at 17:43, vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 21 Nov 2024 at 17:35, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > On Wed, Nov 20, 2024 at 1:29 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Tue, 19 Nov 2024 at 12:43, Nisha Moond <nisha.moond412@gmail.com> wrote:
> > > >
> > > > Attached is the v49 patch set:
> > > > - Fixed the bug reported in [1].
> > > > - Addressed comments in [2] and [3].
> > > >
> > > > I've split the patch into two, implementing the suggested idea in
> > > > comment #5 of [2] separately in 001:
> > > >
> > > > Patch-001: Adds additional error reports (for all invalidation types)
> > > > in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> > > > true.
> > > > Patch-002: The original patch with comments addressed.
> > >
> > > This Assert can fail:
> > >
> >
> > Attached v50 patch-set addressing review comments in [1] and [2].
>
> We are setting inactive_since when the replication slot is released.
> We are marking the slot as inactive only if it has been released.
> However, there's a scenario where the network connection between the
> publisher and subscriber may be lost where the replication slot is not
> released, but no changes are replicated due to the network problem. In
> this case, no updates would occur in the replication slot for a period
> exceeding the replication_slot_inactive_timeout.
> Should we invalidate these replication slots as well, or is it
> intentionally left out?

On further thinking, I felt we can keep the current implementation as
is and simply add a brief comment in the code to address this.
Additionally, we can mention it in the commit message for clarity.

Regards,
Vignesh

  292. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-11-28T14:10:32Z

    On Wed, 27 Nov 2024 at 16:25, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Wed, Nov 27, 2024 at 8:39 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Nisha,
> >
> > Here are some review comments for the patch v50-0002.
> >
> > ======
> > src/backend/replication/slot.c
> >
> > InvalidatePossiblyObsoleteSlot:
> >
> > 1.
> > + if (now &&
> > + TimestampDifferenceExceeds(s->inactive_since, now,
> > +    replication_slot_inactive_timeout_sec * 1000))
> >
> > Previously this was using an additional call to SlotInactiveTimeoutCheckAllowed:
> >
> > + if (SlotInactiveTimeoutCheckAllowed(s) &&
> > + TimestampDifferenceExceeds(s->inactive_since, now,
> > +    replication_slot_inactive_timeout * 1000))
> >
> > Is it OK to skip that call? e.g. can the slot fields possibly change
> > between assigning the 'now' and acquiring the mutex? If not, then the
> > current code is fine. The only reason for asking is because it is
> > slightly suspicious that it was not done this "easy" way in the first
> > place.
> >
> Good catch! While the mutex was being acquired right after the now
> assignment, there was a rare chance of another process modifying the
> slot in the meantime. So, I reverted the change in v51. To optimize
> the SlotInactiveTimeoutCheckAllowed() call, it's sufficient to check
> it here instead of during the 'now' assignment.
>
> Attached v51 patch-set addressing all comments in [1] and [2].

Few comments:
1) replication_slot_inactive_timeout can be mentioned in logical
replication config, we could mention something like:
Logical replication slot is also affected by replication_slot_inactive_timeout

2.a) Is this change applicable only for inactive timeout or it is
applicable to others like wal removed, wal level etc also? If it is
applicable to all of them we could move this to the first patch and
update the commit message:
+                * If the slot can be acquired, do so and mark it as
invalidated. If
+                * the slot is already ours, mark it as invalidated.
Otherwise, we'll
+                * signal the owning process below and retry.
                 */
-               if (active_pid == 0)
+               if (active_pid == 0 ||
+                       (MyReplicationSlot == s &&
+                        active_pid == MyProcPid))

2.b) Also this MyReplicationSlot and active_pid check can be in same line:
+                       (MyReplicationSlot == s &&
+                        active_pid == MyProcPid))


3) Error detail should start in upper case here similar to how others are done:
+                       case RS_INVAL_INACTIVE_TIMEOUT:
+                               appendStringInfo(&err_detail,
_("inactivity exceeded the time limit set by \"%s\"."),
+
"replication_slot_inactive_timeout");
+                               break;

4) Since this change is not related to this patch, we can move this to
the first patch and update the commit message:
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data,
size_t startup_data_len)
 static void
 update_synced_slots_inactive_since(void)
 {
-       TimestampTz now = 0;
+       TimestampTz now;

        /*
         * We need to update inactive_since only when we are promoting
standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
        /* The slot sync worker or SQL function mustn't be running by now */
        Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);

+       /* Use same inactive_since time for all slots */
+       now = GetCurrentTimestamp();

5) Since this change is not related to this patch, we can move this to
the first patch.
@@ -2250,6 +2350,7 @@ RestoreSlotFromDisk(const char *name)
        bool            restored = false;
        int                     readBytes;
        pg_crc32c       checksum;
+       TimestampTz now;

        /* no need to lock here, no concurrent access allowed yet */

@@ -2410,6 +2511,9 @@ RestoreSlotFromDisk(const char *name)
                                                NameStr(cp.slotdata.name)),
                                 errhint("Change \"wal_level\" to be
\"replica\" or higher.")));

+       /* Use same inactive_since time for all slots */
+       now = GetCurrentTimestamp();
+
        /* nothing can be active yet, don't lock anything */
        for (i = 0; i < max_replication_slots; i++)
        {
@@ -2440,9 +2544,11 @@ RestoreSlotFromDisk(const char *name)
                /*
                 * Set the time since the slot has become inactive
after loading the
                 * slot from the disk into memory. Whoever acquires
the slot i.e.
-                * makes the slot active will reset it.
+                * makes the slot active will reset it. Avoid calling
+                * ReplicationSlotSetInactiveSince() here, as it will
not set the time
+                * for invalid slots.
                 */
-               slot->inactive_since = GetCurrentTimestamp();
+               slot->inactive_since = now;

[1] - https://www.postgresql.org/docs/current/logical-replication-config.html

Regards,
Vignesh

  293. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-29T12:36:02Z

    On Thu, Nov 28, 2024 at 2:44 PM vignesh C <vignesh21@gmail.com> wrote:
>
> >
> > We are setting inactive_since when the replication slot is released.
> > We are marking the slot as inactive only if it has been released.
> > However, there's a scenario where the network connection between the
> > publisher and subscriber may be lost where the replication slot is not
> > released, but no changes are replicated due to the network problem. In
> > this case, no updates would occur in the replication slot for a period
> > exceeding the replication_slot_inactive_timeout.
> > Should we invalidate these replication slots as well, or is it
> > intentionally left out?
>
> On further thinking, I felt we can keep the current implementation as
> is and simply add a brief comment in the code to address this.
> Additionally, we can mention it in the commit message for clarity.
>

Thank you for the clarification. I’ve included the explanatory comment
in patch-002.

Attached the v52 patch-set addressing above as well as all other
comments till now in [1], [2], [3], and [4].

[1] https://www.postgresql.org/message-id/CAHut%2BPto1Yz9Fqp07LLP9uvx3sRHe5SOUKuFM1sUF9QA5aLfBA%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPs%3DH6EBO1ssGfykrJfUQQGh76L0eKuU5XkR9GMs96ZT3g%40mail.gmail.com
[3] https://www.postgresql.org/message-id/TYAPR01MB56927564EEE26E5433198405F5292%40TYAPR01MB5692.jpnprd01.prod.outlook.com
[4] https://www.postgresql.org/message-id/CALDaNm1F2YrswzM_WM37BYmiZ9Cf60UD_mgtm8HnMHRGA7tx4g%40mail.gmail.com

--
Thanks,
Nisha

  294. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-29T12:36:33Z

    On Tue, Nov 19, 2024 at 12:47 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Thu, Nov 14, 2024 at 5:29 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> >
> > 12.
> >   /*
> > - * If the slot can be acquired, do so and mark it invalidated
> > - * immediately.  Otherwise we'll signal the owning process, below, and
> > - * retry.
> > + * If the slot can be acquired, do so and mark it as invalidated. If
> > + * the slot is already ours, mark it as invalidated. Otherwise, we'll
> > + * signal the owning process below and retry.
> >   */
> > - if (active_pid == 0)
> > + if (active_pid == 0 ||
> > + (MyReplicationSlot == s &&
> > + active_pid == MyProcPid))
> >
> > I wasn't sure how this change belongs to this patch, because the logic
> > of the previous review comment said for the case of invalidation due
> > to inactivity that active_id must be 0. e.g. Assert(s->active_pid ==
> > 0);
> >
>
> I don't fully understand the purpose of this change yet. I'll look
> into it further and get back.
>

This change applies to all types of invalidation, not just
inactive_timeout case, so moved the change to patch-001. It’s a
general optimization for the case when the current process is the
active PID for the slot.
Also, the Assert(s->active_pid == 0); has been removed (in v50) as it
was unnecessary.

--
Thanks,
Nisha

  295. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-29T12:36:56Z

    On Thu, Nov 28, 2024 at 1:29 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Nisha,
>
> >
> > Attached v51 patch-set addressing all comments in [1] and [2].
> >
>
> Thanks for working on the feature! I've stated to review the patch.
> Here are my comments - sorry if there are something which have already been discussed.
> The thread is too long to follow correctly.
>
> Comments for 0001
> =============
>
> 01. binary_upgrade_logical_slot_has_caught_up
>
> ISTM that error_if_invalid is set to true when the slot can be moved forward, otherwise
> it is set to false. Regarding the binary_upgrade_logical_slot_has_caught_up, however,
> only valid slots will be passed to the funciton (see pg_upgrade/info.c) so I feel
> it is OK to set to true. Thought?
>

Right, corrected the call with error_if_invalid as true.

> Comments for 0002
> =============
>
> 03. check_replication_slot_inactive_timeout
>
> Can we overwrite replication_slot_inactive_timeout to zero when pg_uprade (and also
> pg_createsubscriber?) starts a server process? Several parameters have already been
> specified via -c option at that time. This can avoid an error while the upgrading.
> Note that this part is still needed even if you accept the comment. Users can
> manually boot with upgrade mode.
>

Done.

> 06. found bug
>
> While testing the patch, I found that slots can be invalidated too early when when
> the GUC is quite large. I think because an overflow is caused in InvalidatePossiblyObsoleteSlot().
>
> - Reproducer
>
> I set the replication_slot_inactive_timeout to INT_MAX and executed below commands,
> and found that the slot is invalidated.
>
> ```
> postgres=# SHOW replication_slot_inactive_timeout;
>  replication_slot_inactive_timeout
> -----------------------------------
>  2147483647s
> (1 row)
> postgres=# SELECT * FROM pg_create_logical_replication_slot('test', 'test_decoding');
>  slot_name |    lsn
> -----------+-----------
>  test      | 0/18B7F38
> (1 row)
> postgres=# CHECKPOINT ;
> CHECKPOINT
> postgres=# SELECT slot_name, inactive_since, invalidation_reason FROM pg_replication_slots ;
>  slot_name |        inactive_since         | invalidation_reason
> -----------+-------------------------------+---------------------
>  test      | 2024-11-28 07:50:25.927594+00 | inactive_timeout
> (1 row)
> ```
>
> - analysis
>
> In InvalidatePossiblyObsoleteSlot(), replication_slot_inactive_timeout_sec * 1000
> is passed to the third argument of TimestampDifferenceExceeds(), which is also the
> integer datatype. This causes an overflow and parameter is handled as the small
> value.
>
> - solution
>
> I think there are two possible solutions. You can choose one of them:
>
> a. Make the maximum INT_MAX/1000, or
> b. Change the unit to millisecond.
>

Fixed. It is reasonable to align with other timeout parameters by
using milliseconds as the unit.

--
Thanks,
Nisha

  296. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-11-29T12:37:32Z

    On Thu, Nov 28, 2024 at 5:20 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha. Here are some review comments for patch v51-0002.
>
> ======
> src/backend/replication/slot.c
>
> ReplicationSlotAcquire:
>
> 2.
> GENERAL.
>
> This just is a question/idea. It may not be feasible to change. It
> seems like there is a lot of overlap between the error messages in
> 'ReplicationSlotAcquire' which are saying "This slot has been
> invalidated because...", and with the other function
> 'ReportSlotInvalidation' which is kind of the same but called in
> different circumstances and with slightly different message text. I
> wondered if there is a way to use common code to unify these messages
> instead of having a nearly duplicate set of messages for all the
> invalidation causes?
>

The error handling could be moved to a new function; however, as you
pointed out, the contexts in which these functions are called differ.
IMO, a single error message may not suit both cases. For example,
ReportSlotInvalidation provides additional details and a hint in its
message, which isn’t necessary for ReplicationSlotAcquire.
Thoughts?

--
Thanks,
Nisha

  297. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-03T07:00:08Z

    Hi Nisha, here are a couple of review comments for patch v52-0001.

======
Commit Message

Add check if slot is already acquired, then mark it invalidate directly.

~

/slot/the slot/

"mark it invalidate" ?

Maybe you meant:
"then invalidate it directly", or
"then mark it 'invalidated' directly", or
etc.

======
src/backend/replication/logical/slotsync.c

1.
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data,
size_t startup_data_len)
 static void
 update_synced_slots_inactive_since(void)
 {
- TimestampTz now = 0;
+ TimestampTz now;

  /*
  * We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
  /* The slot sync worker or SQL function mustn't be running by now */
  Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);

+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+

Something is broken with these changes.

AFAICT, the result after applying patch 0001 still has code:
/* Use the same inactive_since time for all the slots. */
if (now == 0)
  now = GetCurrentTimestamp();

So the end result has multiple/competing assignments to variable 'now'.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  298. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-12-03T07:39:13Z

    Dear Nisha,

Thanks for updating the patch!

> Fixed. It is reasonable to align with other timeout parameters by
> using milliseconds as the unit.

It looks you just replaced to GUC_UNIT_MS, but the documentation and
postgresql.conf.sample has not been changed yet. They should follow codes.
Anyway, here are other comments, mostly cosmetic.

01. slot.c

```
+int         replication_slot_inactive_timeout_ms = 0;
```

According to other lines, we should add a short comment for the GUC.

02. 050_invalidate_slots.pl

Do you have a reason why you use the number 050? I feel it can be 043.

03. 050_invalidate_slots.pl

Also, not sure the file name is correct. This file contains only a slot invalidation due to the
replication_slot_inactive_timeout. But I feel current name is too general.

04. 050_invalidate_slots.pl

```
+use Time::HiRes qw(usleep);
```

This line is not needed because usleep() is not used in this file.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  299. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-04T09:30:52Z

    On Tue, Dec 3, 2024 at 1:09 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Nisha,
>
> Thanks for updating the patch!
>
> > Fixed. It is reasonable to align with other timeout parameters by
> > using milliseconds as the unit.
>
> It looks you just replaced to GUC_UNIT_MS, but the documentation and
> postgresql.conf.sample has not been changed yet. They should follow codes.
> Anyway, here are other comments, mostly cosmetic.
>

Here is v53 patch-set addressing all the comments in [1] and [2].

[1] https://www.postgresql.org/message-id/CAHut%2BPsQM79f34LLBGq4UeRuZ1URWP6JNZtdN2khYPrLc1YqrQ%40mail.gmail.com
[2] https://www.postgresql.org/message-id/TYAPR01MB5692B7687EE7981AA91BA5B9F5362%40TYAPR01MB5692.jpnprd01.prod.outlook.com

--
Thanks,
Nisha

  300. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-12-04T10:26:53Z

    On Wed, 4 Dec 2024 at 15:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Tue, Dec 3, 2024 at 1:09 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Nisha,
> >
> > Thanks for updating the patch!
> >
> > > Fixed. It is reasonable to align with other timeout parameters by
> > > using milliseconds as the unit.
> >
> > It looks you just replaced to GUC_UNIT_MS, but the documentation and
> > postgresql.conf.sample has not been changed yet. They should follow codes.
> > Anyway, here are other comments, mostly cosmetic.
> >
>
> Here is v53 patch-set addressing all the comments in [1] and [2].

Currently, replication slots are invalidated based on the
replication_slot_inactive_timeout only during a checkpoint. This means
that if the checkpoint_timeout is set to a higher value than the
replication_slot_inactive_timeout, slot invalidation will occur only
when the checkpoint is triggered. Identifying the invalidation slots
might be slightly delayed in this case. As an alternative, users can
forcefully invalidate inactive slots that have exceeded the
replication_slot_inactive_timeout by forcing a checkpoint. I was
thinking we could suggest this in the documentation.

+       <para>
+        Slot invalidation due to inactive timeout occurs during checkpoint.
+        The duration of slot inactivity is calculated using the slot's
+        <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+        value.
+       </para>
+

We could accurately invalidate the slots using the checkpointer
process by calculating the invalidation time based on the active_since
timestamp and the replication_slot_inactive_timeout, and then set the
checkpointer's main wait-latch accordingly for triggering the next
checkpoint. Ideally, a different process handling this task would be
better, but there is currently no dedicated daemon capable of
identifying and managing slots across streaming replication, logical
replication, and other slots used by plugins. Additionally,
overloading the checkpointer with this responsibility may not be
ideal. As an alternative, we could document about this delay in
identifying and mention that it could be triggered by forceful manual
checkpoint.

Regards,
Vignesh

  301. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-12-04T10:46:06Z

    On Wed, 4 Dec 2024 at 15:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Tue, Dec 3, 2024 at 1:09 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Nisha,
> >
> > Thanks for updating the patch!
> >
> > > Fixed. It is reasonable to align with other timeout parameters by
> > > using milliseconds as the unit.
> >
> > It looks you just replaced to GUC_UNIT_MS, but the documentation and
> > postgresql.conf.sample has not been changed yet. They should follow codes.
> > Anyway, here are other comments, mostly cosmetic.
> >
>
> Here is v53 patch-set addressing all the comments in [1] and [2].

CFBot is failing at [1] because the file name is changed to
043_invalidate_inactive_slots, the meson.build file should be updated
accordingly:
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
       't/040_standby_failover_slots_sync.pl',
       't/041_checkpoint_at_promote.pl',
       't/042_low_level_backup.pl',
+      't/050_invalidate_slots.pl',
     ],
   },
 }

[1] - https://cirrus-ci.com/task/6266479424831488

Regards,
Vignesh

  302. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-05T00:35:20Z

    Hi Nisha,

Here are my review comments for the v53* patch set

//////////

Patch v53-0001.

======
src/backend/replication/slot.c

1.
+ if (error_if_invalid &&
+ s->data.invalidated != RS_INVAL_NONE)

Looks like some unnecessary wrapping here. I think this condition can
be on one line.

//////////

Patch v53-0002.

======
GENERAL - How about using the term "idle"?

1.
I got to wondering why this new GUC was called
"replication_slot_inactive_timeout", with invalidation_reason =
"inactive_timeout". When I look at similar GUCs I don't see words like
"inactivity" or "inactive" anywhere; Instead, they are using the term
"idle" to refer to when something is inactive:
e.g.
#idle_in_transaction_session_timeout = 0 # in milliseconds, 0 is disabled
#idle_session_timeout = 0 # in milliseconds, 0 is disabled

I know the "inactive" term is used a bit in the slot code but that is
(mostly) not exposed to the user. Therefore, I am beginning to feel it
would be better (e.g. more consistent) to use "idle" for the
user-facing stuff. e.g.
New Slot GUC = "idle_replication_slot_timeout"
Slot invalidation_reason = "idle_timeout"

Of course, changing this will cascade to impact quite a lot of other
things in the patch -- comments, error messages, some function names
etc.

======
doc/src/sgml/logical-replication.sgml

2.
+   <para>
+    Logical replication slot is also affected by
+    <link linkend="guc-replication-slot-inactive-timeout"><varname>replication_slot_inactive_timeout</varname></link>.
+   </para>
+

/Logical replication slot is also affected by/Logical replication
slots are also affected by/

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  303. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-05T01:13:40Z

    On Wed, Dec 4, 2024 at 9:27 PM vignesh C <vignesh21@gmail.com> wrote:
>
...
>
> Currently, replication slots are invalidated based on the
> replication_slot_inactive_timeout only during a checkpoint. This means
> that if the checkpoint_timeout is set to a higher value than the
> replication_slot_inactive_timeout, slot invalidation will occur only
> when the checkpoint is triggered. Identifying the invalidation slots
> might be slightly delayed in this case. As an alternative, users can
> forcefully invalidate inactive slots that have exceeded the
> replication_slot_inactive_timeout by forcing a checkpoint. I was
> thinking we could suggest this in the documentation.
>
> +       <para>
> +        Slot invalidation due to inactive timeout occurs during checkpoint.
> +        The duration of slot inactivity is calculated using the slot's
> +        <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
> +        value.
> +       </para>
> +
>
> We could accurately invalidate the slots using the checkpointer
> process by calculating the invalidation time based on the active_since
> timestamp and the replication_slot_inactive_timeout, and then set the
> checkpointer's main wait-latch accordingly for triggering the next
> checkpoint. Ideally, a different process handling this task would be
> better, but there is currently no dedicated daemon capable of
> identifying and managing slots across streaming replication, logical
> replication, and other slots used by plugins. Additionally,
> overloading the checkpointer with this responsibility may not be
> ideal. As an alternative, we could document about this delay in
> identifying and mention that it could be triggered by forceful manual
> checkpoint.
>

Hi Vignesh.

I felt that manipulating the checkpoint timing behind the scenes
without the user's consent might be a bit of an overreach.

But there might still be something else we could do:

1. We can add the documentation note like you suggested ("we could
document about this delay in identifying and mention that it could be
triggered by forceful manual checkpoint").

2. We can also detect such delays in the code. When the invalidation
occurs (e.g. code fragment below) we could check if there was some
excessive lag between the slot becoming idle and it being invalidated.
If the lag is too much (whatever "too much" means) we can log a hint
for the user to increase the checkpoint frequency (or whatever else we
might advise them to do).

+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (IsSlotInactiveTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+    replication_slot_inactive_timeout_ms))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;

pseudo-code:
if (slot invalidation occurred much later after the
replication_slot_inactive_timeout GUC elapsed)
{
  elog(LOG, "This slot was inactive for a period of %s. Slot timeout
invalidation only occurs at a checkpoint so if you want inactive slots
to be invalidated in a more timely manner consider reducing the time
between checkpoints or executing a manual checkpoint.
(replication_slot_inactive_timeout = %s; checkpoint_timeout = %s,
....)"
}

+ }

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  304. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-12-06T05:34:29Z

    On Thu, 5 Dec 2024 at 06:44, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Wed, Dec 4, 2024 at 9:27 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> ...
> >
> > Currently, replication slots are invalidated based on the
> > replication_slot_inactive_timeout only during a checkpoint. This means
> > that if the checkpoint_timeout is set to a higher value than the
> > replication_slot_inactive_timeout, slot invalidation will occur only
> > when the checkpoint is triggered. Identifying the invalidation slots
> > might be slightly delayed in this case. As an alternative, users can
> > forcefully invalidate inactive slots that have exceeded the
> > replication_slot_inactive_timeout by forcing a checkpoint. I was
> > thinking we could suggest this in the documentation.
> >
> > +       <para>
> > +        Slot invalidation due to inactive timeout occurs during checkpoint.
> > +        The duration of slot inactivity is calculated using the slot's
> > +        <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
> > +        value.
> > +       </para>
> > +
> >
> > We could accurately invalidate the slots using the checkpointer
> > process by calculating the invalidation time based on the active_since
> > timestamp and the replication_slot_inactive_timeout, and then set the
> > checkpointer's main wait-latch accordingly for triggering the next
> > checkpoint. Ideally, a different process handling this task would be
> > better, but there is currently no dedicated daemon capable of
> > identifying and managing slots across streaming replication, logical
> > replication, and other slots used by plugins. Additionally,
> > overloading the checkpointer with this responsibility may not be
> > ideal. As an alternative, we could document about this delay in
> > identifying and mention that it could be triggered by forceful manual
> > checkpoint.
> >
>
> Hi Vignesh.
>
> I felt that manipulating the checkpoint timing behind the scenes
> without the user's consent might be a bit of an overreach.

Agree

> But there might still be something else we could do:
>
> 1. We can add the documentation note like you suggested ("we could
> document about this delay in identifying and mention that it could be
> triggered by forceful manual checkpoint").

Yes, that makes sense

> 2. We can also detect such delays in the code. When the invalidation
> occurs (e.g. code fragment below) we could check if there was some
> excessive lag between the slot becoming idle and it being invalidated.
> If the lag is too much (whatever "too much" means) we can log a hint
> for the user to increase the checkpoint frequency (or whatever else we
> might advise them to do).
>
> + /*
> + * Check if the slot needs to be invalidated due to
> + * replication_slot_inactive_timeout GUC.
> + */
> + if (IsSlotInactiveTimeoutPossible(s) &&
> + TimestampDifferenceExceeds(s->inactive_since, now,
> +    replication_slot_inactive_timeout_ms))
> + {
> + invalidation_cause = cause;
> + inactive_since = s->inactive_since;
>
> pseudo-code:
> if (slot invalidation occurred much later after the
> replication_slot_inactive_timeout GUC elapsed)
> {
>   elog(LOG, "This slot was inactive for a period of %s. Slot timeout
> invalidation only occurs at a checkpoint so if you want inactive slots
> to be invalidated in a more timely manner consider reducing the time
> between checkpoints or executing a manual checkpoint.
> (replication_slot_inactive_timeout = %s; checkpoint_timeout = %s,
> ....)"
> }
>
> + }

Determining the correct time may be challenging for users, as it
depends on when the active_since value is set, as well as when the
checkpoint_timeout occurs and the subsequent checkpoint is triggered.
Even if the user sets it to an appropriate value, there is still a
possibility of delayed identification due to the timing of when the
slot's active_timeout is being set. Including this information in the
documentation should be sufficient.

Regards,
Vignesh

  305. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-10T11:51:09Z

    On Fri, Dec 6, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
>
>
> Determining the correct time may be challenging for users, as it
> depends on when the active_since value is set, as well as when the
> checkpoint_timeout occurs and the subsequent checkpoint is triggered.
> Even if the user sets it to an appropriate value, there is still a
> possibility of delayed identification due to the timing of when the
> slot's active_timeout is being set. Including this information in the
> documentation should be sufficient.
>

+1
v54 documents this information as suggested.

Attached the v54 patch-set addressing all the comments till now in
[1], [2] and [3].

[1] https://www.postgresql.org/message-id/CALDaNm0mTWwg0z4v-sorq08S2CdZmL2s%2Brh4nHpWeJaBQ2F%2Bmg%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CALDaNm1STyk%3DS_EAihWP9SowBkS5dJ32JfEqmG5tTeC2Ct39yg%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CAHut%2BPtHbYNxPvtMfs7jARbsVcFXL1%3DC9SO3Q93NgVDgbKN7LQ%40mail.gmail.com

--
Thanks,
Nisha

  306. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-11T02:44:05Z

    Hi Nisha.

Here are some review comments for patch v54-0002.

(I had also checked patch v54-0001, but have no further review
comments for that one).

======
doc/src/sgml/config.sgml

1.
+       <para>
+        Slot invalidation due to idle timeout occurs during checkpoint.
+        If the <varname>checkpoint_timeout</varname> exceeds
+        <varname>idle_replication_slot_timeout</varname>, the slot
+        invalidation will be delayed until the next checkpoint is triggered.
+        To avoid delays, users can force a checkpoint to promptly invalidate
+        inactive slots. The duration of slot inactivity is calculated
using the slot's
+        <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+        value.
+       </para>
+

The wording of "If the checkpoint_timeout exceeds
idle_replication_slot_timeout, the slot invalidation will be delayed
until the next checkpoint is triggered." seems slightly misleading,
because AFAIK it is not conditional on the GUC value differences like
that -- i.e. slot invalidation is *always* delayed until the next
checkpoint occurs.

SUGGESTION:
Slot invalidation due to idle timeout occurs during checkpoint.
Because checkpoints happen at checkpoint_timeout intervals, there can
be some lag between when the idle_replication_slot_timeout was
exceeded and when the slot invalidation is triggered at the next
checkpoint. To avoid such lags, users can force...

=======
src/backend/replication/slot.c

2. GENERAL

+/* Invalidate replication slots idle beyond this time; '0' disables it */
+int idle_replication_slot_timeout_ms = 0;

I noticed this patch is using a variety of ways of describing the same thing:
* guc var: Invalidate replication slots idle beyond this time...
* guc_tables: ... the amount of time a replication slot can remain
idle before it will be invalidated.
* docs: means that the slot has remained idle beyond the duration
specified by the idle_replication_slot_timeout parameter
* errmsg: ... slot has been invalidated because inactivity exceeded
the time limit set by ...
* etc..

They are all the same, but they are all worded slightly differently:
* "idle" vs "inactivity" vs ...
* "time" vs "amount of time" vs "duration" vs "time limit" vs ...

There may not be a one-size-fits-all, but still, it might be better to
try to search for all different phrasing and use common wording as
much as possible.

~~~

CheckPointReplicationSlots:

3.
+ * XXX: Slot invalidation due to 'idle_timeout' occurs only for
+ * released slots, based on 'idle_replication_slot_timeout'. Active
+ * slots in use for replication are excluded, preventing accidental
+ * invalidation. Slots where communication between the publisher and
+ * subscriber is down are also excluded, as they are managed by the
+ * 'wal_sender_timeout'.

Maybe a slight rewording like below is better. Maybe not. YMMV.

SUGGESTION:
XXX: Slot invalidation due to 'idle_timeout' applies only to released
slots, and is based on the 'idle_replication_slot_timeout' GUC. Active
slots
currently in use for replication are excluded to prevent accidental
invalidation.  Slots...

======
src/bin/pg_upgrade/server.c

4.
+ /*
+ * Use idle_replication_slot_timeout=0 to prevent slot invalidation due to
+ * inactive_timeout by checkpointer process during upgrade.
+ */
+ if (GET_MAJOR_VERSION(cluster->major_version) >= 1800)
+ appendPQExpBufferStr(&pgoptions, " -c idle_replication_slot_timeout=0");
+

/inactive_timeout/idle_timeout/

======
src/test/recovery/t/043_invalidate_inactive_slots.pl

5.
+# Wait for slot to first become idle and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $idle_timeout) = @_;
+ my $node_name = $node->name;

AFAICT this 'idle_timeout' parameter is passed units of "seconds", so
it would be better to call it something like 'idle_timeout_s' to make
the units clear.

~~~

6.
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $idle_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;

Ditto above review comment #5 -- better to call it something like
'idle_timeout_s' to make the units clear.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  307. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-12-11T06:51:11Z

    On Tue, 10 Dec 2024 at 17:21, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Fri, Dec 6, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> >
> > Determining the correct time may be challenging for users, as it
> > depends on when the active_since value is set, as well as when the
> > checkpoint_timeout occurs and the subsequent checkpoint is triggered.
> > Even if the user sets it to an appropriate value, there is still a
> > possibility of delayed identification due to the timing of when the
> > slot's active_timeout is being set. Including this information in the
> > documentation should be sufficient.
> >
>
> +1
> v54 documents this information as suggested.
>
> Attached the v54 patch-set addressing all the comments till now in

Few comments on the test added:
1) Can we remove this and set idle_replication_slot_timeout while the
standby node is created itself during append_conf:
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $idle_timeout_1s = 1;
+$standby1->safe_psql(
+       'postgres', qq[
+    ALTER SYSTEM SET idle_replication_slot_timeout TO '${idle_timeout_1s}s';
+]);
+$standby1->reload;

2) You can move these statements before the standby node is created:
+# Create sync slot on the primary
+$primary->psql('postgres',
+       q{SELECT pg_create_logical_replication_slot('sync_slot1',
'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+       'postgres', qq[
+    SELECT pg_create_physical_replication_slot(slot_name :=
'sb_slot1', immediately_reserve := true);
+]);

3) Do we need autovacuum as off for these tests, is there any
probability of a test failure without this. I felt it should not
impact these tests, if not we can remove this:
+# Avoid unpredictability
+$primary->append_conf(
+       'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});

4) Generally we mention single char in single quotes, we can update "t" to 't':
+       ),
+       "t",
+       'logical slot sync_slot1 is synced to standby');
+

5) Similarly here too:
+                 WHERE slot_name = 'sync_slot1'
+                       AND invalidation_reason IS NULL;}
+       ),
+       "t",
+       'check that synced slot sync_slot1 has not been invalidated on
standby');

6) This standby offset is not used anywhere, it can be removed:
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.

Regards,
Vignesh

  308. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-12-12T04:12:35Z

    On Tue, 10 Dec 2024 at 17:21, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Fri, Dec 6, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> >
> > Determining the correct time may be challenging for users, as it
> > depends on when the active_since value is set, as well as when the
> > checkpoint_timeout occurs and the subsequent checkpoint is triggered.
> > Even if the user sets it to an appropriate value, there is still a
> > possibility of delayed identification due to the timing of when the
> > slot's active_timeout is being set. Including this information in the
> > documentation should be sufficient.
> >
>
> +1
> v54 documents this information as suggested.
>
> Attached the v54 patch-set addressing all the comments till now in
> [1], [2] and [3].

Now that we support idle_replication_slot_timeout in milliseconds, we
can set this value from 1s to 1ms or 10millseconds and change sleep to
usleep, this will bring down the test execution time significantly:
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $idle_timeout_1s = 1;
+$standby1->safe_psql(
+       'postgres', qq[
+    ALTER SYSTEM SET idle_replication_slot_timeout TO '${idle_timeout_1s}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+               'postgres',
+               q{SELECT count(*) = 1 FROM pg_replication_slots
+                 WHERE slot_name = 'sync_slot1' AND synced
+                       AND NOT temporary
+                       AND invalidation_reason IS NULL;}
+       ),
+       "t",
+       'logical slot sync_slot1 is synced to standby');
+
+# Give enough time for inactive_since to exceed the timeout
+sleep($idle_timeout_1s + 1);

Regards,
Vignesh

  309. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-13T10:58:58Z

    On Thu, Dec 12, 2024 at 9:42 AM vignesh C <vignesh21@gmail.com> wrote:
>
>
> Now that we support idle_replication_slot_timeout in milliseconds, we
> can set this value from 1s to 1ms or 10millseconds and change sleep to
> usleep, this will bring down the test execution time significantly:

+1
v55 implements the test using idle_replication_slot_timeout=1ms,
significantly reducing the test time.

Attached the v55 patch set which addresses all the comments in [1], [2] as well.

[1] https://www.postgresql.org/message-id/CAHut%2BPvx294U-XBB6-BvabesUNxbnuDQmk-VOFm%3DpbcNWSsHvQ%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CALDaNm2wHDnboo0FCj247HiBMHAHqy0se8NTH4fDCdscxdjhcg%40mail.gmail.com

--
Thanks,
Nisha

  310. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-13T11:00:27Z

    On Wed, Dec 11, 2024 at 8:14 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha.
>
> Here are some review comments for patch v54-0002.
> ======
> src/test/recovery/t/043_invalidate_inactive_slots.pl
>
> 5.
> +# Wait for slot to first become idle and then get invalidated
> +sub wait_for_slot_invalidation
> +{
> + my ($node, $slot, $offset, $idle_timeout) = @_;
> + my $node_name = $node->name;
>
> AFAICT this 'idle_timeout' parameter is passed units of "seconds", so
> it would be better to call it something like 'idle_timeout_s' to make
> the units clear.
>

As per the suggestion in [1], the test has been updated to use
idle_timeout=1ms. Since the parameter uses the default unit of
"milliseconds," keeping it as 'idle_timeout' seems reasonable to me.

> ~~~
>
> 6.
> +# Trigger slot invalidation and confirm it in the server log
> +sub trigger_slot_invalidation
> +{
> + my ($node, $slot, $offset, $idle_timeout) = @_;
> + my $node_name = $node->name;
> + my $invalidated = 0;
>
> Ditto above review comment #5 -- better to call it something like
> 'idle_timeout_s' to make the units clear.
>

The 'idle_timeout' parameter name remains unchanged as explained above.

[1] https://www.postgresql.org/message-id/CALDaNm1FQS04aG0C0gCRpvi-o-OTdq91y6Az34YKN-dVc9r5Ng%40mail.gmail.com

--
Thanks,
Nisha

  311. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-16T04:28:24Z

    Hi Nisha.

Thanks for the v55* patches.

I have no comments for patch v55-0001.

I have only 1 comment for patch v55-0002 regarding some remaining
nitpicks (below) about the consistency of phrases.

======

I scanned again over all the phrases for consistency:

CURRENT PATCH:

Docs (idle_replication_slot_timeout): Invalidate replication slots
that are idle for longer than this amount of time
Docs (idle_timeout): means that the slot has remained idle longer than
the duration specified by the idle_replication_slot_timeout parameter.

Code (guc var comment):  Invalidate replication slots idle longer than this time
Code (guc_tables): Sets the time limit for how long a replication slot
can remain idle before it is invalidated.

Msg (errdetail): This slot has been invalidated because it has
remained idle longer than the configured \"%s\" time.
Msg (errdetail): The slot has been inactive since %s and has remained
idle longer than the configured \"%s\" time.

~

NITPICKS:

nit -- There are still some variations "amount of time" versus "time"
versus "duration".  I think the term "duration" best describe the
maing so we can use that everywhere.

nit - Should consistently say "remained idle" instead of just "idle"
or "are idle",

nit - The last errdetail is also rearranged a bit because IMO we don't
need to say inactive and idle in the same sentence.

nit - Just say "longer than" instead of sometimes saying "for longer than"

~

SUGGESTIONS:

Docs (idle_replication_slot_timeout): Invalidate replication slots
that have remained idle longer than this duration.
Docs (idle_timeout): means that the slot has remained idle longer than
the configured idle_replication_slot_timeout duration.

Code (guc var comment):  Invalidate replication slots that have
remained idle longer than this duration.
Code (guc_tables): Sets the duration a replication slot can remain
idle before it is invalidated.

Msg (errdetail): This slot has been invalidated because it has
remained idle longer than the configured \"%s\" duration.
Msg (errdetail): The slot has remained idle since %s, which is longer
than the configured \"%s\" duration.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  312. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-16T10:40:28Z

    On Mon, Dec 16, 2024 at 9:58 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha.
>
> Thanks for the v55* patches.
>
> I have no comments for patch v55-0001.
>
> I have only 1 comment for patch v55-0002 regarding some remaining
> nitpicks (below) about the consistency of phrases.
>
> ======
>
> SUGGESTIONS:
>
> Docs (idle_replication_slot_timeout): Invalidate replication slots
> that have remained idle longer than this duration.
> Docs (idle_timeout): means that the slot has remained idle longer than
> the configured idle_replication_slot_timeout duration.
>
> Code (guc var comment):  Invalidate replication slots that have
> remained idle longer than this duration.
> Code (guc_tables): Sets the duration a replication slot can remain
> idle before it is invalidated.
>
> Msg (errdetail): This slot has been invalidated because it has
> remained idle longer than the configured \"%s\" duration.
> Msg (errdetail): The slot has remained idle since %s, which is longer
> than the configured \"%s\" duration.
>

Here is the v56 patch set with the above comments incorporated.

--
Thanks
Nisha

  313. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-16T22:46:49Z

    On Mon, Dec 16, 2024 at 9:40 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Mon, Dec 16, 2024 at 9:58 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
...
> > SUGGESTIONS:
> >
> > Docs (idle_replication_slot_timeout): Invalidate replication slots
> > that have remained idle longer than this duration.
> > Docs (idle_timeout): means that the slot has remained idle longer than
> > the configured idle_replication_slot_timeout duration.
> >
> > Code (guc var comment):  Invalidate replication slots that have
> > remained idle longer than this duration.
> > Code (guc_tables): Sets the duration a replication slot can remain
> > idle before it is invalidated.
> >
> > Msg (errdetail): This slot has been invalidated because it has
> > remained idle longer than the configured \"%s\" duration.
> > Msg (errdetail): The slot has remained idle since %s, which is longer
> > than the configured \"%s\" duration.
> >
>
> Here is the v56 patch set with the above comments incorporated.
>

Hi Nisha.

Thanks for the updates.

- Both patches could be applied cleanly.
- Tests (make check, TAP subscriber, TAP recovery) are all passing.
- The rendering of the documentation changes from patch 0002 looked good.
- I have no more review comments.

So, the v56* patchset LGTM.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  314. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2024-12-20T09:42:06Z

    On Mon, Dec 16, 2024 at 4:10 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Here is the v56 patch set with the above comments incorporated.
>

Review comments:
===============
1.
+ {
+ {"idle_replication_slot_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the duration a replication slot can remain idle before "
+ "it is invalidated."),
+ NULL,
+ GUC_UNIT_MS
+ },
+ &idle_replication_slot_timeout_ms,

I think users are going to keep idele_slot timeout at least in hours.
So, millisecond seems the wrong choice to me. I suggest to keep the
units in minutes. I understand that writing a test would be
challenging as spending a minute or more on one test is not advisable.
But I don't see any test testing the other GUCs that are in minutes
(wal_summary_keep_time and log_rotation_age). The default value should
be one day.

2.
+ /*
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
+ */
+ if (error_if_invalid && s->data.invalidated != RS_INVAL_NONE)
+ {
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("This slot has been invalidated
because the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("This slot has been invalidated
because the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("This slot has been invalidated
because \"%s\" is insufficient for slot."),
+ "wal_level");
+ break;
+
+ case RS_INVAL_NONE:
+ pg_unreachable();
+ }
+
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+    NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+ }
+

This should be moved to a separate function.

3.
+static inline bool
+IsSlotIdleTimeoutPossible(ReplicationSlot *s)

Would it be better to name this function as CanInvalidateIdleSlot()?
The current name doesn't seem to match with similar other
functionalities.

-- 
With Regards,
Amit Kapila.

  315. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-24T11:36:55Z

    On Fri, Dec 20, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

> On Mon, Dec 16, 2024 at 4:10 PM Nisha Moond <nisha.moond412@gmail.com>
> wrote:
> >
> > Here is the v56 patch set with the above comments incorporated.
> >
>
> Review comments:
> ===============
> 1.
> + {
> + {"idle_replication_slot_timeout", PGC_SIGHUP, REPLICATION_SENDING,
> + gettext_noop("Sets the duration a replication slot can remain idle
> before "
> + "it is invalidated."),
> + NULL,
> + GUC_UNIT_MS
> + },
> + &idle_replication_slot_timeout_ms,
>
> I think users are going to keep idele_slot timeout at least in hours.
> So, millisecond seems the wrong choice to me. I suggest to keep the
> units in minutes. I understand that writing a test would be
> challenging as spending a minute or more on one test is not advisable.
> But I don't see any test testing the other GUCs that are in minutes
> (wal_summary_keep_time and log_rotation_age). The default value should
> be one day.
>

+1
- Changed the GUC unit to "minute".

Regarding the tests, we have two potential options:
 1) Introduce an additional "debug_xx" GUC parameter with units of seconds
or milliseconds, only for testing purposes.
 2) Skip writing tests for this, similar to other GUCs with units in
minutes.

IMO, adding an additional GUC just for testing may not be worthwhile. It's
reasonable to proceed without the test.

Thoughts?

The attached v57 patch-set addresses all the comments. I have kept the test
case in the patch for now, it takes 2-3 minutes to complete.

--
Thanks,
Nisha

  316. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Mihail Nikalayeu <michail.nikolaev@gmail.com> — 2024-12-24T12:57:15Z

    Hello everyone!

Yesterday I got a strange set of test errors, probably somehow related to
that patch.
It happened on changed master branch (based
on d96d1d5152f30d15678e08e75b42756101b7cab6) but I don't think my changes
were affecting it.

My setup is a little bit tricky: Windows 11 run WSL2 with Ubuntu, meson.

So, `recovery ` suite started failing on:

1) at /src/test/recovery/t/019_replslot_limit.pl line 530.
2) at /src/test/recovery/t/040_standby_failover_slots_sync.pl line 198.

It was failing almost every run, one test or another. I was lurking around
for about 10 min, and..... it just stopped failing. And I can't reproduce
it anymore.

But I have logs of two fails. I am not sure if it is helpful, but decided
to mail them here just in case.

Best regards,
Mikhail.

  317. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2024-12-26T06:02:20Z

    On Tuesday, December 24, 2024 8:57 PM Michail Nikolaev <michail.nikolaev@gmail.com>  wrote:

Hi,

> Yesterday I got a strange set of test errors, probably somehow related to
> that patch. It happened on changed master branch (based on
> d96d1d5152f30d15678e08e75b42756101b7cab6) but I don't think my changes were
> affecting it.
> 
> My setup is a little bit tricky: Windows 11 run WSL2 with Ubuntu, meson.
> 
> So, `recovery ` suite started failing on:
> 
> 1) at /src/test/recovery/t/http://019_replslot_limit.pl line 530.
> 2) at /src/test/recovery/t/http://040_standby_failover_slots_sync.pl line
>    198.
> 
> It was failing almost every run, one test or another. I was lurking around
> for about 10 min, and..... it just stopped failing. And I can't reproduce it
> anymore.
> 
> But I have logs of two fails. I am not sure if it is helpful, but decided to
> mail them here just in case.

Thanks for reporting the issue.

After checking the log, I think the failure is caused by the unexpected
behavior of the local system clock.

It's clear from the '019_replslot_limit_primary4.log'[1] that the clock went
backwards which makes the slot's inactive_since go backwards as well. That's
why the last testcase didn't pass.

And for 040_standby_failover_slots_sync, we can see that the clock of standby
lags behind that of the primary, which caused the inactive_since of newly synced
slot on standby to be earlier than the one on the primary.

So, I think it's not a bug in the committed patch but an issue in the testing
environment. Besides, since we have not seen such failures on BF, I think it
may not be necessary to improve the testcases.

[1]
2024-12-24 01:37:19.967 CET [161409] sub STATEMENT:  START_REPLICATION SLOT "lsub4_slot" LOGICAL 0/0 (proto_version '4', streaming 'parallel', origin 'any', publication_names '"pub"')
...
2024-12-24 01:37:20.025 CET [161447] 019_replslot_limit.pl LOG:  statement: SELECT '0/30003D8' <= replay_lsn AND state = 'streaming'
...
2024-12-24 01:37:19.388 CET [161097] LOG:  received fast shutdown request

Best Regards,
Hou zj

  318. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Mihail Nikalayeu <michail.nikolaev@gmail.com> — 2024-12-26T13:56:50Z

    Hello, Hou!

> So, I think it's not a bug in the committed patch but an issue in the
testing
>  venvironment. Besides, since we have not seen such failures on BF, I
think it
>  may not be necessary to improve the testcases.

Thanks for your analysis!
Yes, probably WSL2/Windows interactions cause strange system clock moving.
It looks like it is a common issue with WSL2 [0].

[0]: https://github.com/microsoft/WSL/issues/10006

Best regards,
Mikhail.

  319. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2024-12-27T03:52:18Z

    On Tue, 24 Dec 2024 at 17:07, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Fri, Dec 20, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Mon, Dec 16, 2024 at 4:10 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>> >
>> > Here is the v56 patch set with the above comments incorporated.
>> >
>>
>> Review comments:
>> ===============
>> 1.
>> + {
>> + {"idle_replication_slot_timeout", PGC_SIGHUP, REPLICATION_SENDING,
>> + gettext_noop("Sets the duration a replication slot can remain idle before "
>> + "it is invalidated."),
>> + NULL,
>> + GUC_UNIT_MS
>> + },
>> + &idle_replication_slot_timeout_ms,
>>
>> I think users are going to keep idele_slot timeout at least in hours.
>> So, millisecond seems the wrong choice to me. I suggest to keep the
>> units in minutes. I understand that writing a test would be
>> challenging as spending a minute or more on one test is not advisable.
>> But I don't see any test testing the other GUCs that are in minutes
>> (wal_summary_keep_time and log_rotation_age). The default value should
>> be one day.
>
>
> +1
> - Changed the GUC unit to "minute".
>
> Regarding the tests, we have two potential options:
>  1) Introduce an additional "debug_xx" GUC parameter with units of seconds or milliseconds, only for testing purposes.
>  2) Skip writing tests for this, similar to other GUCs with units in minutes.
>
> IMO, adding an additional GUC just for testing may not be worthwhile. It's reasonable to proceed without the test.
>
> Thoughts?
>
> The attached v57 patch-set addresses all the comments. I have kept the test case in the patch for now, it takes 2-3 minutes to complete.

Few comments:
1) We have disabled the similar configuration max_slot_wal_keep_size
by setting to -1, as this GUC also is in similar lines, should we
disable this and let the user configure it?
+/*
+ * Invalidate replication slots that have remained idle longer than this
+ * duration; '0' disables it.
+ */
+int                    idle_replication_slot_timeout_min =
HOURS_PER_DAY * MINS_PER_HOUR;
+

2) I felt this behavior is an existing behavior, so this can also be
moved to 0001 patch:
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index a586156614..199d7248ee 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2566,7 +2566,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
       </para>
       <para>
         The time when the slot became inactive. <literal>NULL</literal> if the
-        slot is currently being streamed.
+        slot is currently being streamed. If the slot becomes invalidated,
+        this value will remain unchanged until server shutdown.


3) Can we change the comment below to "We don't allow the value of
idle_replication_slot_timeout other than 0 during the binary upgrade.
See start_postmaster() in pg_upgrade for more details.":
+ * The idle_replication_slot_timeout must be disabled (set to 0)
+ * during the binary upgrade.
+ */
+bool
+check_idle_replication_slot_timeout(int *newval, void **extra,
GucSource source)

Regards,
Vignesh

  320. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-30T01:15:19Z

    Hi Nisha.

Here are some review comments for patch v57-0001.

======
src/backend/replication/slot.c

1.
+
+/*
+ * Raise an error based on the invalidation cause of the slot.
+ */
+static void
+RaiseSlotInvalidationError(ReplicationSlot *slot)
+{
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+
+ switch (slot->data.invalidated)

1a.
/invalidation cause of the slot./slot's invalidation cause./

~

1b.
This function does not expect to be called with slot->data.invalidated
== RS_INVAL_NONE, so I think it will be better to assert that
up-front.

~

1c.
This code could be simplified if you declare/initialize the variable
together, like:

StringInfo err_detail = makeStringInfo();

~~~

2.
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("This slot has been invalidated
because the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("This slot has been invalidated
because the required rows have been removed."));
+ break;

Since there are no format strings here, appendStringInfoString can be
used directly in some places.

======

FYI. I've attached a diffs patch that implements some of the
above-suggested changes.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  321. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-30T05:02:19Z

    Hi Nisha,

Here are some review comments for the patch v57-0001.

======
src/backend/replication/slot.c

1.
+/*
+ * Invalidate replication slots that have remained idle longer than this
+ * duration; '0' disables it.
+ */
+int idle_replication_slot_timeout_min = HOURS_PER_DAY * MINS_PER_HOUR;

IMO it would be better to have the suffix "_mins" instead of "_min"
here to avoid any confusion with "minimum".

~~~

2.
+/*
+ * Can invalidate an idle replication slot?
+ *

Not an English sentence.

======
src/backend/utils/adt/timestamp.c

3.
+ /* Return if the difference meets or exceeds the threshold */
+ return (secs >= threshold_sec);

That comment may not be necessary; it is saying just the same as the code.

======
src/backend/utils/misc/guc_tables.c

4.
+ {
+ {"idle_replication_slot_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the duration a replication slot can remain idle before "
+ "it is invalidated."),
+ NULL,
+ GUC_UNIT_MIN
+ },
+ &idle_replication_slot_timeout_min,
+ HOURS_PER_DAY * MINS_PER_HOUR, 0, INT_MAX / SECS_PER_MINUTE,
+ check_idle_replication_slot_timeout, NULL, NULL
+ },
+

Maybe it's better to include a comment that says "24 hours".

(e.g. like wal_summary_keep_time does)

======
src/backend/utils/misc/postgresql.conf.sample

5.
 #track_commit_timestamp = off # collect timestamp of transaction commit
  # (change requires restart)
+#idle_replication_slot_timeout = 1d # in minutes; 0 disables


I felt it might be better to say 24h here instead of 1d. And, that
would also be consistent with the docs, which said the default was 24
hours.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  322. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2024-12-30T05:34:45Z

    On Tue, Dec 24, 2024 at 10:37 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Fri, Dec 20, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Mon, Dec 16, 2024 at 4:10 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>> >
>> > Here is the v56 patch set with the above comments incorporated.
>> >
>>
>> Review comments:
>> ===============
>> 1.
>> + {
>> + {"idle_replication_slot_timeout", PGC_SIGHUP, REPLICATION_SENDING,
>> + gettext_noop("Sets the duration a replication slot can remain idle before "
>> + "it is invalidated."),
>> + NULL,
>> + GUC_UNIT_MS
>> + },
>> + &idle_replication_slot_timeout_ms,
>>
>> I think users are going to keep idele_slot timeout at least in hours.
>> So, millisecond seems the wrong choice to me. I suggest to keep the
>> units in minutes. I understand that writing a test would be
>> challenging as spending a minute or more on one test is not advisable.
>> But I don't see any test testing the other GUCs that are in minutes
>> (wal_summary_keep_time and log_rotation_age). The default value should
>> be one day.
>
>
> +1
> - Changed the GUC unit to "minute".
>
> Regarding the tests, we have two potential options:
>  1) Introduce an additional "debug_xx" GUC parameter with units of seconds or milliseconds, only for testing purposes.
>  2) Skip writing tests for this, similar to other GUCs with units in minutes.
>
> IMO, adding an additional GUC just for testing may not be worthwhile. It's reasonable to proceed without the test.
>
> Thoughts?
>
> The attached v57 patch-set addresses all the comments. I have kept the test case in the patch for now, it takes 2-3 minutes to complete.
>

Hi Nisha.

I think we are often too quick to throw out perfectly good tests.
Citing that some similar GUCs don't do testing as a reason to skip
them just seems to me like an example of "two wrongs don't make a
right".

There is a third option.

Keep the tests. Because they take excessive time to run, that simply
means you should run them *conditionally* based on the PG_TEST_EXTRA
environment variable so they don't impact the normal BF execution. The
documentation [1] says this env var is for "resource intensive" tests
-- AFAIK this is exactly the scenario we find ourselves in, so is
exactly what this env var was meant for.

Search other *.pl tests for PG_TEST_EXTRA to see some examples.

======
[1] https://www.postgresql.org/docs/17/regress-run.html

Kind Regards,
Peter Smith.
Fujitsu Australia

  323. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-31T07:00:32Z

    On Mon, Dec 30, 2024 at 11:05 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Dec 24, 2024 at 10:37 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > On Fri, Dec 20, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> On Mon, Dec 16, 2024 at 4:10 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >> >
> >> > Here is the v56 patch set with the above comments incorporated.
> >> >
> >>
> >> Review comments:
> >> ===============
> >> 1.
> >> + {
> >> + {"idle_replication_slot_timeout", PGC_SIGHUP, REPLICATION_SENDING,
> >> + gettext_noop("Sets the duration a replication slot can remain idle before "
> >> + "it is invalidated."),
> >> + NULL,
> >> + GUC_UNIT_MS
> >> + },
> >> + &idle_replication_slot_timeout_ms,
> >>
> >> I think users are going to keep idele_slot timeout at least in hours.
> >> So, millisecond seems the wrong choice to me. I suggest to keep the
> >> units in minutes. I understand that writing a test would be
> >> challenging as spending a minute or more on one test is not advisable.
> >> But I don't see any test testing the other GUCs that are in minutes
> >> (wal_summary_keep_time and log_rotation_age). The default value should
> >> be one day.
> >
> >
> > +1
> > - Changed the GUC unit to "minute".
> >
> > Regarding the tests, we have two potential options:
> >  1) Introduce an additional "debug_xx" GUC parameter with units of seconds or milliseconds, only for testing purposes.
> >  2) Skip writing tests for this, similar to other GUCs with units in minutes.
> >
> > IMO, adding an additional GUC just for testing may not be worthwhile. It's reasonable to proceed without the test.
> >
> > Thoughts?
> >
> > The attached v57 patch-set addresses all the comments. I have kept the test case in the patch for now, it takes 2-3 minutes to complete.
> >
>
> Hi Nisha.
>
> I think we are often too quick to throw out perfectly good tests.
> Citing that some similar GUCs don't do testing as a reason to skip
> them just seems to me like an example of "two wrongs don't make a
> right".
>
> There is a third option.
>
> Keep the tests. Because they take excessive time to run, that simply
> means you should run them *conditionally* based on the PG_TEST_EXTRA
> environment variable so they don't impact the normal BF execution. The
> documentation [1] says this env var is for "resource intensive" tests
> -- AFAIK this is exactly the scenario we find ourselves in, so is
> exactly what this env var was meant for.
>
> Search other *.pl tests for PG_TEST_EXTRA to see some examples.
>

Thank you for the suggestion! I’ve added the tests under the
PG_TEST_EXTRA condition. Now, the '043_invalidate_inactive_slots.pl'
tests will only be executed when
PG_TEST_EXTRA=idle_replication_slot_timeout is set.

Attached the v58 patch set, addressing the above and the comments in
[1], [2], and [3].

[1] https://www.postgresql.org/message-id/CALDaNm14QrW5j6su%2BEAqjwnHbiwXJwO%2Byk73_%3D7yvc5TVY-43g%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPvDsM%3D%2BvTbM-xX6DD-PavONs2kGn03MZbCPGGL2t60TRA%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CAHut%2BPs2ecNTfG3vsGb91CYpEzWtffyvkOzk1jqwhqTCwH8HQA%40mail.gmail.com

--
Thanks,
Nisha

  324. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2024-12-31T07:05:10Z

    On Fri, Dec 27, 2024 at 9:22 AM vignesh C <vignesh21@gmail.com> wrote:
>
>
> Few comments:
> 1) We have disabled the similar configuration max_slot_wal_keep_size
> by setting to -1, as this GUC also is in similar lines, should we
> disable this and let the user configure it?
> +/*
> + * Invalidate replication slots that have remained idle longer than this
> + * duration; '0' disables it.
> + */
> +int                    idle_replication_slot_timeout_min =
> HOURS_PER_DAY * MINS_PER_HOUR;
> +
>

I’m okay with setting the default to either '1-day' or 'Off'. Let’s
wait for feedback from others.

>
> 2) I felt this behavior is an existing behavior, so this can also be
> moved to 0001 patch:
> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
> index a586156614..199d7248ee 100644
> --- a/doc/src/sgml/system-views.sgml
> +++ b/doc/src/sgml/system-views.sgml
> @@ -2566,7 +2566,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
>        </para>
>        <para>
>          The time when the slot became inactive. <literal>NULL</literal> if the
> -        slot is currently being streamed.
> +        slot is currently being streamed. If the slot becomes invalidated,
> +        this value will remain unchanged until server shutdown.
>

You are correct that the 'inactive_since' value getting reset on
server restart has been the existing behavior.
However, earlier, there was no guarantee that it would remain
unchanged for invalid slots. The new function
"ReplicationSlotSetInactiveSince()" in patch 002, ensures that the
value does not change for invalidated slots until the server is shut
down. Therefore, I feel the doc addition in patch 002 is appropriate.

>
> 3) Can we change the comment below to "We don't allow the value of
> idle_replication_slot_timeout other than 0 during the binary upgrade.
> See start_postmaster() in pg_upgrade for more details.":
> + * The idle_replication_slot_timeout must be disabled (set to 0)
> + * during the binary upgrade.
> + */
> +bool
> +check_idle_replication_slot_timeout(int *newval, void **extra,
> GucSource source)

Done.

--
Thanks,
Nisha

  325. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-01-02T00:13:38Z

    Hi Nisha.

My review comments for patch v58-0001.

======
src/backend/replication/slot.c

InvalidatePossiblyObsoleteSlot:

1.
  /*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately.  Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
  */
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s && active_pid == MyProcPid))
  {

As you previously explained [1] "This change applies to all types of
invalidation, not just inactive_timeout case [...] It's a general
optimization for the case when the current process is the active PID
for the slot."

In that case, should this be in a separate patch that can be pushed to
master by itself, i.e. independent of anything else in this thread
that is being done for the purpose of implementing the timeout
feature?

======
[1] https://www.postgresql.org/message-id/CABdArM5tcYTQ2zeAPWTciTnea4jj6sPUjVY9M1O-4wWoTBjFgw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Austalia

  326. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-01-02T02:46:20Z

    Hi Nisha,

Here are some minor review comments for patch v58-0002.

======
src/backend/replication/slot.c

check_replication_slot_inactive_timeout:

1.
+
+/*
+ * GUC check_hook for idle_replication_slot_timeout
+ *
+ * We don't allow the value of idle_replication_slot_timeout other
+ * than 0 during the binary upgrade.
+ * See start_postmaster() in pg_upgrade for more details.
+ */

If you want to express it this way, then it seems there are some
wrong/missing words:

SUGGESTION #1.
We don't allow any value of idle_replication_slot_timeout other than 0
during a binary upgrade.

SUGGESTION #2.
We don't allow the value of idle_replication_slot_timeout to be other
than 0 during a binary upgrade.

~

(But, I prefer more terse comments which are not negative-sounding. YMMV).

SUGGESTION #3 (nearly identical text to the actual error message)
The value of idle_replication_slot_timeout must be set to 0 during a
binary upgrade.

======
src/test/recovery/README

2.
+If you want to test idle_replication_slot_timeout, add
+PG_TEST_EXTRA=idle_replication_slot_timeout
+to the "make" command. The test takes over 2 minutes, so not done
+by default.
+

Maybe it's better to use consistent wording with the other tests like
this one already in the README:

/The test/This test/

/so not done by default./so it's not done by default./


======
.../t/043_invalidate_inactive_slots.pl

3.
+# Copyright (c) 2024, PostgreSQL Global Development Group
+

Happy New Year.

/2024/2025/

~~~

4.
+
+# The test takes over two minutes to complete. Run it only if
+# idle_replication_slot_timeout is specified in PG_TEST_EXTRA.
+if (  !$ENV{PG_TEST_EXTRA}
+ || $ENV{PG_TEST_EXTRA} !~ /\bidle_replication_slot_timeout\b/)
+{
+ plan skip_all =>
+   'A time consuming test, idle_replication_slot_timeout is not
enabled in PG_TEST_EXTRA';
+}

4a.
I noticed the other skipping TAP tests like this have a simpler
message without giving a reason, so maybe it's better to be consistent
with those:

SUGGESTION:
plan skip_all => "test idle_replication_slot_timeout not enabled in
PG_TEST_EXTRA";

~

4b.
Should the check be done right at the top of the file (e.g. even
before the "# Testcase start" comment)?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  327. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-02T10:26:28Z

    On Thu, Jan 2, 2025 at 5:44 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha.
>
> My review comments for patch v58-0001.
>
> ======
> src/backend/replication/slot.c
>
> InvalidatePossiblyObsoleteSlot:
>
> 1.
>   /*
> - * If the slot can be acquired, do so and mark it invalidated
> - * immediately.  Otherwise we'll signal the owning process, below, and
> - * retry.
> + * If the slot can be acquired, do so and mark it as invalidated. If
> + * the slot is already ours, mark it as invalidated. Otherwise, we'll
> + * signal the owning process below and retry.
>   */
> - if (active_pid == 0)
> + if (active_pid == 0 ||
> + (MyReplicationSlot == s && active_pid == MyProcPid))
>   {
>
> As you previously explained [1] "This change applies to all types of
> invalidation, not just inactive_timeout case [...] It's a general
> optimization for the case when the current process is the active PID
> for the slot."
>
> In that case, should this be in a separate patch that can be pushed to
> master by itself, i.e. independent of anything else in this thread
> that is being done for the purpose of implementing the timeout
> feature?

The patch-001 has additional general optimizations similar to the one
you mentioned, which are not strictly required for this feature.
Let’s wait for input from others on splitting the patches or
addressing it in a separate thread.

--
Thanks,
Nisha

  328. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-02T10:27:10Z

    On Thu, Jan 2, 2025 at 8:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha,
>
> Here are some minor review comments for patch v58-0002.
>

Thank you for your feedback! Please find the v59 patch set addressing
all the comments.
Note: There are no new changes in patch-0001.

--
Thanks,
Nisha

  329. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-01-13T06:52:04Z

    On Thu, 2 Jan 2025 at 15:57, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Thu, Jan 2, 2025 at 8:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Nisha,
> >
> > Here are some minor review comments for patch v58-0002.
> >
>
> Thank you for your feedback! Please find the v59 patch set addressing
> all the comments.
> Note: There are no new changes in patch-0001.

Few comments:
1) I felt we should not invalidate the slots for which has no
effective xmin set as they will not be holding the WAL files from
deletion. This can happen when user created slots with
immediately_reserve as false and lsn will be actually reserved only
after the first connection from a streaming replication client:
+static inline bool
+CanInvalidateIdleSlot(ReplicationSlot *s)
+{
+       return (idle_replication_slot_timeout_mins > 0 &&
+                       s->inactive_since > 0 &&
+                       !(RecoveryInProgress() && s->data.synced));
+}

2) We can mention this as 1d  instead of 24h as we want to represent 1
day similar to how we have mentioned for log_rotation_age:
index a2ac7575ca..7284edfbc1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -337,6 +337,7 @@
 #wal_sender_timeout = 60s      # in milliseconds; 0 disables
 #track_commit_timestamp = off  # collect timestamp of transaction commit
                                # (change requires restart)
+#idle_replication_slot_timeout = 24h   # in minutes; 0 disables

Regards,
Vignesh

  330. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-01-13T07:18:28Z

    On Mon, Jan 13, 2025 at 5:52 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 2 Jan 2025 at 15:57, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > On Thu, Jan 2, 2025 at 8:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Nisha,
> > >
> > > Here are some minor review comments for patch v58-0002.
> > >
...
>
> 2) We can mention this as 1d  instead of 24h as we want to represent 1
> day similar to how we have mentioned for log_rotation_age:
> index a2ac7575ca..7284edfbc1 100644
> --- a/src/backend/utils/misc/postgresql.conf.sample
> +++ b/src/backend/utils/misc/postgresql.conf.sample
> @@ -337,6 +337,7 @@
>  #wal_sender_timeout = 60s      # in milliseconds; 0 disables
>  #track_commit_timestamp = off  # collect timestamp of transaction commit
>                                 # (change requires restart)
> +#idle_replication_slot_timeout = 24h   # in minutes; 0 disables
>

Hi Vignesh. AFAIK that is due to a previous review comment of mine
where I suggested we should use 24h format here, because this GUC
default is described as "24 hours" in the config.sgml, and I felt the
sample should match its own documentation.

======
Kind Regards,
Peter Smith.
Fujitsu Australia.

  331. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-01-15T03:49:16Z

    On Mon, 13 Jan 2025 at 12:48, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Jan 13, 2025 at 5:52 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Thu, 2 Jan 2025 at 15:57, Nisha Moond <nisha.moond412@gmail.com> wrote:
> > >
> > > On Thu, Jan 2, 2025 at 8:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Nisha,
> > > >
> > > > Here are some minor review comments for patch v58-0002.
> > > >
> ...
> >
> > 2) We can mention this as 1d  instead of 24h as we want to represent 1
> > day similar to how we have mentioned for log_rotation_age:
> > index a2ac7575ca..7284edfbc1 100644
> > --- a/src/backend/utils/misc/postgresql.conf.sample
> > +++ b/src/backend/utils/misc/postgresql.conf.sample
> > @@ -337,6 +337,7 @@
> >  #wal_sender_timeout = 60s      # in milliseconds; 0 disables
> >  #track_commit_timestamp = off  # collect timestamp of transaction commit
> >                                 # (change requires restart)
> > +#idle_replication_slot_timeout = 24h   # in minutes; 0 disables
> >
>
> Hi Vignesh. AFAIK that is due to a previous review comment of mine
> where I suggested we should use 24h format here, because this GUC
> default is described as "24 hours" in the config.sgml, and I felt the
> sample should match its own documentation.

I suggest we reverse the current approach: change the default
configuration value to 1d and update the documentation accordingly. I
preferred to set default values of 1h instead of 60 mins, 1d instead
of 24h, etc.

Regards,
Vignesh

  332. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2025-01-15T06:07:43Z

    On Thu, 2 Jan 2025 at 15:57, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Thu, Jan 2, 2025 at 8:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Nisha,
> >
> > Here are some minor review comments for patch v58-0002.
> >
>
> Thank you for your feedback! Please find the v59 patch set addressing
> all the comments.
> Note: There are no new changes in patch-0001.
>

Hi Nisha,
I reviewed the v59-0001 patch. I have few comments:

1.I think we should update the comment for function
'InvalidatePossiblyObsoleteSlot'
Currently the comment is like:

/*
 * Helper for InvalidateObsoleteReplicationSlots
 *
 * Acquires the given slot and mark it invalid, if necessary and possible.
 *
 * Returns whether ReplicationSlotControlLock was released in the interim (and
 * in that case we're not holding the lock at return, otherwise we are).
 *
 * Sets *invalidated true if the slot was invalidated. (Untouched otherwise.)
 *
 * This is inherently racy, because we release the LWLock
 * for syscalls, so caller must restart if we return true.
 */

I think we should add a comment for the case 'when slot is already ours'.

2. Similarly we should update comment here:
        /*
         * Check if the slot needs to be invalidated. If it needs to be
         * invalidated, and is not currently acquired, acquire it and mark it
         * as having been invalidated.  We do this with the spinlock held to
         * avoid race conditions -- for example the restart_lsn could move
         * forward, or the slot could be dropped.
         */
        SpinLockAcquire(&s->mutex);

Before we release the lock, we are marking the slot as invalidated for
the case when the slot is already acquired by our process. So we
should update it in comment.

3. I think we should also update the following 'if condition':

if (active_pid != 0)
        {
            /*
             * Prepare the sleep on the slot's condition variable before
             * releasing the lock, to close a possible race condition if the
             * slot is released before the sleep below.
             */
We should not enter the if condition for the case when the slot was
already acquired by our process.

Thanks and Regards,
Shlok Kyal

  333. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-16T07:05:16Z

    On Wed, Jan 15, 2025 at 11:37 AM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> On Thu, 2 Jan 2025 at 15:57, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > On Thu, Jan 2, 2025 at 8:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Nisha,
> > >
> > > Here are some minor review comments for patch v58-0002.
> > >
> >
> > Thank you for your feedback! Please find the v59 patch set addressing
> > all the comments.
> > Note: There are no new changes in patch-0001.
> >
>
> Hi Nisha,
> I reviewed the v59-0001 patch. I have few comments:
>
> 1.I think we should update the comment for function
> 'InvalidatePossiblyObsoleteSlot'
> Currently the comment is like:
>
> /*
>  * Helper for InvalidateObsoleteReplicationSlots
>  *
>  * Acquires the given slot and mark it invalid, if necessary and possible.
>  *
>  * Returns whether ReplicationSlotControlLock was released in the interim (and
>  * in that case we're not holding the lock at return, otherwise we are).
>  *
>  * Sets *invalidated true if the slot was invalidated. (Untouched otherwise.)
>  *
>  * This is inherently racy, because we release the LWLock
>  * for syscalls, so caller must restart if we return true.
>  */
>
> I think we should add a comment for the case 'when slot is already ours'.

Done.

> 2. Similarly we should update comment here:
>         /*
>          * Check if the slot needs to be invalidated. If it needs to be
>          * invalidated, and is not currently acquired, acquire it and mark it
>          * as having been invalidated.  We do this with the spinlock held to
>          * avoid race conditions -- for example the restart_lsn could move
>          * forward, or the slot could be dropped.
>          */
>         SpinLockAcquire(&s->mutex);
>
> Before we release the lock, we are marking the slot as invalidated for
> the case when the slot is already acquired by our process. So we
> should update it in comment.
>

Clarified the comments as per the mentioned case.

> 3. I think we should also update the following 'if condition':
>
> if (active_pid != 0)
>         {
>             /*
>              * Prepare the sleep on the slot's condition variable before
>              * releasing the lock, to close a possible race condition if the
>              * slot is released before the sleep below.
>              */
> We should not enter the if condition for the case when the slot was
> already acquired by our process.
>

Thank you for pointing that out. I've included the fix and also
reorganized this section of the code in patch-0001 to improve the
readability of the logic.

Attached the v60 patch set addressing above comments and all other
comments at [1].

[1] https://www.postgresql.org/message-id/CALDaNm2r969ZZPDaAZQEtxcfL-sGUW8AGdbdwC8AcMn1V8w%2Bhw%40mail.gmail.com

--
Thanks,
Nisha

  334. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-16T07:07:18Z

    On Mon, Jan 13, 2025 at 12:22 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 2 Jan 2025 at 15:57, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Thank you for your feedback! Please find the v59 patch set addressing
> > all the comments.
> > Note: There are no new changes in patch-0001.
>
> Few comments:
> 1) I felt we should not invalidate the slots for which has no
> effective xmin set as they will not be holding the WAL files from
> deletion. This can happen when user created slots with
> immediately_reserve as false and lsn will be actually reserved only
> after the first connection from a streaming replication client:
> +static inline bool
> +CanInvalidateIdleSlot(ReplicationSlot *s)
> +{
> +       return (idle_replication_slot_timeout_mins > 0 &&
> +                       s->inactive_since > 0 &&
> +                       !(RecoveryInProgress() && s->data.synced));
> +}
>

IIUC, for both logical and physical replication slots, the effective
xmin remains NULL until the respective nodes establish their first
connection. However, logical slots always reserve WAL immediately,
whereas physical slots do not unless immediately_reserve=true is set.
To avoid unnecessary slot invalidation for slots that are not
reserving WAL when created, it might be better to check the
restart_lsn instead of effective_xmin. If restart_lsn is invalid for a
slot, it indicates that WAL is not reserved for the slot, and we can
safely skip invalidation due to idle_timeout for such slots. This
logic has been implemented in v60.

--
Thanks,
Nisha

  335. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2025-01-17T13:20:41Z

    On Thu, 16 Jan 2025 at 12:35, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Wed, Jan 15, 2025 at 11:37 AM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > On Thu, 2 Jan 2025 at 15:57, Nisha Moond <nisha.moond412@gmail.com> wrote:
> > >
> > > On Thu, Jan 2, 2025 at 8:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Nisha,
> > > >
> > > > Here are some minor review comments for patch v58-0002.
> > > >
> > >
> > > Thank you for your feedback! Please find the v59 patch set addressing
> > > all the comments.
> > > Note: There are no new changes in patch-0001.
> > >
> >
> > Hi Nisha,
> > I reviewed the v59-0001 patch. I have few comments:
> >
> > 1.I think we should update the comment for function
> > 'InvalidatePossiblyObsoleteSlot'
> > Currently the comment is like:
> >
> > /*
> >  * Helper for InvalidateObsoleteReplicationSlots
> >  *
> >  * Acquires the given slot and mark it invalid, if necessary and possible.
> >  *
> >  * Returns whether ReplicationSlotControlLock was released in the interim (and
> >  * in that case we're not holding the lock at return, otherwise we are).
> >  *
> >  * Sets *invalidated true if the slot was invalidated. (Untouched otherwise.)
> >  *
> >  * This is inherently racy, because we release the LWLock
> >  * for syscalls, so caller must restart if we return true.
> >  */
> >
> > I think we should add a comment for the case 'when slot is already ours'.
>
> Done.
>
> > 2. Similarly we should update comment here:
> >         /*
> >          * Check if the slot needs to be invalidated. If it needs to be
> >          * invalidated, and is not currently acquired, acquire it and mark it
> >          * as having been invalidated.  We do this with the spinlock held to
> >          * avoid race conditions -- for example the restart_lsn could move
> >          * forward, or the slot could be dropped.
> >          */
> >         SpinLockAcquire(&s->mutex);
> >
> > Before we release the lock, we are marking the slot as invalidated for
> > the case when the slot is already acquired by our process. So we
> > should update it in comment.
> >
>
> Clarified the comments as per the mentioned case.
>
> > 3. I think we should also update the following 'if condition':
> >
> > if (active_pid != 0)
> >         {
> >             /*
> >              * Prepare the sleep on the slot's condition variable before
> >              * releasing the lock, to close a possible race condition if the
> >              * slot is released before the sleep below.
> >              */
> > We should not enter the if condition for the case when the slot was
> > already acquired by our process.
> >
>
> Thank you for pointing that out. I've included the fix and also
> reorganized this section of the code in patch-0001 to improve the
> readability of the logic.
>
> Attached the v60 patch set addressing above comments and all other
> comments at [1].
>
> [1] https://www.postgresql.org/message-id/CALDaNm2r969ZZPDaAZQEtxcfL-sGUW8AGdbdwC8AcMn1V8w%2Bhw%40mail.gmail.com
>
Hi Nisha,

Thanks for providing an updated patch. I have tested the patch and ran
some tests. The patch works fine. I have few comments:

v60-0001 patch:
1) There is extra line before 'SpinLockRelease(&s->mutex)' :

+       else                    /* Some other process owns the slot */
        {
+
+           SpinLockRelease(&s->mutex);

v60-0002 patch:
1) In the comment:

/*
 * Invalidate slots that require resources about to be removed.
 *
 * Returns true when any slot have got invalidated.
 *
 * Whether a slot needs to be invalidated depends on the cause. A slot is
 * removed if it:
 * - RS_INVAL_WAL_REMOVED: requires a LSN older than the given segment
 * - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
 *   db; dboid may be InvalidOid for shared relations
 * - RS_INVAL_WAL_LEVEL: is logical
 * - RS_INVAL_IDLE_TIMEOUT: idle slot timeout has occurred
 *
 * NB - this runs as part of checkpoint, so avoid raising errors if possible.
 */

It is mentioned that 'A slot is removed if it:'. I think instead of
'removed' it should be 'invalidated'.

Thanks and regards,
Shlok Kyal

  336. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-20T08:03:54Z

    On Fri, Jan 17, 2025 at 6:50 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> Hi Nisha,
>
> Thanks for providing an updated patch. I have tested the patch and ran
> some tests. The patch works fine. I have few comments:
>

Thanks for your review. Attached are the v61 patches.

I've addressed the comments and rebased patches as needed due to the
latest changes on pgHead. The patch-0002 test file name has been
updated from "043_invalidate_inactive_slots.pl" to
"044_invalidate_inactive_slots.pl".

--
Thanks,
Nisha

  337. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-01-21T02:51:51Z

    Some review comments for patch v61-0001.

======
src/backend/replication/slot.c

InvalidatePossiblyObsoleteSlot:

1.
  /*
  * Check if the slot needs to be invalidated. If it needs to be
- * invalidated, and is not currently acquired, acquire it and mark it
- * as having been invalidated.  We do this with the spinlock held to
- * avoid race conditions -- for example the restart_lsn could move
- * forward, or the slot could be dropped.
+ * invalidated, and is already ours, mark it as having been
+ * invalidated; otherwise, acquire it first and then mark it as having
+ * been invalidated. We do this with the spinlock held to avoid race
+ * conditions -- for example the restart_lsn could move forward, or
+ * the slot could be dropped.
  */

Can't you just word this as "mark it as invalidated" (which you do
later anyway) instead of "mark is as having been invalidated"?  (this
is in two places).

~~~

2.
+ /*
+ * The logical replication slots shouldn't be invalidated as GUC
+ * max_slot_wal_keep_size is set to -1 during the binary upgrade.
+ * See check_old_cluster_for_valid_slots() where we ensure that no
+ * invalidated before the upgrade.
+ */
+ Assert(!(*invalidated && SlotIsLogical(s) && IsBinaryUpgrade));

2a.
I know this sentence was the same before you moved it, but "ensure
that no invalidated" seems like there are some words missing.

2b.
TBH, I this part confused me (the code is repeated in a couple of
places). AFAICT the code/comment does not match quite right. The
comment refers to a setting that is "during binary upgrade", yet the
Assert can only pass if IsBinaryUpgrade is false. (??)

I'm unsure of the intent; perhaps it should be like:

if (IsBinaryUpgrade)
Assert(!(*invalidated && SlotIsLogical(s)));

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  338. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-01-21T02:55:39Z

    Some review comments for patch v61-0002

======
src/backend/replication/slot.c

1.
  * Whether a slot needs to be invalidated depends on the cause. A slot is
- * removed if it:
+ * invalidated if it:
  * - RS_INVAL_WAL_REMOVED: requires a LSN older than the given segment
  * - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
  *   db; dboid may be InvalidOid for shared relations
  * - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_IDLE_TIMEOUT: idle slot timeout has occurred

1a.
Firstly the wording seems odd. "Is invalidated if it:" (missing words?)

~

1b.
Secondly, is this comment strictly correct? IIUC it's not *always*
going to be invalidated just because the cause is one of those listed.
e.g. the code calls InvalidatePossiblyObsoleteSlot but it might not
end up invalidating the slot having a cause RS_INVAL_IDLE_TIMEOUT.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  339. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-22T05:16:45Z

    On Tue, Jan 21, 2025 at 8:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Some review comments for patch v61-0002
>
> ======
> src/backend/replication/slot.c
>
> 1.
>   * Whether a slot needs to be invalidated depends on the cause. A slot is
> - * removed if it:
> + * invalidated if it:
>   * - RS_INVAL_WAL_REMOVED: requires a LSN older than the given segment
>   * - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
>   *   db; dboid may be InvalidOid for shared relations
>   * - RS_INVAL_WAL_LEVEL: is logical
> + * - RS_INVAL_IDLE_TIMEOUT: idle slot timeout has occurred
>
> 1a.
> Firstly the wording seems odd. "Is invalidated if it:" (missing words?)
>
> ~
>
> 1b.
> Secondly, is this comment strictly correct? IIUC it's not *always*
> going to be invalidated just because the cause is one of those listed.
> e.g. the code calls InvalidatePossiblyObsoleteSlot but it might not
> end up invalidating the slot having a cause RS_INVAL_IDLE_TIMEOUT.
>

I feel the phrase "A slot is invalidated if it:" is supposed to be
read alongside the respective cause description, such as:
  "A slot is invalidated if it requires an LSN older than…"
  "A slot is invalidated if it requires a snapshot <= the…"
  "A slot is invalidated if it is logical"

IIUC, each listed cause specifies a clear condition under which the
slot should *always* be invalidated for that cause. To maintain
consistency with the header line "A slot is invalidated if it:", I’ve
modified the description/condition for RS_INVAL_IDLE_TIMEOUT
accordingly. Also, corrected the RS_INVAL_WAL_LEVEL description.

Attached the v62 patch set with above mentioned changes and also
addressed comments at [1].

[1] https://www.postgresql.org/message-id/CAHut%2BPvC4uPabeGMvDuTQ4S%2B5eX66Y6%2BtU5QMRmB2jDw-Cj2Cw%40mail.gmail.com

--
Thanks,
Nisha

  340. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-22T05:19:28Z

    On Tue, Jan 21, 2025 at 8:22 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Some review comments for patch v61-0001.
>
> ======
> src/backend/replication/slot.c
>
> InvalidatePossiblyObsoleteSlot:
>
> 1.
>   /*
>   * Check if the slot needs to be invalidated. If it needs to be
> - * invalidated, and is not currently acquired, acquire it and mark it
> - * as having been invalidated.  We do this with the spinlock held to
> - * avoid race conditions -- for example the restart_lsn could move
> - * forward, or the slot could be dropped.
> + * invalidated, and is already ours, mark it as having been
> + * invalidated; otherwise, acquire it first and then mark it as having
> + * been invalidated. We do this with the spinlock held to avoid race
> + * conditions -- for example the restart_lsn could move forward, or
> + * the slot could be dropped.
>   */
>
> Can't you just word this as "mark it as invalidated" (which you do
> later anyway) instead of "mark is as having been invalidated"?  (this
> is in two places).
>

Thanks for pointing it out, I had considered it but was trying to keep
the language consistent with the previous style. I've now made the
change in v62.

> ~~~
>
> 2.
> + /*
> + * The logical replication slots shouldn't be invalidated as GUC
> + * max_slot_wal_keep_size is set to -1 during the binary upgrade.
> + * See check_old_cluster_for_valid_slots() where we ensure that no
> + * invalidated before the upgrade.
> + */
> + Assert(!(*invalidated && SlotIsLogical(s) && IsBinaryUpgrade));
>
> 2a.
> I know this sentence was the same before you moved it, but "ensure
> that no invalidated" seems like there are some words missing.
>

Corrected.

> 2b.
> TBH, I this part confused me (the code is repeated in a couple of
> places). AFAICT the code/comment does not match quite right. The
> comment refers to a setting that is "during binary upgrade", yet the
> Assert can only pass if IsBinaryUpgrade is false. (??)
>
> I'm unsure of the intent; perhaps it should be like:
>
> if (IsBinaryUpgrade)
> Assert(!(*invalidated && SlotIsLogical(s)));
>

This Assert condition is correct, as we don’t want to invalidate slots
during a binary upgrade and it only triggers(raise error) when all
three conditions are true, meaning when 'IsBinaryUpgrade' is also
true.
That said, I agree with your point that we are unnecessarily checking
"(*invalidated && SlotIsLogical(s))" even when not in binary upgrade
mode.
To optimize this, we can first check 'IsBinaryUpgrade' before
evaluating the other conditions:
    Assert(!(IsBinaryUpgrade && *invalidated && SlotIsLogical(s)));
 - Since 'IsBinaryUpgrade' is 'false' most of the time, this approach
short-circuits evaluation, making it more efficient.

However, if you feel that this condition isn’t as clear to read or
understand, we can separate the 'IsBinaryUpgrade' check outside the
'Assert', as you suggested. Let me know what you think!

--
Thanks,
Nisha

  341. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-27T05:30:15Z

    On Wed, Jan 22, 2025 at 10:49 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Tue, Jan 21, 2025 at 8:22 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Some review comments for patch v61-0001.
> >
> > ======
> > src/backend/replication/slot.c
> >
> > 2.
> > + /*
> > + * The logical replication slots shouldn't be invalidated as GUC
> > + * max_slot_wal_keep_size is set to -1 during the binary upgrade.
> > + * See check_old_cluster_for_valid_slots() where we ensure that no
> > + * invalidated before the upgrade.
> > + */
> > + Assert(!(*invalidated && SlotIsLogical(s) && IsBinaryUpgrade));
> >
> > 2a.
> > I know this sentence was the same before you moved it, but "ensure
> > that no invalidated" seems like there are some words missing.
> >
>
> Corrected.
>
> > 2b.
> > TBH, I this part confused me (the code is repeated in a couple of
> > places). AFAICT the code/comment does not match quite right. The
> > comment refers to a setting that is "during binary upgrade", yet the
> > Assert can only pass if IsBinaryUpgrade is false. (??)
> >
> > I'm unsure of the intent; perhaps it should be like:
> >
> > if (IsBinaryUpgrade)
> > Assert(!(*invalidated && SlotIsLogical(s)));
> >
>
> This Assert condition is correct, as we don’t want to invalidate slots
> during a binary upgrade and it only triggers(raise error) when all
> three conditions are true, meaning when 'IsBinaryUpgrade' is also
> true.
> That said, I agree with your point that we are unnecessarily checking
> "(*invalidated && SlotIsLogical(s))" even when not in binary upgrade
> mode.
> To optimize this, we can first check 'IsBinaryUpgrade' before
> evaluating the other conditions:
>     Assert(!(IsBinaryUpgrade && *invalidated && SlotIsLogical(s)));
>  - Since 'IsBinaryUpgrade' is 'false' most of the time, this approach
> short-circuits evaluation, making it more efficient.
>
> However, if you feel that this condition isn’t as clear to read or
> understand, we can separate the 'IsBinaryUpgrade' check outside the
> 'Assert', as you suggested. Let me know what you think!
>

I discussed the above comments further with Peter off-list, and here
are the v63 patches with the following changes:
 patch-001: The Assert and related comments have been updated for clarity.
 patch-002: Comments have been updated at the top of
InvalidateObsoleteReplicationSlots().

--
Thanks,
Nisha

  342. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-27T10:50:05Z

    On Mon, Jan 27, 2025 at 11:00 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> I discussed the above comments further with Peter off-list, and here
> are the v63 patches with the following changes:
>  patch-001: The Assert and related comments have been updated for clarity.
>

The 0001 patch should be discussed in a separate thread as those are
general improvements that are useful even without the main patch we
are trying to achieve in this thread. I suggest we break it into three
patches (a) Ensure the same inactive_since time for all slots, (b)
Raise an error for invalid slots during ReplicationSlotAcquire(); tell
in the commit message, without this patch when such an ERROR would
have otherwise occurred, and (c) Changes in
InvalidatePossiblyObsoleteSlot(), I suggest to leave this change for
later as this impacts the core logic of invalidation.

*
@@ -812,7 +823,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
  Assert(MyReplicationSlot == NULL);
  Assert(failover || two_phase);

- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);

Why don't we want to give ERROR during Alter? I think it is okay to
not give ERROR for invalid slots during Drop as we are anyway removing
such slots.

-- 
With Regards,
Amit Kapila.

  343. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-28T09:56:13Z

    On Mon, Dec 30, 2024 at 11:05 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> I think we are often too quick to throw out perfectly good tests.
> Citing that some similar GUCs don't do testing as a reason to skip
> them just seems to me like an example of "two wrongs don't make a
> right".
>
> There is a third option.
>
> Keep the tests. Because they take excessive time to run, that simply
> means you should run them *conditionally* based on the PG_TEST_EXTRA
> environment variable so they don't impact the normal BF execution. The
> documentation [1] says this env var is for "resource intensive" tests
> -- AFAIK this is exactly the scenario we find ourselves in, so is
> exactly what this env var was meant for.
>
> Search other *.pl tests for PG_TEST_EXTRA to see some examples.
>

I don't see the long-running tests to be added under PG_TEST_EXTRA as
that will make it unusable after some point. Now, if multiple senior
members feel it is okay to add long-running tests under PG_TEST_EXTRA
then I am open to considering it. We can keep this test as a separate
patch so that the patch is being tested in CI or in manual tests
before commit.

-- 
With Regards,
Amit Kapila.

  344. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-28T11:56:51Z

    On Mon, Jan 27, 2025 at 4:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 27, 2025 at 11:00 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > I discussed the above comments further with Peter off-list, and here
> > are the v63 patches with the following changes:
> >  patch-001: The Assert and related comments have been updated for clarity.
> >
>
> The 0001 patch should be discussed in a separate thread as those are
> general improvements that are useful even without the main patch we
> are trying to achieve in this thread. I suggest we break it into three
> patches (a) Ensure the same inactive_since time for all slots, (b)
> Raise an error for invalid slots during ReplicationSlotAcquire(); tell
> in the commit message, without this patch when such an ERROR would
> have otherwise occurred, and (c) Changes in
> InvalidatePossiblyObsoleteSlot(), I suggest to leave this change for
> later as this impacts the core logic of invalidation.
>

I have started a new thread for these general improvements and have
separated the changes (a) and (b) into different patches.

You can find the new thread at [1].

> *
> @@ -812,7 +823,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
>   Assert(MyReplicationSlot == NULL);
>   Assert(failover || two_phase);
>
> - ReplicationSlotAcquire(name, false);
> + ReplicationSlotAcquire(name, false, false);
>
> Why don't we want to give ERROR during Alter? I think it is okay to
> not give ERROR for invalid slots during Drop as we are anyway removing
> such slots.
>

Because ReplicationSlotAlter() already handles errors immediately
after acquiring the slot. It raises errors for invalidated slots and
also raises a different error message if the slot is a physical one.
So, In case of ALTER, I feel it is okay to acquire the slot first
without raising errors and then handle errors in the pre-defined way.
Similar immediate error handling is not available at other places.

[1] https://www.postgresql.org/message-id/CABdArM6pBL5hPnSQ%2B5nEVMANcF4FCH7LQmgskXyiLY75TMnKpw%40mail.gmail.com

--
Thanks,
Nisha

  345. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-28T11:58:29Z

    On Tue, Jan 28, 2025 at 3:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 30, 2024 at 11:05 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > I think we are often too quick to throw out perfectly good tests.
> > Citing that some similar GUCs don't do testing as a reason to skip
> > them just seems to me like an example of "two wrongs don't make a
> > right".
> >
> > There is a third option.
> >
> > Keep the tests. Because they take excessive time to run, that simply
> > means you should run them *conditionally* based on the PG_TEST_EXTRA
> > environment variable so they don't impact the normal BF execution. The
> > documentation [1] says this env var is for "resource intensive" tests
> > -- AFAIK this is exactly the scenario we find ourselves in, so is
> > exactly what this env var was meant for.
> >
> > Search other *.pl tests for PG_TEST_EXTRA to see some examples.
> >
>
> I don't see the long-running tests to be added under PG_TEST_EXTRA as
> that will make it unusable after some point. Now, if multiple senior
> members feel it is okay to add long-running tests under PG_TEST_EXTRA
> then I am open to considering it. We can keep this test as a separate
> patch so that the patch is being tested in CI or in manual tests
> before commit.
>

Please find the attached v64 patches. The changes in this version
w.r.t. older patch v63 are as -
- The changes from the v63-0001 patch have been moved to a separate thread [1].
- The v63-0002 patch has been split into two parts in v64:
  1) 001 patch: Implements the main feature - inactive timeout-based
slot invalidation.
  2) 002 patch: Separates the TAP test "044_invalidate_inactive_slots"
as suggested above.

[1] https://www.postgresql.org/message-id/CABdArM6pBL5hPnSQ%2B5nEVMANcF4FCH7LQmgskXyiLY75TMnKpw%40mail.gmail.com

--
Thanks,
Nisha

  346. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-01-29T06:15:18Z

    On Tue, 28 Jan 2025 at 17:28, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Tue, Jan 28, 2025 at 3:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Dec 30, 2024 at 11:05 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > I think we are often too quick to throw out perfectly good tests.
> > > Citing that some similar GUCs don't do testing as a reason to skip
> > > them just seems to me like an example of "two wrongs don't make a
> > > right".
> > >
> > > There is a third option.
> > >
> > > Keep the tests. Because they take excessive time to run, that simply
> > > means you should run them *conditionally* based on the PG_TEST_EXTRA
> > > environment variable so they don't impact the normal BF execution. The
> > > documentation [1] says this env var is for "resource intensive" tests
> > > -- AFAIK this is exactly the scenario we find ourselves in, so is
> > > exactly what this env var was meant for.
> > >
> > > Search other *.pl tests for PG_TEST_EXTRA to see some examples.
> > >
> >
> > I don't see the long-running tests to be added under PG_TEST_EXTRA as
> > that will make it unusable after some point. Now, if multiple senior
> > members feel it is okay to add long-running tests under PG_TEST_EXTRA
> > then I am open to considering it. We can keep this test as a separate
> > patch so that the patch is being tested in CI or in manual tests
> > before commit.
> >
>
> Please find the attached v64 patches. The changes in this version
> w.r.t. older patch v63 are as -
> - The changes from the v63-0001 patch have been moved to a separate thread [1].
> - The v63-0002 patch has been split into two parts in v64:
>   1) 001 patch: Implements the main feature - inactive timeout-based
> slot invalidation.
>   2) 002 patch: Separates the TAP test "044_invalidate_inactive_slots"
> as suggested above.
>
> [1] https://www.postgresql.org/message-id/CABdArM6pBL5hPnSQ%2B5nEVMANcF4FCH7LQmgskXyiLY75TMnKpw%40mail.gmail.com

Few comments:
1) We can mention about the slot that do not reserve WAL is also not applicable:
+       <para>
+        Note that the idle timeout invalidation mechanism is not
+        applicable for slots on the standby server that are being synced
+        from the primary server (i.e., standby slots having
+        <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+        value <literal>true</literal>).
+        Synced slots are always considered to be inactive because they don't
+        perform logical decoding to produce changes.
+       </para>

2) Similarly we can mention in the commit message also that it will
not be considered for slot that do not reserve WAL:
Note that the idle timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.

3) Since idle_replication_slot_timeout is somewhat similar to
max_slot_wal_keep_size, we can move idle_replication_slot_timeout
after max_slot_wal_keep_size instead of keeping it after
wal_sender_timeout.
+     <varlistentry id="guc-idle-replication-slot-timeout"
xreflabel="idle_replication_slot_timeout">
+      <term><varname>idle_replication_slot_timeout</varname>
(<type>integer</type>)
+      <indexterm>
+       <primary><varname>idle_replication_slot_timeout</varname>
configuration parameter</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Invalidate replication slots that have remained idle longer than this
+        duration. If this value is specified without units, it is taken as
+        minutes. A value of zero disables the idle timeout
invalidation mechanism.
+        The default is one day. This parameter can only be set in the
+        <filename>postgresql.conf</filename> file or on the server
command line.
+       </para>

4) We can try to keep it to less than 80 char wherever possible:
a) Like in this case, "mechanism" can be moved to the next line:
+        duration. If this value is specified without units, it is taken as
+        minutes. A value of zero disables the idle timeout
invalidation mechanism.

b) Similarly here too, "slot's" can be moved to the next line:
+        inactive slots. The duration of slot inactivity is calculated
using the slot's
+        <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>

5) You can use new ereport style to exclude brackets around errcode:
+               ereport(ERROR,
+
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+                                errmsg("can no longer get changes
from replication slot \"%s\"",
+                                               NameStr(s->data.name)),
+                                errdetail("This slot has been
invalidated because it has remained idle longer than the configured
\"%s\" duration.",
+
"idle_replication_slot_timeout")));

Regards,
Vignesh

  347. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-01-29T07:01:04Z

    On Tue, Jan 28, 2025 at 10:58 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> Please find the attached v64 patches. The changes in this version
> w.r.t. older patch v63 are as -
> - The changes from the v63-0001 patch have been moved to a separate thread [1].
> - The v63-0002 patch has been split into two parts in v64:
>   1) 001 patch: Implements the main feature - inactive timeout-based
> slot invalidation.
>   2) 002 patch: Separates the TAP test "044_invalidate_inactive_slots"
> as suggested above.
>

Hi Nisha.

Some review comments for patch v64-0001.

======
1. General

Too much of this patch v64-0001 is identical/duplicated code with the
recent "spin-off" patch v1-0002 [1]. e.g. Most of v1-0001 is now also
embedded in the v64-0001.

This is making for an unnecessarily tricky 2 x review of all the same
code, and it will also cause rebase hassles later.

Even if you wanted the 'error_in_invalid' stuff to be discussed and
pushed separately, I think it will be much easier to keep a "COPY" of
that v1-0002 patch here as a pre-requisite for v64-0001 so then all of
the current code duplications can be removed.

======
src/backend/replication/slot.c

ReplicationSlotAcquire:

2.
+ *
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
  */

and

+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */
+ if (error_if_invalid && s->data.invalidated == RS_INVAL_IDLE_TIMEOUT)
+ {

Although those comments are correct for v1-0001 [1] it is a misleading
comment in the hacked into v64-0001 because here you are only checking
invalidation cause RS_INVAL_IDLE_TIMEOUT but none of the other
possible causes.

~~~

ReportSlotInvalidation:

3.
+ case RS_INVAL_IDLE_TIMEOUT:
+ Assert(inactive_since > 0);
+ /* translator: second %s is a GUC variable name */
+ appendStringInfo(&err_detail,
+ _("The slot has remained idle since %s, which is longer than the
configured \"%s\" duration."),
+ timestamptz_to_str(inactive_since),
+ "idle_replication_slot_timeout");

I have the same question already asked for my review of patch v1-0002
[1]. e.g. Isn't there some mismatch between using the _() macro which
is for translations, and using the errdetail_internal which is for
strings *not* requiring translation?

~~~

InvalidatePossiblyObsoleteSlot:

4.
/*
 * The logical replication slots shouldn't be invalidated as GUC
 * max_slot_wal_keep_size is set to -1 during the binary upgrade. See
 * check_old_cluster_for_valid_slots() where we ensure that no
 * invalidated before the upgrade.
 */
Assert(!(*invalidated && SlotIsLogical(s) && IsBinaryUpgrade));

Unless I am mistaken, all of the v63 cleanups of the above binary
upgrade code assert stuff have vanished somewhere between v63 and v64.
I cannot find them in the spin-off thread. All accidentally lost? (in
2 places)

Not only that but the accompanying comment modification (to mention
"and idle_replication_slot_timeout is set to 0") is also MIA last seen
in v63 (??)

======
[1] https://www.postgresql.org/message-id/CABdArM6pBL5hPnSQ%2B5nEVMANcF4FCH7LQmgskXyiLY75TMnKpw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  348. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-01-29T07:14:26Z

    On Tue, 28 Jan 2025 at 17:28, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Please find the attached v64 patches. The changes in this version
> w.r.t. older patch v63 are as -
> - The changes from the v63-0001 patch have been moved to a separate thread [1].
> - The v63-0002 patch has been split into two parts in v64:
>   1) 001 patch: Implements the main feature - inactive timeout-based
> slot invalidation.
>   2) 002 patch: Separates the TAP test "044_invalidate_inactive_slots"
> as suggested above.

Currently the test takes around 220 seconds for me. We could do the
following changes to bring it down to around 70 to 80 seconds:
1) Set idle_replication_slot_timeout to 70 seconds
+# Avoid unpredictability
+$primary->append_conf(
+       'postgresql.conf', qq{
+checkpoint_timeout = 1h
+});
+$primary->start;

2) I felt just 1 second more is enough unless you anticipate a random
failure, the test passes for me:
+# Give enough time for inactive_since to exceed the timeout
+sleep($idle_timeout_1min * 60 + 10);

3) Since we will be setting it to 70 seconds above, changing the
configuration and reload is not required:
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+       'postgres', qq[
+    ALTER SYSTEM SET idle_replication_slot_timeout TO
'${idle_timeout_1min}min';
+]);
+$primary->reload;

4) Here you can add some comments that 60s has elapsed and the slot
will get invalidated in another 10 seconds, and pass timeout as 10s to
wait_for_slot_invalidation:
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# idle timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+       $idle_timeout_1min);

5) We can have another streaming replication cluster setup, may be
primary2 and standby2 nodes and stop the standby2 immediately along
with the first streaming replication cluster itself:
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+       $idle_timeout_1min);

6) We can rename primary to primary or standby1 to standby to keep the
name consistent:
+# Create standby slot on the primary
+$primary->safe_psql(
+       'postgres', qq[
+    SELECT pg_create_physical_replication_slot(slot_name :=
'sb_slot1', immediately_reserve := true);
+]);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);

Regards,
Vignesh

  349. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-29T08:37:07Z

    On Wed, Jan 29, 2025 at 12:44 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 28 Jan 2025 at 17:28, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Please find the attached v64 patches. The changes in this version
> > w.r.t. older patch v63 are as -
> > - The changes from the v63-0001 patch have been moved to a separate thread [1].
> > - The v63-0002 patch has been split into two parts in v64:
> >   1) 001 patch: Implements the main feature - inactive timeout-based
> > slot invalidation.
> >   2) 002 patch: Separates the TAP test "044_invalidate_inactive_slots"
> > as suggested above.
>
> Currently the test takes around 220 seconds for me. We could do the
> following changes to bring it down to around 70 to 80 seconds:
>

Even then it is too long for a single test to be part of committed
code. So, we can temporarily reduce its time but fixing comments on
this is not a good use of time. We need to write this test in some
other way if we want to see it committed.

-- 
With Regards,
Amit Kapila.

  350. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-31T02:13:16Z

    On Tue, Jan 28, 2025 at 5:28 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Tue, Jan 28, 2025 at 3:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Dec 30, 2024 at 11:05 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > I think we are often too quick to throw out perfectly good tests.
> > > Citing that some similar GUCs don't do testing as a reason to skip
> > > them just seems to me like an example of "two wrongs don't make a
> > > right".
> > >
> > > There is a third option.
> > >
> > > Keep the tests. Because they take excessive time to run, that simply
> > > means you should run them *conditionally* based on the PG_TEST_EXTRA
> > > environment variable so they don't impact the normal BF execution. The
> > > documentation [1] says this env var is for "resource intensive" tests
> > > -- AFAIK this is exactly the scenario we find ourselves in, so is
> > > exactly what this env var was meant for.
> > >
> > > Search other *.pl tests for PG_TEST_EXTRA to see some examples.
> > >
> >
> > I don't see the long-running tests to be added under PG_TEST_EXTRA as
> > that will make it unusable after some point. Now, if multiple senior
> > members feel it is okay to add long-running tests under PG_TEST_EXTRA
> > then I am open to considering it. We can keep this test as a separate
> > patch so that the patch is being tested in CI or in manual tests
> > before commit.
> >
>
> Please find the attached v64 patches. The changes in this version
> w.r.t. older patch v63 are as -
> - The changes from the v63-0001 patch have been moved to a separate thread [1].
> - The v63-0002 patch has been split into two parts in v64:
>   1) 001 patch: Implements the main feature - inactive timeout-based
> slot invalidation.
>   2) 002 patch: Separates the TAP test "044_invalidate_inactive_slots"
> as suggested above.
>
> [1] https://www.postgresql.org/message-id/CABdArM6pBL5hPnSQ%2B5nEVMANcF4FCH7LQmgskXyiLY75TMnKpw%40mail.gmail.com
>

Please find the v65 patch set attached with following changes:

 - patch-0001 is the copy of v4-001 patch used as a base patch from
[1], as suggested by Peter in [2].
 - patch-0002 is the main patch implementing the feature, this has
also addressed the comments from [2] and [3]
 - patch-0003 adds an alternative approach for the TAP test using
injection points to force idle_timeout slot invalidation without
waiting for a minute. This test takes 2-3 seconds to complete.
 - patch-0004 maintains the previous test(v64-0002) which is under
PG_TEST_EXTRA. Also, addressed Vignesh's comments [4] for the test and
now it takes 70-80 seconds to complete.

Note: Patches 0003 and 0004 contain the same TAP test but use
different verification methods. We need to decide which one to keep.

[1] https://www.postgresql.org/message-id/CALDaNm1ZUHnKm%2BPSjjqRrMxcLagrUTS6SADnEsQBfW8rMZFrDA%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPtyUQGee6pHkNN3-ghYhWnY5p-3yWumK7zKupu0S1oVQQ%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CALDaNm1J_mdqCYjQZgfQMVhJrxndPem5ruxpG_67t4C_2My9WQ%40mail.gmail.com
[4] https://www.postgresql.org/message-id/CALDaNm2dAJB%3DfJ2X7EMb7meNTjMyL-%2B-xA93JL_jPkGF4%3DRUYw%40mail.gmail.com

Thank you, Kuroda-san for providing the TAP test using injection
points (patch-0003).

--
Thanks,
Nisha

  351. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-01-31T05:09:46Z

    Hi Nisha.

Here are some review comments for patch v65-0002

======
src/backend/replication/slot.c

ReportSlotInvalidation:

1.
+
+ case RS_INVAL_IDLE_TIMEOUT:
+ Assert(inactive_since > 0);
+ /* translator: second %s is a GUC variable name */
+ appendStringInfo(&err_detail,
+ _("The slot has remained idle since %s, which is longer than the
configured \"%s\" duration."),
+ timestamptz_to_str(inactive_since),
+ "idle_replication_slot_timeout");
+ break;
+

errdetail:

I guess it is no fault of this patch because I see you've only copied
nearby code, but AFAICT this function is still having an each-way bet
by using a mixture of _() macro which is for strings intended be
translated, but then only using them in errdetail_internal() which is
for strings that are NOT intended to be translated. Isn't it
contradictory? Why don't we use errdetail() here?

errhint:

Also, the way the 'hint' is implemented can only be meaningful for
RS_INVAL_WAL_REMOVED. This is also existing code that IMO it was
always strange, but now that this patch has added another kind of
switch (cause) this hint implementation now looks increasingly hacky
to me; it is also inflexible -- e.g. if you ever wanted to add
different hints. A neater implementation would be to make the code
more like how the err_detail is handled, so then the errhint string
would only be assigned within the "case RS_INVAL_WAL_REMOVED:"

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  352. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-31T09:02:34Z

    On Fri, Jan 31, 2025 at 10:40 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> ======
> src/backend/replication/slot.c
>
> ReportSlotInvalidation:
>
> 1.
> +
> + case RS_INVAL_IDLE_TIMEOUT:
> + Assert(inactive_since > 0);
> + /* translator: second %s is a GUC variable name */
> + appendStringInfo(&err_detail,
> + _("The slot has remained idle since %s, which is longer than the
> configured \"%s\" duration."),
> + timestamptz_to_str(inactive_since),
> + "idle_replication_slot_timeout");
> + break;
> +
>
> errdetail:
>
> I guess it is no fault of this patch because I see you've only copied
> nearby code, but AFAICT this function is still having an each-way bet
> by using a mixture of _() macro which is for strings intended be
> translated, but then only using them in errdetail_internal() which is
> for strings that are NOT intended to be translated. Isn't it
> contradictory? Why don't we use errdetail() here?
>

Your question is valid and I don't have an answer. I encourage you to
start a new thread to clarify this.

> errhint:
>
> Also, the way the 'hint' is implemented can only be meaningful for
> RS_INVAL_WAL_REMOVED. This is also existing code that IMO it was
> always strange, but now that this patch has added another kind of
> switch (cause) this hint implementation now looks increasingly hacky
> to me; it is also inflexible -- e.g. if you ever wanted to add
> different hints. A neater implementation would be to make the code
> more like how the err_detail is handled, so then the errhint string
> would only be assigned within the "case RS_INVAL_WAL_REMOVED:"
>

This makes sense to me.

+
+ case RS_INVAL_IDLE_TIMEOUT:
+ Assert(inactive_since > 0);
+ /* translator: second %s is a GUC variable name */
+ appendStringInfo(&err_detail,
+ _("The slot has remained idle since %s, which is longer than the
configured \"%s\" duration."),
+ timestamptz_to_str(inactive_since),
+ "idle_replication_slot_timeout");

I think the above message should be constructed on a model similar to
the following nearby message:"The slot's restart_lsn %X/%X exceeds the
limit by %llu bytes.". So, how about the following: "The slot's idle
time %s exceeds the configured \"%s\" duration"?

Also, similar to max_slot_wal_keep_size, we should give a hint in this
case to increase idle_replication_slot_timeout.

It is not clear why the injection point test is doing
pg_sync_replication_slots() etc. in the patch. The test should be
simple such that after creating a new physical or logical slot, enable
the injection point, then run the manual checkpoint command, and check
the invalidation status of the slot.

-- 
With Regards,
Amit Kapila.

  353. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-01-31T12:20:26Z

    On Fri, Jan 31, 2025 at 2:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jan 31, 2025 at 10:40 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > ======
> > src/backend/replication/slot.c
> >
> > ReportSlotInvalidation:
> >
> > 1.
> > +
> > + case RS_INVAL_IDLE_TIMEOUT:
> > + Assert(inactive_since > 0);
> > + /* translator: second %s is a GUC variable name */
> > + appendStringInfo(&err_detail,
> > + _("The slot has remained idle since %s, which is longer than the
> > configured \"%s\" duration."),
> > + timestamptz_to_str(inactive_since),
> > + "idle_replication_slot_timeout");
> > + break;
> > +
> >
> > errdetail:
> >
> > I guess it is no fault of this patch because I see you've only copied
> > nearby code, but AFAICT this function is still having an each-way bet
> > by using a mixture of _() macro which is for strings intended be
> > translated, but then only using them in errdetail_internal() which is
> > for strings that are NOT intended to be translated. Isn't it
> > contradictory? Why don't we use errdetail() here?
> >
>
> Your question is valid and I don't have an answer. I encourage you to
> start a new thread to clarify this.
>
> > errhint:
> >
> > Also, the way the 'hint' is implemented can only be meaningful for
> > RS_INVAL_WAL_REMOVED. This is also existing code that IMO it was
> > always strange, but now that this patch has added another kind of
> > switch (cause) this hint implementation now looks increasingly hacky
> > to me; it is also inflexible -- e.g. if you ever wanted to add
> > different hints. A neater implementation would be to make the code
> > more like how the err_detail is handled, so then the errhint string
> > would only be assigned within the "case RS_INVAL_WAL_REMOVED:"
> >
>
> This makes sense to me.
>
> +
> + case RS_INVAL_IDLE_TIMEOUT:
> + Assert(inactive_since > 0);
> + /* translator: second %s is a GUC variable name */
> + appendStringInfo(&err_detail,
> + _("The slot has remained idle since %s, which is longer than the
> configured \"%s\" duration."),
> + timestamptz_to_str(inactive_since),
> + "idle_replication_slot_timeout");
>
> I think the above message should be constructed on a model similar to
> the following nearby message:"The slot's restart_lsn %X/%X exceeds the
> limit by %llu bytes.". So, how about the following: "The slot's idle
> time %s exceeds the configured \"%s\" duration"?
>
> Also, similar to max_slot_wal_keep_size, we should give a hint in this
> case to increase idle_replication_slot_timeout.
>
> It is not clear why the injection point test is doing
> pg_sync_replication_slots() etc. in the patch. The test should be
> simple such that after creating a new physical or logical slot, enable
> the injection point, then run the manual checkpoint command, and check
> the invalidation status of the slot.
>

Thanks for the review! I have incorporated the above comments. The
test in patch-002 has been optimized as suggested and now completes in
less than a second.
Please find the attached v66 patch set. The base patch(v65-001) is
committed now, so I have rebased the patches.

Thank you, Kuroda-san, for working on patch-002.

--
Thanks,
Nisha

  354. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-02-01T06:11:53Z

    On Fri, 31 Jan 2025 at 17:50, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Thanks for the review! I have incorporated the above comments. The
> test in patch-002 has been optimized as suggested and now completes in
> less than a second.
> Please find the attached v66 patch set. The base patch(v65-001) is
> committed now, so I have rebased the patches.

Few comments:
1)We should set inactive_since only if the slot can be invalidated:
+                                       /* For testing timeout slot
invalidation */
+                                       if
(IS_INJECTION_POINT_ATTACHED("slot-time-out-inval"))
+                                               s->inactive_since = 1;
+

2) Instead of "alter system set" and reload, let's do this in
$node->append_conf itself:
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$node->safe_psql(
+       'postgres', qq[
+    ALTER SYSTEM SET idle_replication_slot_timeout TO '1min';
+]);
+$node->reload;

3) No need to trigger checkpoint twice, we can move it outside so that
just a single checkpoint will invalidate both the slots:
+sub trigger_slot_invalidation
+{
+       my ($node, $slot, $offset) = @_;
+       my $node_name = $node->name;
+       my $invalidated = 0;
+
+       # Run a checkpoint
+       $node->safe_psql('postgres', "CHECKPOINT");

4) I fel this trigger_slot_invalidation is not required after removing
the checkpoint from the function, let's move the waiting for
"invalidating obsolete replication slot" also to
wait_for_slot_invalidation function:
+       # The slot's invalidation should be logged
+       $node->wait_for_log(qr/invalidating obsolete replication slot
\"$slot\"/,
+               $offset);
+
+       # Check that the invalidation reason is 'idle_timeout'
+       $node->poll_query_until(
+               'postgres', qq[
+               SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+                       WHERE slot_name = '$slot' AND
+                       invalidation_reason = 'idle_timeout';
+       ])

5) Can we move the subroutine to the beginning, I noticed in other
places we have kept it before the tests like in 027_nosuperuser and
040_createsubscriber:
+# Wait for slot to first become idle and then get invalidated
+sub wait_for_slot_invalidation
+{
+       my ($node, $slot, $offset) = @_;
+       my $node_name = $node->name;
+
+       trigger_slot_invalidation($node, $slot, $offset);
+
+       # Check that an invalidated slot cannot be acquired
+       my ($result, $stdout, $stderr);
+       ($result, $stdout, $stderr) = $node->psql(
+               'postgres', qq[
+                       SELECT pg_replication_slot_advance('$slot', '0/1');
+       ]);

6) Since idle_replication_slot_timeout is related more closely with
max_slot_wal_keep_size, let's keep it along with it.
diff --git a/src/backend/utils/misc/postgresql.conf.sample
b/src/backend/utils/misc/postgresql.conf.sample
index 079efa1baa..0ed9eb057e 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -329,6 +329,7 @@
 #wal_sender_timeout = 60s      # in milliseconds; 0 disables
 #track_commit_timestamp = off  # collect timestamp of transaction commit
                                # (change requires restart)
+#idle_replication_slot_timeout = 1d    # in minutes; 0 disables

If you accept the comments, you can merge the changes from the attached patch.

Regards,
Vignesh

  355. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-03T00:46:17Z

    Hi Nisha,

Some review comments for v66-0001.

======
src/backend/replication/slot.c

ReportSlotInvalidation:

1.
  StringInfoData err_detail;
+ StringInfoData err_hint;
  bool hint = false;

  initStringInfo(&err_detail);
+ initStringInfo(&err_hint);


I don't think you still need the 'hint' boolean anymore.

Instead of:
hint ? errhint("%s", err_hint.data) : 0);

You could just do something like:
err_hint.len ? errhint("%s", err_hint.data) : 0);

~~~

2.
+ appendStringInfo(&err_hint, "You might need to increase \"%s\".",
+ "max_slot_wal_keep_size");
  break;
2a.
In this case, shouldn't you really be using macro _("You might need to
increase \"%s\".") so that the common format string would be got using
gettext()?

~

2b.
Should you include a /* translator */ comment here? Other places where
GUC name is substituted do this.

~~~

3.
+ appendStringInfo(&err_hint, "You might need to increase \"%s\".",
+ "idle_replication_slot_timeout");
+ break;

3a.
Ditto above. IMO this common format string should be got using macro.
e.g.: _("You might need to increase \"%s\".")

~

3b.
Should you include a /* translator */ comment here? Other places where
GUC name is substituted do this.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  356. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-03T03:33:47Z

    On Fri, Jan 31, 2025 at 8:02 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jan 31, 2025 at 10:40 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > ======
> > src/backend/replication/slot.c
> >
> > ReportSlotInvalidation:
> >
> > 1.
> > +
> > + case RS_INVAL_IDLE_TIMEOUT:
> > + Assert(inactive_since > 0);
> > + /* translator: second %s is a GUC variable name */
> > + appendStringInfo(&err_detail,
> > + _("The slot has remained idle since %s, which is longer than the
> > configured \"%s\" duration."),
> > + timestamptz_to_str(inactive_since),
> > + "idle_replication_slot_timeout");
> > + break;
> > +
> >
> > errdetail:
> >
> > I guess it is no fault of this patch because I see you've only copied
> > nearby code, but AFAICT this function is still having an each-way bet
> > by using a mixture of _() macro which is for strings intended be
> > translated, but then only using them in errdetail_internal() which is
> > for strings that are NOT intended to be translated. Isn't it
> > contradictory? Why don't we use errdetail() here?
> >
>
> Your question is valid and I don't have an answer. I encourage you to
> start a new thread to clarify this.
>

I think this was a false alarm.

After studying this more deeply, I've changed my mind and now think
the code is OK as-is.

AFAICT errdetail_internal is used when not wanting to translate the
*fmt* string passed to it (see EVALUATE_MESSAGE in elog.c). Now, here
the format string is just "%s" so it's fine to not translate that.
Meanwhile, the string value being substituted to the "%s" was already
translated because of the _(x) macro aka gettext(x).

I found other examples similar to this -- see the
error_view_not_updatable() function in rewriteHandler.c which does:
ereport(ERROR,
...
 detail ? errdetail_internal("%s", _(detail)) : 0,
...

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  357. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-03T05:04:35Z

    On Mon, Feb 3, 2025 at 9:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Jan 31, 2025 at 8:02 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Jan 31, 2025 at 10:40 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > ======
> > > src/backend/replication/slot.c
> > >
> > > ReportSlotInvalidation:
> > >
> > > 1.
> > > +
> > > + case RS_INVAL_IDLE_TIMEOUT:
> > > + Assert(inactive_since > 0);
> > > + /* translator: second %s is a GUC variable name */
> > > + appendStringInfo(&err_detail,
> > > + _("The slot has remained idle since %s, which is longer than the
> > > configured \"%s\" duration."),
> > > + timestamptz_to_str(inactive_since),
> > > + "idle_replication_slot_timeout");
> > > + break;
> > > +
> > >
> > > errdetail:
> > >
> > > I guess it is no fault of this patch because I see you've only copied
> > > nearby code, but AFAICT this function is still having an each-way bet
> > > by using a mixture of _() macro which is for strings intended be
> > > translated, but then only using them in errdetail_internal() which is
> > > for strings that are NOT intended to be translated. Isn't it
> > > contradictory? Why don't we use errdetail() here?
> > >
> >
> > Your question is valid and I don't have an answer. I encourage you to
> > start a new thread to clarify this.
> >
>
> I think this was a false alarm.
>
> After studying this more deeply, I've changed my mind and now think
> the code is OK as-is.
>
> AFAICT errdetail_internal is used when not wanting to translate the
> *fmt* string passed to it (see EVALUATE_MESSAGE in elog.c). Now, here
> the format string is just "%s" so it's fine to not translate that.
> Meanwhile, the string value being substituted to the "%s" was already
> translated because of the _(x) macro aka gettext(x).
>

I didn't get your point about " the "%s" was already translated
because of ...". If we don't want to translate the message then why
add '_(' to it in the first place?

-- 
With Regards,
Amit Kapila.

  358. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-03T05:50:56Z

    On Mon, Feb 3, 2025 at 4:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Feb 3, 2025 at 9:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Fri, Jan 31, 2025 at 8:02 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Fri, Jan 31, 2025 at 10:40 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > ======
> > > > src/backend/replication/slot.c
> > > >
> > > > ReportSlotInvalidation:
> > > >
> > > > 1.
> > > > +
> > > > + case RS_INVAL_IDLE_TIMEOUT:
> > > > + Assert(inactive_since > 0);
> > > > + /* translator: second %s is a GUC variable name */
> > > > + appendStringInfo(&err_detail,
> > > > + _("The slot has remained idle since %s, which is longer than the
> > > > configured \"%s\" duration."),
> > > > + timestamptz_to_str(inactive_since),
> > > > + "idle_replication_slot_timeout");
> > > > + break;
> > > > +
> > > >
> > > > errdetail:
> > > >
> > > > I guess it is no fault of this patch because I see you've only copied
> > > > nearby code, but AFAICT this function is still having an each-way bet
> > > > by using a mixture of _() macro which is for strings intended be
> > > > translated, but then only using them in errdetail_internal() which is
> > > > for strings that are NOT intended to be translated. Isn't it
> > > > contradictory? Why don't we use errdetail() here?
> > > >
> > >
> > > Your question is valid and I don't have an answer. I encourage you to
> > > start a new thread to clarify this.
> > >
> >
> > I think this was a false alarm.
> >
> > After studying this more deeply, I've changed my mind and now think
> > the code is OK as-is.
> >
> > AFAICT errdetail_internal is used when not wanting to translate the
> > *fmt* string passed to it (see EVALUATE_MESSAGE in elog.c). Now, here
> > the format string is just "%s" so it's fine to not translate that.
> > Meanwhile, the string value being substituted to the "%s" was already
> > translated because of the _(x) macro aka gettext(x).
> >
>
> I didn't get your point about " the "%s" was already translated
> because of ...". If we don't want to translate the message then why
> add '_(' to it in the first place?
>

I think this is same point where I was fooling myself yesterday.  In
fact we do want to translate the message seen by the user.

errdetail_internal really means don't translate the ***format
string***. In our case "%s" is not the message at all -- it is just
the a *format string* so translating "%s" is kind of meaningless.

e.g. Normally....

errdetail("translate me") <-- This would translate the fmt string but
here the fmt is also the message; i.e. it will do gettext("translate
me") internally.

errdetail_internal("translate me") <-- This won't translate anything;
you will have the raw fmt string "translate me"

~~

But since ReportSlotInvalidation is building the message on the fly
there is no single report so it is a bit different....

errdetail("%s", "translate me") <-- this would just use gettext("%s")
which is kind of useless. And the "translate me" is just a raw string
and won't be translated.

errdetail_internal("%s", "translate me") <-- this won't translate
anything; the fmt string and the "translate me" are just raw strings

errdetail_internal("%s", _("translate me"))  <-- This won't translate
the fmt string, but to translate %s is useless anyway. OTOH, the _()
macro means it will do gettext("translate me") so the "translate me"
string will get translated before it is substituted. This is
effectively what the ReportSlotInvalidation code is doing.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  359. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-03T06:34:36Z

    On Mon, Feb 3, 2025 at 6:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
>
> 2.
> + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> + "max_slot_wal_keep_size");
>   break;
> 2a.
> In this case, shouldn't you really be using macro _("You might need to
> increase \"%s\".") so that the common format string would be got using
> gettext()?
>
> ~
>
>
> ~~~
>
> 3.
> + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> + "idle_replication_slot_timeout");
> + break;
>
> 3a.
> Ditto above. IMO this common format string should be got using macro.
> e.g.: _("You might need to increase \"%s\".")
>
> ~

Instead, we can directly use '_(' in errhint as we are doing in one
other similar place "errhint("%s", _(view_updatable_error))));". I
think we didn't use it for errdetail because, in one of the cases, it
needs to use ngettext

-- 
With Regards,
Amit Kapila.

  360. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-03T09:25:05Z

    On Fri, Jan 31, 2025 at 5:50 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Please find the attached v66 patch set. The base patch(v65-001) is
> committed now, so I have rebased the patches.
>

*
       <para>
         The time when the slot became inactive. <literal>NULL</literal> if the
-        slot is currently being streamed.
+        slot is currently being streamed. If the slot becomes invalidated,
+        this value will remain unchanged until server shutdown.
...
@@ -2408,7 +2527,9 @@ RestoreSlotFromDisk(const char *name)
  /*
  * Set the time since the slot has become inactive after loading the
  * slot from the disk into memory. Whoever acquires the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will not set the time
+ * for invalid slots.
  */
  slot->inactive_since = GetCurrentTimestamp();

It looks inconsistent to set inactive_since on restart for invalid
slots but not at other times. We don't need to set inactive_since for
invalid slots. The invalid slots should not be updated. Ideally, this
should be taken care in the patch that introduces inactive_since but
we can do that now. Let's do this as a separate patch altogether in a
new thread.

-- 
With Regards,
Amit Kapila.

  361. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-03T12:05:22Z

    On Mon, Feb 3, 2025 at 2:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jan 31, 2025 at 5:50 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Please find the attached v66 patch set. The base patch(v65-001) is
> > committed now, so I have rebased the patches.
> >
>
> *
>        <para>
>          The time when the slot became inactive. <literal>NULL</literal> if the
> -        slot is currently being streamed.
> +        slot is currently being streamed. If the slot becomes invalidated,
> +        this value will remain unchanged until server shutdown.
> ...
> @@ -2408,7 +2527,9 @@ RestoreSlotFromDisk(const char *name)
>   /*
>   * Set the time since the slot has become inactive after loading the
>   * slot from the disk into memory. Whoever acquires the slot i.e.
> - * makes the slot active will reset it.
> + * makes the slot active will reset it. Avoid calling
> + * ReplicationSlotSetInactiveSince() here, as it will not set the time
> + * for invalid slots.
>   */
>   slot->inactive_since = GetCurrentTimestamp();
>
> It looks inconsistent to set inactive_since on restart for invalid
> slots but not at other times. We don't need to set inactive_since for
> invalid slots. The invalid slots should not be updated. Ideally, this
> should be taken care in the patch that introduces inactive_since but
> we can do that now. Let's do this as a separate patch altogether in a
> new thread.

Created a new thread [1] to address the inactive_since update for
invalid slots in a separate patch.

[1] https://www.postgresql.org/message-id/CABdArM7QdifQ_MHmMA%3DCc4v8%2BMeckkwKncm2Nn6tX9wSCQ-%2Biw%40mail.gmail.com

--
Thanks,
Nisha

  362. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2025-02-03T13:05:43Z

    On Fri, 31 Jan 2025 at 17:50, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Fri, Jan 31, 2025 at 2:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Jan 31, 2025 at 10:40 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > ======
> > > src/backend/replication/slot.c
> > >
> > > ReportSlotInvalidation:
> > >
> > > 1.
> > > +
> > > + case RS_INVAL_IDLE_TIMEOUT:
> > > + Assert(inactive_since > 0);
> > > + /* translator: second %s is a GUC variable name */
> > > + appendStringInfo(&err_detail,
> > > + _("The slot has remained idle since %s, which is longer than the
> > > configured \"%s\" duration."),
> > > + timestamptz_to_str(inactive_since),
> > > + "idle_replication_slot_timeout");
> > > + break;
> > > +
> > >
> > > errdetail:
> > >
> > > I guess it is no fault of this patch because I see you've only copied
> > > nearby code, but AFAICT this function is still having an each-way bet
> > > by using a mixture of _() macro which is for strings intended be
> > > translated, but then only using them in errdetail_internal() which is
> > > for strings that are NOT intended to be translated. Isn't it
> > > contradictory? Why don't we use errdetail() here?
> > >
> >
> > Your question is valid and I don't have an answer. I encourage you to
> > start a new thread to clarify this.
> >
> > > errhint:
> > >
> > > Also, the way the 'hint' is implemented can only be meaningful for
> > > RS_INVAL_WAL_REMOVED. This is also existing code that IMO it was
> > > always strange, but now that this patch has added another kind of
> > > switch (cause) this hint implementation now looks increasingly hacky
> > > to me; it is also inflexible -- e.g. if you ever wanted to add
> > > different hints. A neater implementation would be to make the code
> > > more like how the err_detail is handled, so then the errhint string
> > > would only be assigned within the "case RS_INVAL_WAL_REMOVED:"
> > >
> >
> > This makes sense to me.
> >
> > +
> > + case RS_INVAL_IDLE_TIMEOUT:
> > + Assert(inactive_since > 0);
> > + /* translator: second %s is a GUC variable name */
> > + appendStringInfo(&err_detail,
> > + _("The slot has remained idle since %s, which is longer than the
> > configured \"%s\" duration."),
> > + timestamptz_to_str(inactive_since),
> > + "idle_replication_slot_timeout");
> >
> > I think the above message should be constructed on a model similar to
> > the following nearby message:"The slot's restart_lsn %X/%X exceeds the
> > limit by %llu bytes.". So, how about the following: "The slot's idle
> > time %s exceeds the configured \"%s\" duration"?
> >
> > Also, similar to max_slot_wal_keep_size, we should give a hint in this
> > case to increase idle_replication_slot_timeout.
> >
> > It is not clear why the injection point test is doing
> > pg_sync_replication_slots() etc. in the patch. The test should be
> > simple such that after creating a new physical or logical slot, enable
> > the injection point, then run the manual checkpoint command, and check
> > the invalidation status of the slot.
> >
>
> Thanks for the review! I have incorporated the above comments. The
> test in patch-002 has been optimized as suggested and now completes in
> less than a second.
> Please find the attached v66 patch set. The base patch(v65-001) is
> committed now, so I have rebased the patches.
>
> Thank you, Kuroda-san, for working on patch-002.
>

Hi Nisha,

I reviewed the v66 patch. I have few comments:

1. I also feel the default value should be set to '0' as suggested by
Vignesh in 1st point of [1].

2. Should we allow copying of invalidated slots?
Currently we are able to copy slots which are invalidated:

postgres=# select slot_name, active, restart_lsn, wal_status,
inactive_since , invalidation_reason from pg_replication_slots;
 slot_name | active | restart_lsn | wal_status |
inactive_since          | invalidation_reason
-----------+--------+-------------+------------+----------------------------------+---------------------
 test1     | f      | 0/16FDDE0   | lost       | 2025-02-03
18:28:01.802463+05:30 | idle_timeout
(1 row)

postgres=# select pg_copy_logical_replication_slot('test1', 'test2');
 pg_copy_logical_replication_slot
----------------------------------
 (test2,0/16FDE18)
(1 row)

postgres=# select slot_name, active, restart_lsn, wal_status,
inactive_since , invalidation_reason from pg_replication_slots;
 slot_name | active | restart_lsn | wal_status |
inactive_since          | invalidation_reason
-----------+--------+-------------+------------+----------------------------------+---------------------
 test1     | f      | 0/16FDDE0   | lost       | 2025-02-03
18:28:01.802463+05:30 | idle_timeout
 test2     | f      | 0/16FDDE0   | reserved   | 2025-02-03
18:29:53.478023+05:30 |
(2 rows)

3. We have similar behaviour as above for physical slots.

[1]: https://www.postgresql.org/message-id/CALDaNm14QrW5j6su%2BEAqjwnHbiwXJwO%2Byk73_%3D7yvc5TVY-43g%40mail.gmail.com


Thanks and Regards,
Shlok Kyal

  363. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-03T14:33:57Z

    On Mon, Feb 3, 2025 at 12:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Feb 3, 2025 at 6:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> >
> > 2.
> > + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> > + "max_slot_wal_keep_size");
> >   break;
> > 2a.
> > In this case, shouldn't you really be using macro _("You might need to
> > increase \"%s\".") so that the common format string would be got using
> > gettext()?
> >
> > ~
> >
> >
> > ~~~
> >
> > 3.
> > + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> > + "idle_replication_slot_timeout");
> > + break;
> >
> > 3a.
> > Ditto above. IMO this common format string should be got using macro.
> > e.g.: _("You might need to increase \"%s\".")
> >
> > ~
>
> Instead, we can directly use '_(' in errhint as we are doing in one
> other similar place "errhint("%s", _(view_updatable_error))));". I
> think we didn't use it for errdetail because, in one of the cases, it
> needs to use ngettext
>

Please find the v67 patches:
 - The patches have been rebased after separating the inactive_since
related changes.
 - Patches 001 and 002 incorporate the above comments and the comments
from [1] and [2].
 - No change in patch-003 since the last version.


[1] https://www.postgresql.org/message-id/CALDaNm0FS%2BFqQk2dadiJFCMM_MhKROMsJUb%3Db8wtRH6isScQsQ%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPs_6%2BNBOt%2BKpQQaBG2R3T-FLS93TbUC27uzyDMu%3D37n-Q%40mail.gmail.com

--
Thanks,
Nisha

  364. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-03T22:32:17Z

    On Mon, Feb 3, 2025 at 5:34 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Feb 3, 2025 at 6:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> >
> > 2.
> > + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> > + "max_slot_wal_keep_size");
> >   break;
> > 2a.
> > In this case, shouldn't you really be using macro _("You might need to
> > increase \"%s\".") so that the common format string would be got using
> > gettext()?
> >
> > ~
> >
> >
> > ~~~
> >
> > 3.
> > + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> > + "idle_replication_slot_timeout");
> > + break;
> >
> > 3a.
> > Ditto above. IMO this common format string should be got using macro.
> > e.g.: _("You might need to increase \"%s\".")
> >
> > ~
>
> Instead, we can directly use '_(' in errhint as we are doing in one
> other similar place "errhint("%s", _(view_updatable_error))));". I
> think we didn't use it for errdetail because, in one of the cases, it
> needs to use ngettext
>

-1 for this suggestion because this will end up causing a gettext() on
the entire hint where the GUC has already been substituted.

e.g. it is effectively doing

_("You might need to increase \"max_slot_wal_keep_size\".")
_("You might need to increase \"idle_replication_slot_timeout\".")

But that is contrary to the goal of reducing the burden on translators
by using *common* messages wherever possible. IMO we should only
request translation of the *common* part of the hint message.

e.g.
_("You might need to increase \"%s\".")

~~~

We always do GUC name substitution into a *common* fmt message because
then translators only need to maintain a single translated string
instead of many. You can find examples of this everywhere. For
example, notice the GUC is always substituted into the translated fmt
msgs below; they never have the GUC name included explicitly. The
result is just a single fmt message is needed.

$ grep -r . -e 'errhint("You might need to increase' | grep '.c:'
./src/backend/replication/logical/launcher.c: errhint("You might need
to increase \"%s\".", "max_logical_replication_workers")));
./src/backend/replication/logical/launcher.c: errhint("You might need
to increase \"%s\".", "max_worker_processes")));
./src/backend/storage/lmgr/predicate.c: errhint("You might need to
increase \"%s\".", "max_pred_locks_per_transaction")));
./src/backend/storage/lmgr/predicate.c: errhint("You might need to
increase \"%s\".", "max_pred_locks_per_transaction")));
./src/backend/storage/lmgr/predicate.c: errhint("You might need to
increase \"%s\".", "max_pred_locks_per_transaction")));
./src/backend/storage/lmgr/lock.c: errhint("You might need to increase
\"%s\".", "max_locks_per_transaction")));
./src/backend/storage/lmgr/lock.c: errhint("You might need to increase
\"%s\".", "max_locks_per_transaction")));
./src/backend/storage/lmgr/lock.c: errhint("You might need to increase
\"%s\".", "max_locks_per_transaction")));
./src/backend/storage/lmgr/lock.c: errhint("You might need to increase
\"%s\".", "max_locks_per_transaction")));
./src/backend/storage/lmgr/lock.c: errhint("You might need to increase
\"%s\".", "max_locks_per_transaction")));

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  365. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-03T23:38:44Z

    Hi Nisha.

Some review comments for patch v67-0001.

======
src/backend/replication/slot.c

ReportSlotInvalidation:

Please see my previous post [1] where I gave some reasons why I think
the _() macro should be used only for the *common* part of the hint
messages. If you agree, then the following code should be changed.

1.
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_hint, "You might need to increase \"%s\".",
+ "max_slot_wal_keep_size");
  break;

Change to:
_("You might need to increase \"%s\".")

~

2.
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_hint, "You might need to increase \"%s\".",
+ "idle_replication_slot_timeout");
+ break;

Change to:
_("You might need to increase \"%s\".")

~

3.
- hint ? errhint("You might need to increase \"%s\".",
"max_slot_wal_keep_size") : 0);
+ err_hint.len ? errhint("%s", _(err_hint.data)) : 0);

Change to:
err_hint.len ? errhint("%s", err_hint.data) : 0);

======
[1] https://www.postgresql.org/message-id/CAHut%2BPuKCv-S%2BPJ2iybZKiqu0GJ1fSuzy2CcvyRViLou98QpVA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  366. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-04T05:08:39Z

    On Tue, Feb 4, 2025 at 4:02 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Feb 3, 2025 at 5:34 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Feb 3, 2025 at 6:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > >
> > > 2.
> > > + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> > > + "max_slot_wal_keep_size");
> > >   break;
> > > 2a.
> > > In this case, shouldn't you really be using macro _("You might need to
> > > increase \"%s\".") so that the common format string would be got using
> > > gettext()?
> > >
> > > ~
> > >
> > >
> > > ~~~
> > >
> > > 3.
> > > + appendStringInfo(&err_hint, "You might need to increase \"%s\".",
> > > + "idle_replication_slot_timeout");
> > > + break;
> > >
> > > 3a.
> > > Ditto above. IMO this common format string should be got using macro.
> > > e.g.: _("You might need to increase \"%s\".")
> > >
> > > ~
> >
> > Instead, we can directly use '_(' in errhint as we are doing in one
> > other similar place "errhint("%s", _(view_updatable_error))));". I
> > think we didn't use it for errdetail because, in one of the cases, it
> > needs to use ngettext
> >
>
> -1 for this suggestion because this will end up causing a gettext() on
> the entire hint where the GUC has already been substituted.
>
> e.g. it is effectively doing
>
> _("You might need to increase \"max_slot_wal_keep_size\".")
> _("You might need to increase \"idle_replication_slot_timeout\".")
>
> But that is contrary to the goal of reducing the burden on translators
> by using *common* messages wherever possible. IMO we should only
> request translation of the *common* part of the hint message.
>

Fair point. So, we can ignore my suggestion.

-- 
With Regards,
Amit Kapila.

  367. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-04T05:15:09Z

    On Mon, Feb 3, 2025 at 6:35 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> I reviewed the v66 patch. I have few comments:
>
> 1. I also feel the default value should be set to '0' as suggested by
> Vignesh in 1st point of [1].
>

+1. This will ensure that the idle slots won't be invalidated by
default, the same as HEAD. We can change the default value based on
user inputs.

> 2. Should we allow copying of invalidated slots?
> Currently we are able to copy slots which are invalidated:
>
> postgres=# select slot_name, active, restart_lsn, wal_status,
> inactive_since , invalidation_reason from pg_replication_slots;
>  slot_name | active | restart_lsn | wal_status |
> inactive_since          | invalidation_reason
> -----------+--------+-------------+------------+----------------------------------+---------------------
>  test1     | f      | 0/16FDDE0   | lost       | 2025-02-03
> 18:28:01.802463+05:30 | idle_timeout
> (1 row)
>
> postgres=# select pg_copy_logical_replication_slot('test1', 'test2');
>  pg_copy_logical_replication_slot
> ----------------------------------
>  (test2,0/16FDE18)
> (1 row)
>
> postgres=# select slot_name, active, restart_lsn, wal_status,
> inactive_since , invalidation_reason from pg_replication_slots;
>  slot_name | active | restart_lsn | wal_status |
> inactive_since          | invalidation_reason
> -----------+--------+-------------+------------+----------------------------------+---------------------
>  test1     | f      | 0/16FDDE0   | lost       | 2025-02-03
> 18:28:01.802463+05:30 | idle_timeout
>  test2     | f      | 0/16FDDE0   | reserved   | 2025-02-03
> 18:29:53.478023+05:30 |
> (2 rows)
>

Is this related to this patch or the behavior of HEAD? If this
behavior is not introduced by this patch then we should discuss this
in a separate thread. I couldn't think of why anyone wants to copy the
invalid slots, so we should probably prohibit copying invalid slots
but that is a matter of separate discussion unless introduced by this
patch.

-- 
With Regards,
Amit Kapila.

  368. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2025-02-04T09:58:05Z

    On Tue, 4 Feb 2025 at 10:45, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Feb 3, 2025 at 6:35 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > I reviewed the v66 patch. I have few comments:
> >
> > 1. I also feel the default value should be set to '0' as suggested by
> > Vignesh in 1st point of [1].
> >
>
> +1. This will ensure that the idle slots won't be invalidated by
> default, the same as HEAD. We can change the default value based on
> user inputs.
>
> > 2. Should we allow copying of invalidated slots?
> > Currently we are able to copy slots which are invalidated:
> >
> > postgres=# select slot_name, active, restart_lsn, wal_status,
> > inactive_since , invalidation_reason from pg_replication_slots;
> >  slot_name | active | restart_lsn | wal_status |
> > inactive_since          | invalidation_reason
> > -----------+--------+-------------+------------+----------------------------------+---------------------
> >  test1     | f      | 0/16FDDE0   | lost       | 2025-02-03
> > 18:28:01.802463+05:30 | idle_timeout
> > (1 row)
> >
> > postgres=# select pg_copy_logical_replication_slot('test1', 'test2');
> >  pg_copy_logical_replication_slot
> > ----------------------------------
> >  (test2,0/16FDE18)
> > (1 row)
> >
> > postgres=# select slot_name, active, restart_lsn, wal_status,
> > inactive_since , invalidation_reason from pg_replication_slots;
> >  slot_name | active | restart_lsn | wal_status |
> > inactive_since          | invalidation_reason
> > -----------+--------+-------------+------------+----------------------------------+---------------------
> >  test1     | f      | 0/16FDDE0   | lost       | 2025-02-03
> > 18:28:01.802463+05:30 | idle_timeout
> >  test2     | f      | 0/16FDDE0   | reserved   | 2025-02-03
> > 18:29:53.478023+05:30 |
> > (2 rows)
> >
>
> Is this related to this patch or the behavior of HEAD? If this
> behavior is not introduced by this patch then we should discuss this
> in a separate thread. I couldn't think of why anyone wants to copy the
> invalid slots, so we should probably prohibit copying invalid slots
> but that is a matter of separate discussion unless introduced by this
> patch.
>

Hi Amit,

I tested and found that this issue is present in HEAD as well.

There are three types of invalidation in HEAD:
1. "wal_removed"
2. "rows_removed"
3. "wal_level_insufficient"

for copying slot with invalidation "wal_removed" we get an error:

postgres=# select slot_name, active, active_pid, restart_lsn,
wal_status, invalidation_reason from pg_replication_slots;
slot_name | active | active_pid | restart_lsn | wal_status | invalidation_reason
-----------+--------+------------+-------------+------------+---------------------
test1     | f      |            |             | lost       | wal_removed
(1 row)
postgres=#  select pg_copy_logical_replication_slot('test1', 'test2');
ERROR:  cannot copy a replication slot that doesn't reserve WAL


But for slot with invalidation "rows_removed" and
"wal_level_insufficient" we are able to copy the slot:

postgres=# select slot_name, active, active_pid, restart_lsn,
wal_status, invalidation_reason from pg_replication_slots;
slot_name | active | active_pid | restart_lsn | wal_status | invalidation_reason
-----------+--------+------------+-------------+------------+---------------------
slot1     | f      |            | 0/302E718   | lost       | rows_removed
(1 row)
postgres=# select pg_copy_logical_replication_slot('slot1', 'slot2');
pg_copy_logical_replication_slot
----------------------------------
(slot2,0/302E770)
(1 row)
postgres=# select slot_name, active, active_pid, restart_lsn,
wal_status, invalidation_reason from pg_replication_slots;
slot_name | active | active_pid | restart_lsn | wal_status | invalidation_reason
-----------+--------+------------+-------------+------------+---------------------
slot1     | f      |            | 0/302E718   | lost       | rows_removed
slot2     | f      |            | 0/302E718   | reserved   |
(2 rows)

Similarly we can copy slot with invalidation "wal_level_insufficient".
I have started a new thread to address the issue [1].

[1]: https://www.postgresql.org/message-id/CANhcyEU65aH0VYnLiu=OhNNxhnhNhwcXBeT-jvRe1OiJTo_Ayg@mail.gmail.com

Thanks and Regards,
Shlok Kyal

  369. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-04T10:27:51Z

    On Tue, Feb 4, 2025 at 10:45 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Feb 3, 2025 at 6:35 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > I reviewed the v66 patch. I have few comments:
> >
> > 1. I also feel the default value should be set to '0' as suggested by
> > Vignesh in 1st point of [1].
> >
>
> +1. This will ensure that the idle slots won't be invalidated by
> default, the same as HEAD. We can change the default value based on
> user inputs.
>

Here are the v68 patches, incorporating above as well as comments from [1].

Note: The 0003 patch with tests under PG_EXTRA_TESTS is not included
for now. If needed, I'll send it later once the first two patches are
committed.

[1] https://www.postgresql.org/message-id/CAHut%2BPv3mjQxmv5tHfgX%3Do%3D4C2TfX5rNYGS7xWrHBGcSVwr3mQ%40mail.gmail.com

--
Thanks,
Nisha

  370. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-02-04T11:11:47Z

    On Tue, 4 Feb 2025 at 15:58, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Here are the v68 patches, incorporating above as well as comments from [1].
>
Few comments:
1) Let's call TimestampDifferenceExceedsSeconds only if
idle_replication_slot_timeout_mins is set to avoid the
TimestampDifferenceExceedsSeconds function call and timestamp diff
calculation if not required:
+ if (CanInvalidateIdleSlot(s) &&
+ TimestampDifferenceExceedsSeconds(s->inactive_since, now,
+   idle_replication_slot_timeout_mins * SECS_PER_MINUTE))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+ }
+ break;

2) Let's keep the prototype after TimestampDifferenceExceeds to keep
it consistent with the source file and will also make it easy to
search:
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index d26f023fb8..e1d05d6779 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -143,5 +143,8 @@ extern int  date2isoyear(int year, int mon, int mday);
 extern int     date2isoyearday(int year, int mon, int mday);

 extern bool TimestampTimestampTzRequiresRewrite(void);
+extern bool TimestampDifferenceExceedsSeconds(TimestampTz start_time,
+
                   TimestampTz stop_time,
+
                   int threshold_sec);

3)How about we change the below:
+#ifdef USE_INJECTION_POINTS
+
+                                               /*
+                                                * To test idle
timeout slot invalidation, if the
+                                                * slot-time-out-inval
injection point is attached,
+                                                * set inactive_since
to a very old timestamp (1
+                                                * microsecond since
epoch) to immediately invalidate
+                                                * the slot.
+                                                */
+                                               if
(IS_INJECTION_POINT_ATTACHED("slot-time-out-inval"))
+                                                       s->inactive_since = 1;
+#endif
to:
#ifdef USE_INJECTION_POINTS
/*
* To test idle timeout slot invalidation, if the
* slot-time-out-inval injection point is attached,
* set inactive_since to current time and invalidate the slot immediately.
*/
if (IS_INJECTION_POINT_ATTACHED("slot-time-out-inval") &&
idle_replication_slot_timeout_mins)
{
invalidation_cause = cause;
inactive_since = s->inactive_since = now;
}
#else
/*
* Check if the slot needs to be invalidated due to
* idle_replication_slot_timeout GUC.
*/
if (TimestampDifferenceExceedsSeconds(s->inactive_since, now,
  idle_replication_slot_timeout_mins * SECS_PER_MINUTE))
{
invalidation_cause = cause;
inactive_since = s->inactive_since;
}
#endif

We can just invalidate the slot directly without checking the time
difference if idle_replication_slot_timeout_mins is set and
inactive_since can hold the now value.

Regards,
Vignesh

  371. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2025-02-04T12:16:28Z

    Dear Nisha,

Thanks for updating the patch! Here are my comments.

01.
```
+# Test for replication slots invalidation
```

Since the file tests only timeout invalidations, the comment seems too general.

02.
```
+       # Check that an invalidated slot cannot be acquired
+       my ($result, $stdout, $stderr);
+       ($result, $stdout, $stderr) = $node->psql(
+               'postgres', qq[
+                       SELECT pg_replication_slot_advance('$slot', '0/1');
+       ]);
+       ok( $stderr =~ /can no longer access replication slot "$slot"/,
+               "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+         )
+         or die
+         "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
```

This part can be removal because this is not directly related with timeout invalidation.
If needed this can be outside the function and we can confirm only once.

03.
```
+# Initialize primary
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init(allows_streaming => 'logical');
```

I think this node is not "primary" because there are no standby nodes. We can use new('node').
Also some comments which used "primary" can be removed.

04.
```
+$node->psql('postgres',
+       q{SELECT pg_create_logical_replication_slot('logical_slot', 'test_decoding');}
+);
```

Please use safe_psql() instead of psql().

05.
```
my $logstart = -s $node->logfile;
```

According to other tests, the variable name can be $log_offset.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  372. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-04T14:26:00Z

    On Tue, Feb 4, 2025 at 4:42 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 4 Feb 2025 at 15:58, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Here are the v68 patches, incorporating above as well as comments from [1].
> >
> Few comments:
> 1) Let's call TimestampDifferenceExceedsSeconds only if
> idle_replication_slot_timeout_mins is set to avoid the
> TimestampDifferenceExceedsSeconds function call and timestamp diff
> calculation if not required:
> + if (CanInvalidateIdleSlot(s) &&
> + TimestampDifferenceExceedsSeconds(s->inactive_since, now,
> +   idle_replication_slot_timeout_mins * SECS_PER_MINUTE))
> + {
> + invalidation_cause = cause;
> + inactive_since = s->inactive_since;
> + }
> + break;
>

The CanInvalidateIdleSlot(s) call does the check if
idle_replication_slot_timeout_mins is set or not. So we are good here.

> 2) Let's keep the prototype after TimestampDifferenceExceeds to keep
> it consistent with the source file and will also make it easy to
> search:
> diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
> index d26f023fb8..e1d05d6779 100644
> --- a/src/include/utils/timestamp.h
> +++ b/src/include/utils/timestamp.h
> @@ -143,5 +143,8 @@ extern int  date2isoyear(int year, int mon, int mday);
>  extern int     date2isoyearday(int year, int mon, int mday);
>
>  extern bool TimestampTimestampTzRequiresRewrite(void);
> +extern bool TimestampDifferenceExceedsSeconds(TimestampTz start_time,
> +
>                    TimestampTz stop_time,
> +
>                    int threshold_sec);
>

Done.

> 3)How about we change the below:
> +#ifdef USE_INJECTION_POINTS
> +
> +                                               /*
> +                                                * To test idle
> timeout slot invalidation, if the
> +                                                * slot-time-out-inval
> injection point is attached,
> +                                                * set inactive_since
> to a very old timestamp (1
> +                                                * microsecond since
> epoch) to immediately invalidate
> +                                                * the slot.
> +                                                */
> +                                               if
> (IS_INJECTION_POINT_ATTACHED("slot-time-out-inval"))
> +                                                       s->inactive_since = 1;
> +#endif
> to:
> #ifdef USE_INJECTION_POINTS
> /*
> * To test idle timeout slot invalidation, if the
> * slot-time-out-inval injection point is attached,
> * set inactive_since to current time and invalidate the slot immediately.
> */
> if (IS_INJECTION_POINT_ATTACHED("slot-time-out-inval") &&
> idle_replication_slot_timeout_mins)
> {
> invalidation_cause = cause;
> inactive_since = s->inactive_since = now;
> }
> #else
> /*
> * Check if the slot needs to be invalidated due to
> * idle_replication_slot_timeout GUC.
> */
> if (TimestampDifferenceExceedsSeconds(s->inactive_since, now,
>   idle_replication_slot_timeout_mins * SECS_PER_MINUTE))
> {
> invalidation_cause = cause;
> inactive_since = s->inactive_since;
> }
> #endif
>
> We can just invalidate the slot directly without checking the time
> difference if idle_replication_slot_timeout_mins is set and
> inactive_since can hold the now value.
>

+1 to the idea. Implemented it in a slightly different way to avoid
enclosing the main code within "#else".

Here is v69 patch set addressing above and Kuroda-san's comments in [1].

[1] https://www.postgresql.org/message-id/OSCPR01MB14966A918EBB0674E5423EDE0F5F42%40OSCPR01MB14966.jpnprd01.prod.outlook.com

--
Thanks,
Nisha

  373. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-05T03:13:34Z

    Review comments for v69-0001.

======
doc/src/sgml/config.sgml

1.
+       <para>
+        Invalidate replication slots that have remained idle longer than this
+        duration. If this value is specified without units, it is taken as
+        minutes. A value of zero (which is default) disables the idle timeout
+        invalidation mechanism. This parameter can only be set in the
+        <filename>postgresql.conf</filename> file or on the server command
+        line.
+       </para>

Suggest writing "(the default)" instead of "(which is default)" to be
consistent with the wording of other descriptions on this page.

======
src/backend/replication/slot.c

2.
+static inline bool
+CanInvalidateIdleSlot(ReplicationSlot *s)
+{
+ return (idle_replication_slot_timeout_mins > 0 &&
+ !XLogRecPtrIsInvalid(s->data.restart_lsn) &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
 }

I wasn't sure why those conditions were '> 0' instead of just '!= 0'.
IIUC negative values aren't possible for
idle_replication_slot_timeout_mins and active_since anyhow.

But, the current patch code is also ok if you prefer.

~~~

3.
+ if (cause == RS_INVAL_IDLE_TIMEOUT)
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+

SUGGESTION
Assign the current time here to reduce system call overhead while
holding the spinlock in subsequent code.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  374. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-02-05T04:59:52Z

    On Tue, 4 Feb 2025 at 19:56, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Here is v69 patch set addressing above and Kuroda-san's comments in [1].

Few minor suggestions:
1) In the slot invalidation reporting below:
+               case RS_INVAL_IDLE_TIMEOUT:
+                       Assert(inactive_since > 0);
+
+                       /* translator: second %s is a GUC variable name */
+                       appendStringInfo(&err_detail, _("The slot's
idle time %s exceeds the configured \"%s\" duration."),
+
timestamptz_to_str(inactive_since),
+
"idle_replication_slot_timeout");
+                       /* translator: %s is a GUC variable name */
+                       appendStringInfo(&err_hint, _("You might need
to increase \"%s\"."),
+
"idle_replication_slot_timeout");

It is logged like:
2025-02-05 10:04:11.616 IST [330567] DETAIL:  The slot's idle time
2025-02-05 10:02:49.131631+05:30 exceeds the configured
"idle_replication_slot_timeout" duration.

Here even though we tell idle time, we are logging the inactive_since
value which kind of gives a wrong meaning.

How about we change it to:
The slot has been inactive since 2025-02-05 10:02:49.131631+05:30,
which exceeds the configured "idle_replication_slot_timeout" duration.

2) Here we have mentioned about invalidation happens only for a)
released slots b) inactive slots replication slots c) slot where
communication between pub and sub is down
+                * XXX: Slot invalidation due to 'idle_timeout' applies only to
+                * released slots, and is based on the
'idle_replication_slot_timeout'
+                * GUC. Active slots currently in use for replication
are excluded to
+                * prevent accidental invalidation. Slots where
communication between
+                * the publisher and subscriber is down are also
excluded, as they are
+                * managed by the 'wal_sender_timeout'.
+                */
+               InvalidateObsoleteReplicationSlots(RS_INVAL_IDLE_TIMEOUT,
+
            0,
+
            InvalidOid,
+
            InvalidTransactionId);
a) Can we include about slots which does not reserve WAL are also not
considered.
b) Could we present this in a bullet-point format like the following:
+                * XXX: Slot invalidation due to 'idle_timeout' applies only to:
+                * 1) released slots, and is based on the
'idle_replication_slot_timeout'
+                * GUC. 2) Active slots currently in use for
replication are excluded to
+                * prevent accidental invalidation. 3) Slots where
communication between
+                * the publisher and subscriber is down are also
excluded, as they are
+                * managed by the 'wal_sender_timeout'.
+                */
c) While I was initially reviewing the patch I also had the similar
thoughts on my mind, if we could mention the one like "Slots where
communication between the publisher and subscriber is down are also
excluded, as they are managed by the 'wal_sender_timeout'" in the
documentation it might be good.

Regards,
Vignesh

  375. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-05T07:28:09Z

    Hi Nisha,

Some review comments for the patch v69-0002.

======
src/backend/replication/slot.c

1.
+#ifdef USE_INJECTION_POINTS
+
+ /*
+ * To test idle timeout slot invalidation, if the
+ * slot-time-out-inval injection point is attached,
+ * immediately invalidate the slot.
+ */
+ if (IS_INJECTION_POINT_ATTACHED("slot-time-out-inval"))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since = now;
+ break;
+ }
+#endif

1a.
I didn't understand the reason for the assignment ' = now' here. This
is not happening in the normal code path so why do you need to do this
in this test code path? It works for me without doing this.

~

1b.
For testing, I think we should try to keep the injection code
differences minimal -- e.g. share the same (normal build) code as much
as possible. For example, I suggest refactoring like below. Well, it
works for me.

/*
 * Check if the slot needs to be invalidated due to
 * idle_replication_slot_timeout GUC.
 *
 * To test idle timeout slot invalidation, if the
 * "slot-time-out-inval" injection point is attached,
 * immediately invalidate the slot.
 */
if (
#ifdef USE_INJECTION_POINTS
  IS_INJECTION_POINT_ATTACHED("slot-time-out-inval") ||
#endif
  TimestampDifferenceExceedsSeconds(s->inactive_since, now,
    idle_replication_slot_timeout_mins * SECS_PER_MINUTE))
{
  invalidation_cause = cause;
  inactive_since = s->inactive_since;
}

~

1c.
Can we call the injection point "timeout" instead of "time-out"?

======
.../t/044_invalidate_inactive_slots.pl

2.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}

At first, I had no idea how to build for this test. It would be good
to include a link to the injection build instructions in a comment
somewhere near here.

~~~

3.
+# Wait for slot to first become idle and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset) = @_;
+ my $node_name = $node->name;
+

Might be better to call the variable $slot_name instead of $slot.

Also then it will be consistent with $node_name

~~~

4.
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.

I misread this comment at first -- maybe it is clearer to reverse the wording?

/extension injection_points/’injection_points’ extension/

~~~

5.
+# Run a checkpoint which will invalidate the slots
+$node->safe_psql('postgres', "CHECKPOINT");

The explanation seems a bit terse -- I think the comment should
elaborate a bit more to explain that CHECKPOINT is just where the idle
slot timeout is checked, but since the test is using injection point
and the injection code enforces immediate idle timeout THAT is why it
will invalidate the slots...

~~~

6.
+# Wait for slots to become inactive. Note that nobody has acquired the slot
+# yet, so it must get invalidated due to idle timeout.

IIUC this comment means:

SUGGESTION
Note that since nobody has acquired the slot yet, then if it has been
invalidated that can only be due to the idle timeout mechanism.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  376. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-05T09:12:31Z

    On Wed, Feb 5, 2025 at 10:30 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 4 Feb 2025 at 19:56, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Here is v69 patch set addressing above and Kuroda-san's comments in [1].
>
> Few minor suggestions:
> 1) In the slot invalidation reporting below:
> +               case RS_INVAL_IDLE_TIMEOUT:
> +                       Assert(inactive_since > 0);
> +
> +                       /* translator: second %s is a GUC variable name */
> +                       appendStringInfo(&err_detail, _("The slot's
> idle time %s exceeds the configured \"%s\" duration."),
> +
> timestamptz_to_str(inactive_since),
> +
> "idle_replication_slot_timeout");
> +                       /* translator: %s is a GUC variable name */
> +                       appendStringInfo(&err_hint, _("You might need
> to increase \"%s\"."),
> +
> "idle_replication_slot_timeout");
>
> It is logged like:
> 2025-02-05 10:04:11.616 IST [330567] DETAIL:  The slot's idle time
> 2025-02-05 10:02:49.131631+05:30 exceeds the configured
> "idle_replication_slot_timeout" duration.
>
> Here even though we tell idle time, we are logging the inactive_since
> value which kind of gives a wrong meaning.
>
> How about we change it to:
> The slot has been inactive since 2025-02-05 10:02:49.131631+05:30,
> which exceeds the configured "idle_replication_slot_timeout" duration.
>

Would it address your concern if we write the actual idle duration
(now - inactive_since) instead of directly using inactive_since in the
above message?

A few other comments:
1.
+ * 4. The slot is not being synced from the primary while the server
+ *    is in recovery
+ *
+ * Note that the idle timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.

The 4th point in the above comment and the rest of the comment is
mostly saying the same thing.

2.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
  *
  * It is convenient to flush dirty replication slots at the time of checkpoint.
  * Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1924,6 +2007,45 @@ CheckPointReplicationSlots(bool is_shutdown)

Can we try and see how the patch looks if we try to invalidate the
slot due to idle time at the same time when we are trying to
invalidate due to WAL?

-- 
With Regards,
Amit Kapila.

  377. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-06T02:32:07Z

    On Wed, Feb 5, 2025 at 10:30 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 4 Feb 2025 at 19:56, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Here is v69 patch set addressing above and Kuroda-san's comments in [1].
>
> 2) Here we have mentioned about invalidation happens only for a)
> released slots b) inactive slots replication slots c) slot where
> communication between pub and sub is down
> +                * XXX: Slot invalidation due to 'idle_timeout' applies only to
> +                * released slots, and is based on the
> 'idle_replication_slot_timeout'
> +                * GUC. Active slots currently in use for replication
> are excluded to
> +                * prevent accidental invalidation. Slots where
> communication between
> +                * the publisher and subscriber is down are also
> excluded, as they are
> +                * managed by the 'wal_sender_timeout'.
> +                */
> +               InvalidateObsoleteReplicationSlots(RS_INVAL_IDLE_TIMEOUT,
> +
>             0,
> +
>             InvalidOid,
> +
>             InvalidTransactionId);
> a) Can we include about slots which does not reserve WAL are also not
> considered.

We have included all the info regarding which slots are excluded in
the documents, so I feel we can remove the XXX: comment from here.
(done in v70).

> c) While I was initially reviewing the patch I also had the similar
> thoughts on my mind, if we could mention the one like "Slots where
> communication between the publisher and subscriber is down are also
> excluded, as they are managed by the 'wal_sender_timeout'" in the
> documentation it might be good.
>

v70 adds the suggested info in the docs.

--
Thanks,
Nisha

  378. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-06T02:32:13Z

    On Wed, Feb 5, 2025 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha,
>
> Some review comments for the patch v69-0002.
>
> ======
> .../t/044_invalidate_inactive_slots.pl
>
> 2.
> +if ($ENV{enable_injection_points} ne 'yes')
> +{
> + plan skip_all => 'Injection points not supported by this build';
> +}
>
> At first, I had no idea how to build for this test. It would be good
> to include a link to the injection build instructions in a comment
> somewhere near here.
>

I’ve added comments with build instructions in v70, but I’m not sure
if a link to the documentation is necessary. I didn’t find similar
instructions in other injection point-dependent tests. Let’s see what
others think.

--
Thanks,
Nisha

  379. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-06T02:32:29Z

    On Wed, Feb 5, 2025 at 2:42 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Feb 5, 2025 at 10:30 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, 4 Feb 2025 at 19:56, Nisha Moond <nisha.moond412@gmail.com> wrote:
> > >
> > > Here is v69 patch set addressing above and Kuroda-san's comments in [1].
> >
> > Few minor suggestions:
> > 1) In the slot invalidation reporting below:
> > +               case RS_INVAL_IDLE_TIMEOUT:
> > +                       Assert(inactive_since > 0);
> > +
> > +                       /* translator: second %s is a GUC variable name */
> > +                       appendStringInfo(&err_detail, _("The slot's
> > idle time %s exceeds the configured \"%s\" duration."),
> > +
> > timestamptz_to_str(inactive_since),
> > +
> > "idle_replication_slot_timeout");
> > +                       /* translator: %s is a GUC variable name */
> > +                       appendStringInfo(&err_hint, _("You might need
> > to increase \"%s\"."),
> > +
> > "idle_replication_slot_timeout");
> >
> > It is logged like:
> > 2025-02-05 10:04:11.616 IST [330567] DETAIL:  The slot's idle time
> > 2025-02-05 10:02:49.131631+05:30 exceeds the configured
> > "idle_replication_slot_timeout" duration.
> >
> > Here even though we tell idle time, we are logging the inactive_since
> > value which kind of gives a wrong meaning.
> >
> > How about we change it to:
> > The slot has been inactive since 2025-02-05 10:02:49.131631+05:30,
> > which exceeds the configured "idle_replication_slot_timeout" duration.
> >
>
> Would it address your concern if we write the actual idle duration
> (now - inactive_since) instead of directly using inactive_since in the
> above message?
>

Simply using the raw timestamp difference (now - inactive_since) would
look odd. We should convert it into a user-friendly format. Since
idle_replication_slot_timeout is in minutes, we can express the
difference in minutes and seconds in the log.
For example:
DETAIL: The slot's idle time of 1 minute and 7 seconds exceeds the
configured "idle_replication_slot_timeout" duration.

This has been implemented in v70.
Thoughts?

> A few other comments:
> 1.
> + * 4. The slot is not being synced from the primary while the server
> + *    is in recovery
> + *
> + * Note that the idle timeout invalidation mechanism is not
> + * applicable for slots on the standby server that are being synced
> + * from the primary server (i.e., standby slots having 'synced' field 'true').
> + * Synced slots are always considered to be inactive because they don't
> + * perform logical decoding to produce changes.
>
> The 4th point in the above comment and the rest of the comment is
> mostly saying the same thing.
>

Done. I've merged the additional info and 4th point.

> 2.
> + * Flush all replication slots to disk. Also, invalidate obsolete slots during
> + * non-shutdown checkpoint.
>   *
>   * It is convenient to flush dirty replication slots at the time of checkpoint.
>   * Additionally, in case of a shutdown checkpoint, we also identify the slots
> @@ -1924,6 +2007,45 @@ CheckPointReplicationSlots(bool is_shutdown)
>
> Can we try and see how the patch looks if we try to invalidate the
> slot due to idle time at the same time when we are trying to
> invalidate due to WAL?
>

I'll consider the suggested change in the next version.
~~~~

Here are the v70 patches -  addressed above and other comments in [1],
[2] and [3].

[1] https://www.postgresql.org/message-id/CAHut%2BPvW3pr3P3hXwBskXrDmJYKedmqRaPZcL4iLRQ51%3DXxOBw%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CALDaNm0X_vgAxKPT%2Bc14yqKcgE5-x4XBdXsCAVqD6_aa-QYUvg%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CAHut%2BPtCpOnifF9wnhJ%3Djo7KLmtT%3DMikuYnM9GGPTVA80rq7OA%40mail.gmail.com

--
Thanks,
Nisha

  380. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-06T04:47:05Z

    On Thu, Feb 6, 2025 at 8:02 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Wed, Feb 5, 2025 at 2:42 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Would it address your concern if we write the actual idle duration
> > (now - inactive_since) instead of directly using inactive_since in the
> > above message?
> >
>
> Simply using the raw timestamp difference (now - inactive_since) would
> look odd. We should convert it into a user-friendly format. Since
> idle_replication_slot_timeout is in minutes, we can express the
> difference in minutes and seconds in the log.
> For example:
> DETAIL: The slot's idle time of 1 minute and 7 seconds exceeds the
> configured "idle_replication_slot_timeout" duration.
>

This is better but the implementation should be done on the caller
side mainly because we don't want to call a new GetCurrentTimestamp()
in ReportSlotInvalidation.

>
> > 2.
> > + * Flush all replication slots to disk. Also, invalidate obsolete slots during
> > + * non-shutdown checkpoint.
> >   *
> >   * It is convenient to flush dirty replication slots at the time of checkpoint.
> >   * Additionally, in case of a shutdown checkpoint, we also identify the slots
> > @@ -1924,6 +2007,45 @@ CheckPointReplicationSlots(bool is_shutdown)
> >
> > Can we try and see how the patch looks if we try to invalidate the
> > slot due to idle time at the same time when we are trying to
> > invalidate due to WAL?
> >
>
> I'll consider the suggested change in the next version.
>

FYI, we discussed this previously (1), but the conclusion that it
won't help much (as it will not help to remove WAL immediately) is
incorrect, especially if we do what is suggested now.

Apart from this, I have made minor changes in the comments. Please
review and include them unless you disagree.

(1) - https://www.postgresql.org/message-id/CALj2ACXe8%2BxSNdMXTMaSRWUwX7v61Ad4iddUwnn%3DdjSwx3GLLg%40mail.gmail.com


-- 
With Regards,
Amit Kapila.

  381. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-06T04:49:21Z

    On Thu, Feb 6, 2025 at 10:17 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Feb 6, 2025 at 8:02 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
>
> >
> > > 2.
> > > + * Flush all replication slots to disk. Also, invalidate obsolete slots during
> > > + * non-shutdown checkpoint.
> > >   *
> > >   * It is convenient to flush dirty replication slots at the time of checkpoint.
> > >   * Additionally, in case of a shutdown checkpoint, we also identify the slots
> > > @@ -1924,6 +2007,45 @@ CheckPointReplicationSlots(bool is_shutdown)
> > >
> > > Can we try and see how the patch looks if we try to invalidate the
> > > slot due to idle time at the same time when we are trying to
> > > invalidate due to WAL?
> > >
> >
> > I'll consider the suggested change in the next version.
> >
>
> FYI, we discussed this previously (1), but the conclusion that it
> won't help much (as it will not help to remove WAL immediately) is
> incorrect, especially if we do what is suggested now.
>

The above sentence is incomplete. Let me re-write it. We discussed
this previously, but the conclusion that it won't help much (as it
will not help to remove WAL immediately) at the time shutdown
checkpoint is incorrect, especially if we do what is suggested now.
So, we should try to invalidate the slots even during shutdown
checkpoints.

-- 
With Regards,
Amit Kapila.

  382. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-06T10:38:17Z

    On Thu, Feb 6, 2025 at 10:17 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Feb 6, 2025 at 8:02 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > On Wed, Feb 5, 2025 at 2:42 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > Would it address your concern if we write the actual idle duration
> > > (now - inactive_since) instead of directly using inactive_since in the
> > > above message?
> > >
> >
> > Simply using the raw timestamp difference (now - inactive_since) would
> > look odd. We should convert it into a user-friendly format. Since
> > idle_replication_slot_timeout is in minutes, we can express the
> > difference in minutes and seconds in the log.
> > For example:
> > DETAIL: The slot's idle time of 1 minute and 7 seconds exceeds the
> > configured "idle_replication_slot_timeout" duration.
> >
>
> This is better but the implementation should be done on the caller
> side mainly because we don't want to call a new GetCurrentTimestamp()
> in ReportSlotInvalidation.
>

Done.

> >
> > > 2.
> > > + * Flush all replication slots to disk. Also, invalidate obsolete slots during
> > > + * non-shutdown checkpoint.
> > >   *
> > >   * It is convenient to flush dirty replication slots at the time of checkpoint.
> > >   * Additionally, in case of a shutdown checkpoint, we also identify the slots
> > > @@ -1924,6 +2007,45 @@ CheckPointReplicationSlots(bool is_shutdown)
> > >
> > > Can we try and see how the patch looks if we try to invalidate the
> > > slot due to idle time at the same time when we are trying to
> > > invalidate due to WAL?
> > >
> >
> > I'll consider the suggested change in the next version.
> >
>

Done the changes as suggested in v71.

> FYI, we discussed this previously (1), but the conclusion that it
> won't help much (as it will not help to remove WAL immediately) is
> incorrect, especially if we do what is suggested now.
>
> Apart from this, I have made minor changes in the comments. Please
> review and include them unless you disagree.
>

Done.
~~~~
Here are the v71 patches with the above comments incorporated.

--
Thanks,
Nisha

  383. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-02-06T13:56:33Z

    On Thu, 6 Feb 2025 at 16:08, Nisha Moond <nisha.moond412@gmail.com> wrote:
> Here are the v71 patches with the above comments incorporated.

Few comments:
1) While changing the switch to an if condition, the behavior of the
break statement has changed. Previously, it would exit the switch, but
now it exits the main for loop without releasing the locks. These
should be replaced with a goto to ensure the locks are properly
released.
+                       if (cause & RS_INVAL_HORIZON)
+                       {
+                               if (!SlotIsLogical(s))
                                        break;
-                               case RS_INVAL_WAL_LEVEL:
-                                       if (SlotIsLogical(s))
-                                               invalidation_cause = cause;
+                               /* invalid DB oid signals a shared relation */
+                               if (dboid != InvalidOid && dboid !=
s->data.database)
                                        break;

2) None of this initialization is required, as we will be setting
these values before using it:
+       int                     minutes = 0;
+       int                     secs = 0;
+       long            elapsed_secs = 0;

Regards,
Vignesh

  384. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-07T02:29:50Z

    Hi Nisha,

Some review comments for v71-0001.

======
src/backend/access/transam/xlog.c

1.
  XLByteToSeg(RedoRecPtr, _logSegNo, wal_segment_size);
  KeepLogSeg(recptr, &_logSegNo);
- if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
+ if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED |
RS_INVAL_IDLE_TIMEOUT,
     _logSegNo, InvalidOid,
     InvalidTransactionId))
  {
@@ -7792,7 +7792,7 @@ CreateRestartPoint(int flags)
  replayPtr = GetXLogReplayRecPtr(&replayTLI);
  endptr = (receivePtr < replayPtr) ? replayPtr : receivePtr;
  KeepLogSeg(endptr, &_logSegNo);
- if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
+ if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED |
RS_INVAL_IDLE_TIMEOUT,
     _logSegNo, InvalidOid,
     InvalidTransactionId))

It seems fundamentally strange to me to assign multiple simultaneous
causes like this. IMO you can't invalidate something that is invalid
already. I gues v71 was an attempt to implement Amit's:

> > Can we try and see how the patch looks if we try to invalidate the
> > slot due to idle time at the same time when we are trying to
> > invalidate due to WAL?

But, AFAICT the current code now has a confused mixture of:
'cause' parameter meaning "this is the invalidation cause", versus
'cause' parameter meaning "here is a mask of possible causes"

======
src/backend/replication/slot.c

SlotInvalidationCauses[]

2.
  [RS_INVAL_WAL_REMOVED] = "wal_removed",
  [RS_INVAL_HORIZON] = "rows_removed",
  [RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_IDLE_TIMEOUT] = "idle_timeout",
 };

By using bit flags in the enum (see slot.h) and designated
initializers here in SlotInvalidationCauses[], you'll end up with 9
entries (0-0x08) instead of 4, and the other undesignated entries will
be all NULL. Maybe it is intended, but if it is I think it is strange
to be indexing by bit flags so at least you should have a comment.

If you really need bitflags then perhaps it is better to maintain them
in addition to the v70 enum values (??)

~~~

3.
 /* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_IDLE_TIMEOUT

Hmm. The impact of using bit flags has (probably) unintended
consequences. e.g. Now you've made the GetSlotInvalidationCause()
function worse than before because now it will be iterating over all
the undesignated NULL entries of the array when searching for the
matching cause.

~~~

4.
+ /* Calculate the idle time duration of the slot */
+ elapsed_secs = (now - inactive_since) / USECS_PER_SEC;
+ minutes = elapsed_secs / SECS_PER_MINUTE;
+ secs = elapsed_secs % SECS_PER_MINUTE;
+
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("The slot's idle time of %d minutes
and %d seconds exceeds the configured \"%s\" duration."),
+ minutes, secs, "idle_replication_slot_timeout");

Idleness timeout durations defined like 1d aren't going to look pretty
using this log format. We already discussed off-list about how to make
this better, but not done yet?

~~~

5.
+ if (cause & RS_INVAL_HORIZON)
+ {
+ if (!SlotIsLogical(s))
  break;

The meaning of the 'break' here is different to before. Now breaking
the entire for-loop instead of just breaking from the switch.
(same already posted by Vignesh)

~~~

6.
  ReportSlotInvalidation(invalidation_cause, true, active_pid,
     slotname, restart_lsn,
-    oldestLSN, snapshotConflictHorizon);
+    oldestLSN, snapshotConflictHorizon,
+    inactive_since, now);

  if (MyBackendType == B_STARTUP)
  (void) SendProcSignal(active_pid,
@@ -1785,7 +1881,8 @@
InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,

  ReportSlotInvalidation(invalidation_cause, false, active_pid,
     slotname, restart_lsn,
-    oldestLSN, snapshotConflictHorizon);
+    oldestLSN, snapshotConflictHorizon,
+    inactive_since, now);

If the cause was not already (masked with) RS_INVAL_IDLE_TIMEOUT then
AFAICT 'now' will still be 0 here.

This seems an unexpected quirk, which at best is quite misleading.
Even if the code sty like this I felt ReportSlotInvalidation should
Assert 'now' must be 0 unless the cause passed was
RS_INVAL_IDLE_TIMEOUT.

~~~

CheckPointReplicationSlots:

7.
 /*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.

Since the v70 code was removed in v71, the function now is the same as
master. So did we need the function comment change?

======
src/include/replication/slot.h

8.
- * When adding a new invalidation cause here, remember to update
+ * When adding a new invalidation cause here, the value must be powers of 2
+ * (e.g., 1, 2, 4...) for proper bitwise operations. Also, remember to update
  * SlotInvalidationCauses and RS_INVAL_MAX_CAUSES.
  */
 typedef enum ReplicationSlotInvalidationCause
 {
- RS_INVAL_NONE,
+ RS_INVAL_NONE = 0x00,
  /* required WAL has been removed */
- RS_INVAL_WAL_REMOVED,
+ RS_INVAL_WAL_REMOVED = 0x01,
  /* required rows have been removed */
- RS_INVAL_HORIZON,
+ RS_INVAL_HORIZON = 0x02,
  /* wal_level insufficient for slot */
- RS_INVAL_WAL_LEVEL,
+ RS_INVAL_WAL_LEVEL = 0x04,
+ /* idle slot timeout has occurred */
+ RS_INVAL_IDLE_TIMEOUT = 0x08,
 } ReplicationSlotInvalidationCause;

8a.
IMO enums are intended for  discrete values like "red" or "blue", but
not combinations of values like "reddy-bluey". AFAIK this kind of
usage is not normal and is discouraged in C programming.

So if you need bitflags then really the bit flags should be #define etc.

~

8b.
Does it make sense? You can't invalidate something that is already
invalid, so what does it even mean to have multiple simultaneous
ReplicationSlotInvalidationCause values? AFAICT it was only done like
this to CHECK for multiple **possible**  causes, but this point is not
very clear

~

8c.
This introduces a side-effect that now the char *const
SlotInvalidationCauses[] array in slot.c will have 8 entries, half of
them NULL. Already mentioned elsewhere. And, this will get
increasingly worse if more invalidation reasons get added. 8,16,32,64
mostly unused entries etc...

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  385. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-07T05:52:54Z

    On Fri, Feb 7, 2025 at 8:00 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> ======
> src/backend/access/transam/xlog.c
>
> 1.
>   XLByteToSeg(RedoRecPtr, _logSegNo, wal_segment_size);
>   KeepLogSeg(recptr, &_logSegNo);
> - if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
> + if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED |
> RS_INVAL_IDLE_TIMEOUT,
>      _logSegNo, InvalidOid,
>      InvalidTransactionId))
>   {
> @@ -7792,7 +7792,7 @@ CreateRestartPoint(int flags)
>   replayPtr = GetXLogReplayRecPtr(&replayTLI);
>   endptr = (receivePtr < replayPtr) ? replayPtr : receivePtr;
>   KeepLogSeg(endptr, &_logSegNo);
> - if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
> + if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED |
> RS_INVAL_IDLE_TIMEOUT,
>      _logSegNo, InvalidOid,
>      InvalidTransactionId))
>
> It seems fundamentally strange to me to assign multiple simultaneous
> causes like this. IMO you can't invalidate something that is invalid
> already. I gues v71 was an attempt to implement Amit's:
>

The idea is to invalidate the slot either due to WAL_REMOVED or
IDLE_TIMEOUT in one go during the checkpoint instead of taking
multiple passes over the slots during the checkpoint. Feel free to
suggest if you can think of a better way to implement it.

-- 
With Regards,
Amit Kapila.

  386. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-07T13:05:33Z

    On Fri, Feb 7, 2025 at 8:00 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha,
>
> Some review comments for v71-0001.
>
> ======
> src/backend/replication/slot.c
>
> SlotInvalidationCauses[]
>
> 2.
>   [RS_INVAL_WAL_REMOVED] = "wal_removed",
>   [RS_INVAL_HORIZON] = "rows_removed",
>   [RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
> + [RS_INVAL_IDLE_TIMEOUT] = "idle_timeout",
>  };
>
> By using bit flags in the enum (see slot.h) and designated
> initializers here in SlotInvalidationCauses[], you'll end up with 9
> entries (0-0x08) instead of 4, and the other undesignated entries will
> be all NULL. Maybe it is intended, but if it is I think it is strange
> to be indexing by bit flags so at least you should have a comment.
>
> If you really need bitflags then perhaps it is better to maintain them
> in addition to the v70 enum values (??)
>
> ~~~
>
> 3.
>  /* Maximum number of invalidation causes */
> -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
> +#define RS_INVAL_MAX_CAUSES RS_INVAL_IDLE_TIMEOUT
>
> Hmm. The impact of using bit flags has (probably) unintended
> consequences. e.g. Now you've made the GetSlotInvalidationCause()
> function worse than before because now it will be iterating over all
> the undesignated NULL entries of the array when searching for the
> matching cause.
>

Introduced a new struct, "SlotInvalidationCauseMap", to store
invalidation cause enums and their corresponding cause_name strings.
Replaced "SlotInvalidationCauses[]" with a map (array of structs),
eliminating extra NULL spaces and reducing unnecessary iterations.
With this change, a new function, GetSlotInvalidationCauseName(), was
added to retrieve the cause_name string for a given cause_enum.

> ~~~
>
> 4.
> + /* Calculate the idle time duration of the slot */
> + elapsed_secs = (now - inactive_since) / USECS_PER_SEC;
> + minutes = elapsed_secs / SECS_PER_MINUTE;
> + secs = elapsed_secs % SECS_PER_MINUTE;
> +
> + /* translator: %s is a GUC variable name */
> + appendStringInfo(&err_detail, _("The slot's idle time of %d minutes
> and %d seconds exceeds the configured \"%s\" duration."),
> + minutes, secs, "idle_replication_slot_timeout");
>
> Idleness timeout durations defined like 1d aren't going to look pretty
> using this log format. We already discussed off-list about how to make
> this better, but not done yet?
>

There was an off-list suggestion to include the configured GUC value
in the err_detail message for better clarity. This change makes it
easier for users to compare, especially for large values like 1d.
For example, if the timeout duration is set to 1d, the message will
now appear as:

" The slot's idle time of 1440 minutes and 54 seconds exceeds the
configured "idle_replication_slot_timeout" duration of 1440 minutes."

Thoughts?

> ~~~
>
> 6.
>   ReportSlotInvalidation(invalidation_cause, true, active_pid,
>      slotname, restart_lsn,
> -    oldestLSN, snapshotConflictHorizon);
> +    oldestLSN, snapshotConflictHorizon,
> +    inactive_since, now);
>
>   if (MyBackendType == B_STARTUP)
>   (void) SendProcSignal(active_pid,
> @@ -1785,7 +1881,8 @@
> InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
>
>   ReportSlotInvalidation(invalidation_cause, false, active_pid,
>      slotname, restart_lsn,
> -    oldestLSN, snapshotConflictHorizon);
> +    oldestLSN, snapshotConflictHorizon,
> +    inactive_since, now);
>
> If the cause was not already (masked with) RS_INVAL_IDLE_TIMEOUT then
> AFAICT 'now' will still be 0 here.
>
> This seems an unexpected quirk, which at best is quite misleading.
> Even if the code sty like this I felt ReportSlotInvalidation should
> Assert 'now' must be 0 unless the cause passed was
> RS_INVAL_IDLE_TIMEOUT.
>

'now' will be non-zero even when  RS_INVAL_IDLE_TIMEOUT is masked with
other possible causes like RS_INVAL_WAL_REMOVED, and the slot gets
invalidated first due to RS_INVAL_WAL_REMOVED.
 Therefore, 'now' being non-zero is not exclusive to
RS_INVAL_IDLE_TIMEOUT. However, since it must be non-zero when the
cause in ReportSlotInvalidation() is RS_INVAL_IDLE_TIMEOUT, I've added
an Assert for the same.

> ~~~
> ======
> src/include/replication/slot.h
>
> 8.
> - * When adding a new invalidation cause here, remember to update
> + * When adding a new invalidation cause here, the value must be powers of 2
> + * (e.g., 1, 2, 4...) for proper bitwise operations. Also, remember to update
>   * SlotInvalidationCauses and RS_INVAL_MAX_CAUSES.
>   */
>  typedef enum ReplicationSlotInvalidationCause
>  {
> - RS_INVAL_NONE,
> + RS_INVAL_NONE = 0x00,
>   /* required WAL has been removed */
> - RS_INVAL_WAL_REMOVED,
> + RS_INVAL_WAL_REMOVED = 0x01,
>   /* required rows have been removed */
> - RS_INVAL_HORIZON,
> + RS_INVAL_HORIZON = 0x02,
>   /* wal_level insufficient for slot */
> - RS_INVAL_WAL_LEVEL,
> + RS_INVAL_WAL_LEVEL = 0x04,
> + /* idle slot timeout has occurred */
> + RS_INVAL_IDLE_TIMEOUT = 0x08,
>  } ReplicationSlotInvalidationCause;
>
> 8a.
> IMO enums are intended for  discrete values like "red" or "blue", but
> not combinations of values like "reddy-bluey". AFAIK this kind of
> usage is not normal and is discouraged in C programming.
>
> So if you need bitflags then really the bit flags should be #define etc.
>

I feel using the ReplicationSlotInvalidationCause type instead of
"int" (in case we use macros) improves code readability and
maintainability.
OTOH, keeping the enums as they are in v70, and defining new macros
for the very similar purpose could add unnecessary complexity to code
management.

> ~
>
> 8b.
> Does it make sense? You can't invalidate something that is already
> invalid, so what does it even mean to have multiple simultaneous
> ReplicationSlotInvalidationCause values? AFAICT it was only done like
> this to CHECK for multiple **possible**  causes, but this point is not
> very clear
>

Added comments at the top of InvalidateObsoleteReplicationSlots() to
clarify that it tries to invalidate slots for multiple possible causes
in a single pass, as explained in [1].

> ~
>
> 8c.
> This introduces a side-effect that now the char *const
> SlotInvalidationCauses[] array in slot.c will have 8 entries, half of
> them NULL. Already mentioned elsewhere. And, this will get
> increasingly worse if more invalidation reasons get added. 8,16,32,64
> mostly unused entries etc...
>

This issue is now resolved by replacing SlotInvalidationCauses[] with
a new array of structures.

> ======

Attached v72 patches, addressed the above comments as well as
Vignesh's comments in [2].
 - There are no new changes in patch-002.

[1] https://www.postgresql.org/message-id/CAA4eK1Jatoapf2NoX2nJOJ8k-RvEZM%3DMoFUvNWPz4rRR1simQw%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CALDaNm3wx8ihfkidveKuK%3DgGujS_yc9sEgq6ev-T%2BW3zeHM88g%40mail.gmail.com

--
Thanks,
Nisha

  387. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2025-02-08T06:58:45Z

    On Friday, February 7, 2025 9:06 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> 
> Attached v72 patches, addressed the above comments as well as Vignesh's
> comments in [2].
>  - There are no new changes in patch-002.

Thanks for updating the patch, I have few review comments:

1.
> InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause cause,

I think the type of first parameter 'cause' is not appropriate anymore since
it's now a bitmap flag instead of an enum.

2.
> -StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
> -				 "array length mismatch");
> +#define	RS_INVAL_MAX_CAUSES (sizeof(InvalidationCauses) / sizeof(InvalidationCauses[0]))

I'd like to confirm if the current value of the RS_INVAL_MAX_CAUSES is correct.
Previously, the value is arrary_length - 1, while now it seems equal to the
arrary_length.

And ISTM we could directly call lengthof() here.

3.

+			if (cause & RS_INVAL_HORIZON)
+			{
+				if (!SlotIsLogical(s))
+					goto invalidation_marked;

I am not sure if this logic is correct. Even if the slot would not be
invalidated due to RS_INVAL_HORIZON, we should continue to check other causes.

Besides, instead of using a goto, I personally prefer to move all these codes
into a separate function which would return a single invalidation cause.

4.
-	if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
+	if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED | RS_INVAL_IDLE_TIMEOUT,
 										   _logSegNo, InvalidOid,
 										   InvalidTransactionId))

I think this change could trigger an unnecessary WAL position re-calculation when
slots are invalidated only due to RS_INVAL_IDLE_TIMEOUT. But since it would not be
a frequent operation so I am OK to leave it unless we have better ideas.

Best Regards,
Hou zj

  388. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-10T06:03:02Z

    Hi Nisha,

Some review comments for v72-0001.

======
GENERAL

My preference was to just keep the enum as per v70 for the *actual*
cause, and introduce a separate set of bit flags for *possible* causes
to be checked. This creates a clear code separation between the actual
and possible. It also eliminates the need to jump through hoops just
to map a cause to its name.

You wrote:

> OTOH, keeping the enums as they are in v70, and defining new macros
for the very similar purpose could add unnecessary complexity to code
management.

Since both the enum and the bit flags would be defined in slot.h
adjacent to each other I don't foresee much complexity. I concede, a
dev might write code and accidentally muddle the enum instead of the
flag  or vice versa but that's an example of sloppiness, not
complexity. Certainly there would be fewer necessary changes than what
are in the v72 patch due to all the cause/causename mappings. For
example,

slot.h - Now, introduces a NEW typedef SlotInvalidationCauseMap
slot.h - Now, need extern for NEW function GetSlotInvalidationCauseName

slot.c - Now, needed minor rewrite of GetSlotInvalidationCause instead
of leaving it as-is
slot.c - Now, needs a whole NEW looping function
GetSlotInvalidationCauseName instead of direct array index.

Several place now must call to the GetSlotInvalidationCauseName where
previously a simple direct array lookup was done
slot.c - NEW call in ReplicationSlotAcquire
slotfuncs.c - NEW call in pg_get_replication_slots

~

FWIW, I've attached a topup patch using my idea just to see what it
might look like. The result was 20 lines less code.

Anyway, YMMV.

======

Other review comments follow ...

src/backend/replication/slot.c

InvalidatePossiblyObsoleteSlot:

1.
Having all those 'gotos' seems like something best avoided. Can you
try removing them to see if it improves this function? IIUC you maybe
can try rid all of them using  logic like:

- assign invalidation_cause = NONE outside the loop
- loop until invalidation_cause != NONE
- include 'invalidation_cause == NONE' condition with all the bit flag checks
- reassign invalidation_cause = NONE in the racy place where you want
to continue the loop.

and instead just keep looping and checking while the
'invalidation_cause' remains NONE.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  389. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-10T06:09:45Z

    On Sat, Feb 8, 2025 at 12:28 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> 3.
>
> +                       if (cause & RS_INVAL_HORIZON)
> +                       {
> +                               if (!SlotIsLogical(s))
> +                                       goto invalidation_marked;
>
> I am not sure if this logic is correct. Even if the slot would not be
> invalidated due to RS_INVAL_HORIZON, we should continue to check other causes.
>

Isn't this comment apply to even the next condition (if (dboid !=
InvalidOid && dboid != s->data.database))? We need to probably
continue to check other invalidation causes unless one is set.

> Besides, instead of using a goto, I personally prefer to move all these codes
> into a separate function which would return a single invalidation cause.
>

Instead of using goto label (invalidation_marked:), won't it be better
if we use a boolean invalidation_marked and convert all if's to if ..
else if .. else cases?

> 4.
> -       if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
> +       if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED | RS_INVAL_IDLE_TIMEOUT,
>                                                                                    _logSegNo, InvalidOid,
>                                                                                    InvalidTransactionId))
>
> I think this change could trigger an unnecessary WAL position re-calculation when
> slots are invalidated only due to RS_INVAL_IDLE_TIMEOUT.
>

Why is that unnecessary? If some slots got invalidated due to timeout,
we don't want to retain the WAL corresponding to them.


-- 
With Regards,
Amit Kapila.

  390. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-10T06:10:14Z

    On Fri, Feb 7, 2025 at 4:53 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Feb 7, 2025 at 8:00 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > ======
> > src/backend/access/transam/xlog.c
> >
> > 1.
> >   XLByteToSeg(RedoRecPtr, _logSegNo, wal_segment_size);
> >   KeepLogSeg(recptr, &_logSegNo);
> > - if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
> > + if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED |
> > RS_INVAL_IDLE_TIMEOUT,
> >      _logSegNo, InvalidOid,
> >      InvalidTransactionId))
> >   {
> > @@ -7792,7 +7792,7 @@ CreateRestartPoint(int flags)
> >   replayPtr = GetXLogReplayRecPtr(&replayTLI);
> >   endptr = (receivePtr < replayPtr) ? replayPtr : receivePtr;
> >   KeepLogSeg(endptr, &_logSegNo);
> > - if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
> > + if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED |
> > RS_INVAL_IDLE_TIMEOUT,
> >      _logSegNo, InvalidOid,
> >      InvalidTransactionId))
> >
> > It seems fundamentally strange to me to assign multiple simultaneous
> > causes like this. IMO you can't invalidate something that is invalid
> > already. I gues v71 was an attempt to implement Amit's:
> >
>
> The idea is to invalidate the slot either due to WAL_REMOVED or
> IDLE_TIMEOUT in one go during the checkpoint instead of taking
> multiple passes over the slots during the checkpoint. Feel free to
> suggest if you can think of a better way to implement it.
>

Hi Amit,

My preference already suggested was to have a separation between the
concepts of *actual* causes (e.g. discrete enum values like in v70)
and *possible* causes to be checked (using #defines for bit flags).

My v72-0001 review [1] includes a top-up patch to show what doing it
this way might look like.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPupn_S0mrM2zB%2BFwAbPqVak7jwSjRhU3WyA18QC1HU__g%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  391. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-10T09:45:07Z

    On Mon, Feb 10, 2025 at 11:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Some review comments for v72-0001.
>
> ======
> GENERAL
>
> My preference was to just keep the enum as per v70 for the *actual*
> cause, and introduce a separate set of bit flags for *possible* causes
> to be checked. This creates a clear code separation between the actual
> and possible. It also eliminates the need to jump through hoops just
> to map a cause to its name.
>
> You wrote:
>
> > OTOH, keeping the enums as they are in v70, and defining new macros
> for the very similar purpose could add unnecessary complexity to code
> management.
>
> Since both the enum and the bit flags would be defined in slot.h
> adjacent to each other I don't foresee much complexity. I concede, a
> dev might write code and accidentally muddle the enum instead of the
> flag  or vice versa but that's an example of sloppiness, not
> complexity. Certainly there would be fewer necessary changes than what
> are in the v72 patch due to all the cause/causename mappings. For
> example,
>
> slot.h - Now, introduces a NEW typedef SlotInvalidationCauseMap
> slot.h - Now, need extern for NEW function GetSlotInvalidationCauseName
>
> slot.c - Now, needed minor rewrite of GetSlotInvalidationCause instead
> of leaving it as-is
> slot.c - Now, needs a whole NEW looping function
> GetSlotInvalidationCauseName instead of direct array index.
>
> Several place now must call to the GetSlotInvalidationCauseName where
> previously a simple direct array lookup was done
> slot.c - NEW call in ReplicationSlotAcquire
> slotfuncs.c - NEW call in pg_get_replication_slots
>
> ~
>
> FWIW, I've attached a topup patch using my idea just to see what it
> might look like. The result was 20 lines less code.
>

I don't like the idea of maintaining the same information in two
different ways (as enum and bit flags). We already have a few cases of
defining bit flags as part of enums like ScanOptions and relopt_kind,
so I feel following that model would be a better approach.

-- 
With Regards,
Amit Kapila.

  392. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2025-02-10T09:56:49Z

    On Monday, February 10, 2025 2:10 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> 
> On Sat, Feb 8, 2025 at 12:28 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
> wrote:
> >
> > 3.
> >
> > +                       if (cause & RS_INVAL_HORIZON)
> > +                       {
> > +                               if (!SlotIsLogical(s))
> > +                                       goto invalidation_marked;
> >
> > I am not sure if this logic is correct. Even if the slot would not be
> > invalidated due to RS_INVAL_HORIZON, we should continue to check other
> causes.
> >
> 
> Isn't this comment apply to even the next condition (if (dboid != InvalidOid &&
> dboid != s->data.database))? We need to probably continue to check other
> invalidation causes unless one is set.

Yes, both places need to be fixed.

> 
> > Besides, instead of using a goto, I personally prefer to move all
> > these codes into a separate function which would return a single invalidation
> cause.
> >
> 
> Instead of using goto label (invalidation_marked:), won't it be better if we use a
> boolean invalidation_marked and convert all if's to if ..
> else if .. else cases?

Yes, I think that would be better.

> 
> > 4.
> > -       if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED,
> > +       if (InvalidateObsoleteReplicationSlots(RS_INVAL_WAL_REMOVED
> |
> > + RS_INVAL_IDLE_TIMEOUT,
> >
> _logSegNo, InvalidOid,
> >
> > InvalidTransactionId))
> >
> > I think this change could trigger an unnecessary WAL position
> > re-calculation when slots are invalidated only due to
> RS_INVAL_IDLE_TIMEOUT.
> >
> 
> Why is that unnecessary? If some slots got invalidated due to timeout, we don't
> want to retain the WAL corresponding to them.

Sorry, I mistakenly thought that the slot only protected dead tuples.
Please disregard this comment.

Best Regards,
Hou zj

  393. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-10T12:03:07Z

    On Sat, Feb 8, 2025 at 12:28 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Friday, February 7, 2025 9:06 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Attached v72 patches, addressed the above comments as well as Vignesh's
> > comments in [2].
> >  - There are no new changes in patch-002.
>
> Thanks for updating the patch, I have few review comments:
>
> 1.
> > InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause cause,
>
> I think the type of first parameter 'cause' is not appropriate anymore since
> it's now a bitmap flag instead of an enum.
>

Changed the type to 'int' and updated the name of 'cause' in both
InvalidateObsoleteReplicationSlots() and
InvalidatePossiblyObsoleteSlot(), as both now use the bitmap flag.

> 2.
> > -StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
> > -                              "array length mismatch");
> > +#define      RS_INVAL_MAX_CAUSES (sizeof(InvalidationCauses) / sizeof(InvalidationCauses[0]))
>
> I'd like to confirm if the current value of the RS_INVAL_MAX_CAUSES is correct.
> Previously, the value is arrary_length - 1, while now it seems equal to the
> arrary_length.
>
> And ISTM we could directly call lengthof() here.
>

Done.

> 3.
>
> +                       if (cause & RS_INVAL_HORIZON)
> +                       {
> +                               if (!SlotIsLogical(s))
> +                                       goto invalidation_marked;
>
> I am not sure if this logic is correct. Even if the slot would not be
> invalidated due to RS_INVAL_HORIZON, we should continue to check other causes.
>

Used goto here since we do not expect RS_INVAL_HORIZON to be combined
with any other "cause" and to keep the pgHead behavior.
However, with the bitflag approach, the code should be future-safe, so
replacing goto in v73 should handle this now.

> Besides, instead of using a goto, I personally prefer to move all these codes
> into a separate function which would return a single invalidation cause.
>

Done.

~~~~
Here are the v73 patches incorporating the comments above and the
subsequent comments from [1].
 - patch 002 is rebased on 001 with no new changes.

[1] https://www.postgresql.org/message-id/CAA4eK1K%2BAMtGfD3WRK_ivdAeS-CBOUBKJbr-6ku175P1x%3Dwk4g%40mail.gmail.com

--
Thanks,
Nisha

  394. Re: Introduce XID age and inactive timeout based replication slot invalidation

    vignesh C <vignesh21@gmail.com> — 2025-02-10T12:42:12Z

    On Mon, 10 Feb 2025 at 17:33, Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Here are the v73 patches incorporating the comments above and the
> subsequent comments from [1].
>  - patch 002 is rebased on 001 with no new changes.

Few comments:
1) For some reason SlotInvalidationCauses was with PGDLLIMPORT, this
is removed now. This is required if it needs to be accessible by
loaded modules. Is there any impact or is it ok?
-extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
+typedef struct SlotInvalidationCauseMap
+{
+       int                     cause;
+       const char *cause_name;
+}                      SlotInvalidationCauseMap;

2) The new structure should be added to typedefs.list:
+typedef struct SlotInvalidationCauseMap
+{
+       int                     cause;
+       const char *cause_name;
+}                      SlotInvalidationCauseMap;

3) After adding you can run pgindent on slot.h to indent the following code:
+typedef struct SlotInvalidationCauseMap
+{
+       int                     cause;
+       const char *cause_name;
+}                      SlotInvalidationCauseMap;

Regards,
Vignesh

  395. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-10T13:27:25Z

    On Mon, Feb 10, 2025 at 6:12 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 10 Feb 2025 at 17:33, Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Here are the v73 patches incorporating the comments above and the
> > subsequent comments from [1].
> >  - patch 002 is rebased on 001 with no new changes.
>
> Few comments:
> 1) For some reason SlotInvalidationCauses was with PGDLLIMPORT, this
> is removed now. This is required if it needs to be accessible by
> loaded modules. Is there any impact or is it ok?
> -extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
> +typedef struct SlotInvalidationCauseMap
> +{
> +       int                     cause;
> +       const char *cause_name;
> +}                      SlotInvalidationCauseMap;
>
> 2) The new structure should be added to typedefs.list:
> +typedef struct SlotInvalidationCauseMap
> +{
> +       int                     cause;
> +       const char *cause_name;
> +}                      SlotInvalidationCauseMap;
>
> 3) After adding you can run pgindent on slot.h to indent the following code:
> +typedef struct SlotInvalidationCauseMap
> +{
> +       int                     cause;
> +       const char *cause_name;
> +}                      SlotInvalidationCauseMap;
>

Addressed above comments, please find the attached v74 patches.
Also, corrected the type of parameter "possible_causes" to 'uint32' in
InvalidateObsoleteReplicationSlots() and
InvalidatePossiblyObsoleteSlot().

--
Thanks,
Nisha

  396. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-11T03:19:20Z

    Hi Nisha.

Some review comments about v74-0001

======
src/backend/replication/slot.c

1.
 /* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
-
-StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
- "array length mismatch");
+#define RS_INVAL_MAX_CAUSES (lengthof(InvalidationCauses)-1)

The static assert was here to protect against dev mistakes in keeping
the lookup table up-to-date with the enum of slot.h. So it's not a
good idea to remove it...

IMO the RS_INVAL_MAX_CAUSES should be relocated to slot.h where the
enum is defined and where the devs know exactly how many invalidation
types there are. Then this static assert can be put back in to do its
job of ensuring the integrity properly again for this lookup table.

~~~

InvalidatePossiblyObsoleteSlot:

2.
+ if (possible_causes & RS_INVAL_IDLE_TIMEOUT)
+ {
+ /*
+ * Assign the current time here to avoid system call overhead
+ * while holding the spinlock in subsequent code.
+ */
+ now = GetCurrentTimestamp();
+ }
+

I felt that any minuscule benefit gained from having this conditional
'now' assignment is outweighed by the subsequent confusion/doubt
caused by passing around a 'now' to other functions where it may or
may not still be zero depending on different processing. IMO we should
just remove all doubts and always assign it so that "now always means
now".

~~~

3.
+ if (possible_causes & RS_INVAL_IDLE_TIMEOUT)

IMO every bitwise check like this should also be checking
(invalidation_cause == RS_INVAL_NONE). Maybe you omitted it here
because this is the first but I think it will be safer anyhow in case
the code gets shuffled around in future and the extra condition gets
overlooked.

~~~

4.
+ if (possible_causes & RS_INVAL_WAL_REMOVED)
+ {
+ if (initial_restart_lsn != InvalidXLogRecPtr &&
+ initial_restart_lsn < oldestLSN)
+ invalidation_cause = RS_INVAL_WAL_REMOVED;
+ }
+ if (invalidation_cause == RS_INVAL_NONE &&
+ (possible_causes & RS_INVAL_HORIZON))
+ {
+ if (SlotIsLogical(s) &&
+ /* invalid DB oid signals a shared relation */
+ (dboid == InvalidOid || dboid == s->data.database) &&
+ TransactionIdIsValid(initial_effective_xmin) &&
+ TransactionIdPrecedesOrEquals(initial_effective_xmin,
+   snapshotConflictHorizon))
+ invalidation_cause = RS_INVAL_HORIZON;
+ else if (TransactionIdIsValid(initial_catalog_effective_xmin) &&
+ TransactionIdPrecedesOrEquals(initial_catalog_effective_xmin,
+    snapshotConflictHorizon))
+ invalidation_cause = RS_INVAL_HORIZON;
+ }
+ if (invalidation_cause == RS_INVAL_NONE &&

I suggest adding blank lines where the bit conditions are to delineate
each of the different invalidation checks.

~~~

InvalidateObsoleteReplicationSlots:

5
- Assert(cause != RS_INVAL_HORIZON ||
TransactionIdIsValid(snapshotConflictHorizon));
- Assert(cause != RS_INVAL_WAL_REMOVED || oldestSegno > 0);
- Assert(cause != RS_INVAL_NONE);
+ Assert(!(possible_causes & RS_INVAL_HORIZON) ||
TransactionIdIsValid(snapshotConflictHorizon));
+ Assert(!(possible_causes & RS_INVAL_WAL_REMOVED) || oldestSegno > 0);
+ Assert(!(possible_causes & RS_INVAL_NONE));

AFAIK the  RS_INVAL_NONE is defined as 0, so doing bit-wise operations
on 0 seems bogus.

Do you mean just Assert(possible_causes != NONE);

======
src/include/replication/slot.h

6.
-extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
+typedef struct SlotInvalidationCauseMap
+{
+ int cause;
+ const char *cause_name;
+} SlotInvalidationCauseMap;
+
+extern PGDLLIMPORT const SlotInvalidationCauseMap InvalidationCauses[];

6a.
AFAIK. there is no longer any external access to this lookup table so
why do you need this extern. Similarly, why is this typedef even here
instead of declared in the slot.c module.

~

6b.
Why is the field 'cause' declared as int instead of
ReplicationSlotInvalidationCause?


======

Please the attached top-up patch as a code example of some of my
suggestions above -- in particular the relocating of
RS_INVAL_MAX_CAUSES and the typedef, and the reinstating of the static
insert for the lookup table.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  397. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2025-02-11T06:12:31Z

    On Monday, February 10, 2025 8:03 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> 
> On Sat, Feb 8, 2025 at 12:28 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
> wrote:
> >
> 
> > 3.
> >
> > +                       if (cause & RS_INVAL_HORIZON)
> > +                       {
> > +                               if (!SlotIsLogical(s))
> > +                                       goto invalidation_marked;
> >
> > I am not sure if this logic is correct. Even if the slot would not be
> > invalidated due to RS_INVAL_HORIZON, we should continue to check other
> causes.
> >
> 
> Used goto here since we do not expect RS_INVAL_HORIZON to be combined
> with any other "cause" and to keep the pgHead behavior.
> However, with the bitflag approach, the code should be future-safe, so
> replacing goto in v73 should handle this now.

I think the following logic needs some adjustments.

+			if (invalidation_cause == RS_INVAL_NONE &&
+				(possible_causes & RS_INVAL_HORIZON))
+			{
+				if (SlotIsLogical(s) &&
+				/* invalid DB oid signals a shared relation */
+					(dboid == InvalidOid || dboid == s->data.database) &&
+					TransactionIdIsValid(initial_effective_xmin) &&
+					TransactionIdPrecedesOrEquals(initial_effective_xmin,
+												  snapshotConflictHorizon))
+					invalidation_cause = RS_INVAL_HORIZON;
+				else if (TransactionIdIsValid(initial_catalog_effective_xmin) &&
+						 TransactionIdPrecedesOrEquals(initial_catalog_effective_xmin,
+													   snapshotConflictHorizon))
+					invalidation_cause = RS_INVAL_HORIZON;
+			}

I think we assign RS_INVAL_HORIZON to invalidation_cause only when the slot is
logical and the dboid is valid, but it is not guaranteed in the second if
condition ("else if (TransactionIdIsValid(initial_catalog_effective_xmin)").

Here is a top-up patch to fix this.

Best Regards,
Hou zj

  398. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-11T06:17:34Z

    On Tue, Feb 11, 2025 at 8:49 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
>
> InvalidatePossiblyObsoleteSlot:
>
> 2.
> + if (possible_causes & RS_INVAL_IDLE_TIMEOUT)
> + {
> + /*
> + * Assign the current time here to avoid system call overhead
> + * while holding the spinlock in subsequent code.
> + */
> + now = GetCurrentTimestamp();
> + }
> +
>
> I felt that any minuscule benefit gained from having this conditional
> 'now' assignment is outweighed by the subsequent confusion/doubt
> caused by passing around a 'now' to other functions where it may or
> may not still be zero depending on different processing. IMO we should
> just remove all doubts and always assign it so that "now always means
> now".
>

I think we shouldn't pass now to another function, but rather do all
required computations in the caller (probably in a separate inline
function). And keep the above code as is.

-- 
With Regards,
Amit Kapila.

  399. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-11T13:36:05Z

    On Tue, Feb 11, 2025 at 8:49 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Nisha.
>
> Some review comments about v74-0001
>
> ======
> src/backend/replication/slot.c
>
> 1.
>  /* Maximum number of invalidation causes */
> -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
> -
> -StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
> - "array length mismatch");
> +#define RS_INVAL_MAX_CAUSES (lengthof(InvalidationCauses)-1)
>
> The static assert was here to protect against dev mistakes in keeping
> the lookup table up-to-date with the enum of slot.h. So it's not a
> good idea to remove it...
>
> IMO the RS_INVAL_MAX_CAUSES should be relocated to slot.h where the
> enum is defined and where the devs know exactly how many invalidation
> types there are. Then this static assert can be put back in to do its
> job of ensuring the integrity properly again for this lookup table.
>

How about keeping RS_INVAL_MAX_CAUSES dynamic in slot.c (as it was)
and updating the static assert to ensure the lookup table stays
up-to-date with the enums?
The change has been implemented in v75.

> ~~~
>
> ======

Here are v75 patches addressing comments in [1], [2] and [3].
 - A new function, "EvaluateSlotInvalidationCause()", has been
introduced to separate the invalidation_cause evaluation logic from
InvalidatePossiblyObsoleteSlot().
 - Also, another new inline function "CalculateTimeDuration()" added
as suggested in [3].

[1] https://www.postgresql.org/message-id/CAHut%2BPvsvHWoiEkGTP4NfVNsADsy-Jan3Dvp%2B_GW3gmPDHf5Qw%40mail.gmail.com
[2] https://www.postgresql.org/message-id/OS0PR01MB57163889BE5F9F30DD3318A394FD2%40OS0PR01MB5716.jpnprd01.prod.outlook.com
[3] https://www.postgresql.org/message-id/CAA4eK1LuvXa6sVj3xuLoe2X%3D0xjbJXrnJePbpXQZaTMws8pZqg%40mail.gmail.com

--
Thanks,
Nisha

  400. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-11T13:36:49Z

    On Tue, Feb 11, 2025 at 11:42 AM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Monday, February 10, 2025 8:03 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > On Sat, Feb 8, 2025 at 12:28 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
> > wrote:
> > >
> >
> > > 3.
> > >
> > > +                       if (cause & RS_INVAL_HORIZON)
> > > +                       {
> > > +                               if (!SlotIsLogical(s))
> > > +                                       goto invalidation_marked;
> > >
> > > I am not sure if this logic is correct. Even if the slot would not be
> > > invalidated due to RS_INVAL_HORIZON, we should continue to check other
> > causes.
> > >
> >
> > Used goto here since we do not expect RS_INVAL_HORIZON to be combined
> > with any other "cause" and to keep the pgHead behavior.
> > However, with the bitflag approach, the code should be future-safe, so
> > replacing goto in v73 should handle this now.
>
> I think the following logic needs some adjustments.
>
> +                       if (invalidation_cause == RS_INVAL_NONE &&
> +                               (possible_causes & RS_INVAL_HORIZON))
> +                       {
> +                               if (SlotIsLogical(s) &&
> +                               /* invalid DB oid signals a shared relation */
> +                                       (dboid == InvalidOid || dboid == s->data.database) &&
> +                                       TransactionIdIsValid(initial_effective_xmin) &&
> +                                       TransactionIdPrecedesOrEquals(initial_effective_xmin,
> +                                                                                                 snapshotConflictHorizon))
> +                                       invalidation_cause = RS_INVAL_HORIZON;
> +                               else if (TransactionIdIsValid(initial_catalog_effective_xmin) &&
> +                                                TransactionIdPrecedesOrEquals(initial_catalog_effective_xmin,
> +                                                                                                          snapshotConflictHorizon))
> +                                       invalidation_cause = RS_INVAL_HORIZON;
> +                       }
>
> I think we assign RS_INVAL_HORIZON to invalidation_cause only when the slot is
> logical and the dboid is valid, but it is not guaranteed in the second if
> condition ("else if (TransactionIdIsValid(initial_catalog_effective_xmin)").
>
> Here is a top-up patch to fix this.

Thank you for reviewing and providing the fix! v75 addresses this bug
with a slightly different approach after introducing the new function
EvaluateSlotInvalidationCause().

--
Thanks,
Nisha

  401. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2025-02-11T14:22:49Z

    Hello,

I find this proposed patch a bit strange and I feel it needs more
explanation.

When this thread started, Bharath justified his patches saying that a
slot that's inactive for a very long time could be problematic because
of XID wraparound.  Fine, that sounds a reasonable feature.  If you
wanted to invalidate slots whose xmins were too old, I would support
that.  He submitted that as his 0004 patch then.

However, he also chose to submit 0003 with invalidation based on a
timeout.  This is far less convincing a feature to me.  The
justification for the time out seems to be that ... it's difficult to
have a one-size-fits-all value because size of disks vary. (???)
Or something like that.  Really?  I mean -- yes, this will prevent
problems in toy databases when run in developer's laptops.  It will not
prevent any problems in production databases.  Do we really want a
setting that is only useful for toy situations rather than production?


Anyway, the thread is way too long, but after some initial pieces were
committed, Nisha took over and submitting patches derived from Bharath's
0003, and at some point the initial 0004 was dropped.  But 0004 was the
more useful one, I thought, so what's going on?

I'm baffled.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
Officer Krupke, what are we to do?
Gee, officer Krupke, Krup you! (West Side Story, "Gee, Officer Krupke")

  402. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2025-02-11T16:09:26Z

    On Tue, Feb 11, 2025 at 03:22:49PM +0100, Álvaro Herrera wrote:
> I find this proposed patch a bit strange and I feel it needs more
> explanation.
> 
> When this thread started, Bharath justified his patches saying that a
> slot that's inactive for a very long time could be problematic because
> of XID wraparound.  Fine, that sounds a reasonable feature.  If you
> wanted to invalidate slots whose xmins were too old, I would support
> that.  He submitted that as his 0004 patch then.
> 
> However, he also chose to submit 0003 with invalidation based on a
> timeout.  This is far less convincing a feature to me.  The
> justification for the time out seems to be that ... it's difficult to
> have a one-size-fits-all value because size of disks vary. (???)
> Or something like that.  Really?  I mean -- yes, this will prevent
> problems in toy databases when run in developer's laptops.  It will not
> prevent any problems in production databases.  Do we really want a
> setting that is only useful for toy situations rather than production?
> 
> 
> Anyway, the thread is way too long, but after some initial pieces were
> committed, Nisha took over and submitting patches derived from Bharath's
> 0003, and at some point the initial 0004 was dropped.  But 0004 was the
> more useful one, I thought, so what's going on?
> 
> I'm baffled.

I agree, and I am also baffled because I think this discussion has happened
at least once already on this thread.  I still feel like the XID-based
parameter makes more sense.  For replication slots, two primary concerns
are 1) storage, for which we have max_slot_wal_keep_size and 2) XID
wraparound, for which we don't really have anything today.  A timeout might
be useful in some contexts, but if the goal is to prevent wraparound, why
not target that directly?

-- 
nathan

  403. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-11T21:23:37Z

    On Wed, Feb 12, 2025 at 12:36 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> On Tue, Feb 11, 2025 at 8:49 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Nisha.
> >
> > Some review comments about v74-0001
> >
> > ======
> > src/backend/replication/slot.c
> >
> > 1.
> >  /* Maximum number of invalidation causes */
> > -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
> > -
> > -StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
> > - "array length mismatch");
> > +#define RS_INVAL_MAX_CAUSES (lengthof(InvalidationCauses)-1)
> >
> > The static assert was here to protect against dev mistakes in keeping
> > the lookup table up-to-date with the enum of slot.h. So it's not a
> > good idea to remove it...
> >
> > IMO the RS_INVAL_MAX_CAUSES should be relocated to slot.h where the
> > enum is defined and where the devs know exactly how many invalidation
> > types there are. Then this static assert can be put back in to do its
> > job of ensuring the integrity properly again for this lookup table.
> >
>
> How about keeping RS_INVAL_MAX_CAUSES dynamic in slot.c (as it was)
> and updating the static assert to ensure the lookup table stays
> up-to-date with the enums?
> The change has been implemented in v75.
>

Latest v75-001 patch code looks like:

+static const SlotInvalidationCauseMap InvalidationCauses[] = {
+ {RS_INVAL_NONE, "none"},
+ {RS_INVAL_WAL_REMOVED, "wal_removed"},
+ {RS_INVAL_HORIZON, "rows_removed"},
+ {RS_INVAL_WAL_LEVEL, "wal_level_insufficient"},
+ {RS_INVAL_IDLE_TIMEOUT, "idle_timeout"},
 };

 /* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES (lengthof(InvalidationCauses)-1)

-StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
+/*
+ * Ensure that the lookup table is up-to-date with the enums defined in
+ * ReplicationSlotInvalidationCause. Shifting 1 left by
+ * (RS_INVAL_MAX_CAUSES - 1) should give the highest defined value in
+ * the enum.
+ */
+StaticAssertDecl(RS_INVAL_IDLE_TIMEOUT == (1 << (RS_INVAL_MAX_CAUSES - 1)),
  "array length mismatch");

Where:
1.  RS_INVAL_MAX_CAUSES is based on the length of lookup table so it is 4
2.  the StaticAssert then confirms that the enum RS_INVAL_IDLE_TIMEOUT
 is the 4th enum entry

AFAICT that is not useful. The purpose of the static assert is (like
your comment says) to "Ensure that the lookup table is up-to-date with
the enums". Imagine if I added another (5th cause) enum called
RS_INVAL_BANANA but accidentally overlook updating the lookup table.
The code above isn't going to detect that -- the lookup table length
is still 4 (instead of 5) but RS_INVAL_IDLE_TIMEOUT is still the 4th
enum so the assert is happy. Hence my original suggestion to define
RS_INVAL_MAX_CAUSES adjacent to the enum in slot.h.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  404. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-12T03:55:53Z

    On Tue, Feb 11, 2025 at 9:39 PM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Feb 11, 2025 at 03:22:49PM +0100, Álvaro Herrera wrote:
> > I find this proposed patch a bit strange and I feel it needs more
> > explanation.
> >
> > When this thread started, Bharath justified his patches saying that a
> > slot that's inactive for a very long time could be problematic because
> > of XID wraparound.  Fine, that sounds a reasonable feature.  If you
> > wanted to invalidate slots whose xmins were too old, I would support
> > that.  He submitted that as his 0004 patch then.
> >
> > However, he also chose to submit 0003 with invalidation based on a
> > timeout.  This is far less convincing a feature to me.  The
> > justification for the time out seems to be that ... it's difficult to
> > have a one-size-fits-all value because size of disks vary. (???)
> > Or something like that.  Really?  I mean -- yes, this will prevent
> > problems in toy databases when run in developer's laptops.  It will not
> > prevent any problems in production databases.  Do we really want a
> > setting that is only useful for toy situations rather than production?
> >
> >
...
> >
> > I'm baffled.
>
> I agree, and I am also baffled because I think this discussion has happened
> at least once already on this thread.
>

Yes, we previously discussed this topic and Robert seems to prefer a
time-based parameter for invalidating the slot (1)(2) as it is easier
to reason in terms of time. The other points discussed previously were
that there are tools that create a lot of slots and sometimes forget
to clean up slots. Bharath has seen this in production and we now have
the tool pg_createsubscriber that creates a slot-per-database, so if
for some reason, such slots are not cleaned on the tool's exit, such a
parameter could save the cluster. See (3)(4).

Also, we previously didn't have a good experience with XID-based
threshold parameters like vacuum_defer_cleanup_age as mentioned by
Robert (1). AFAICU from the previous discussion we need a time-based
parameter and we didn't rule out xid_age based parameter as another
parameter.

(1) - https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
(2) - https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
(3) - https://www.postgresql.org/message-id/CALj2ACVFV%3DyUa3DXXfJLOtJxUM8qzC_mEECMJ2iekDGPeQLkTw%40mail.gmail.com
(4) - https://www.postgresql.org/message-id/CAA4eK1L3awyzWMuymLJUm8SoFEQe%3DDa9KUwCcAfC31RNJ1xdJA%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  405. RE: Introduce XID age and inactive timeout based replication slot invalidation

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2025-02-12T07:46:22Z

    On Wednesday, February 12, 2025 11:56 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> 
> On Tue, Feb 11, 2025 at 9:39 PM Nathan Bossart
> <nathandbossart@gmail.com> wrote:
> >
> > On Tue, Feb 11, 2025 at 03:22:49PM +0100, Álvaro Herrera wrote:
> > > I find this proposed patch a bit strange and I feel it needs more
> > > explanation.
> > >
> > > When this thread started, Bharath justified his patches saying that
> > > a slot that's inactive for a very long time could be problematic
> > > because of XID wraparound.  Fine, that sounds a reasonable feature.
> > > If you wanted to invalidate slots whose xmins were too old, I would
> > > support that.  He submitted that as his 0004 patch then.
> > >
> > > However, he also chose to submit 0003 with invalidation based on a
> > > timeout.  This is far less convincing a feature to me.  The
> > > justification for the time out seems to be that ... it's difficult
> > > to have a one-size-fits-all value because size of disks vary. (???)
> > > Or something like that.  Really?  I mean -- yes, this will prevent
> > > problems in toy databases when run in developer's laptops.  It will
> > > not prevent any problems in production databases.  Do we really want
> > > a setting that is only useful for toy situations rather than production?
> > >
> > >
> ...
> > >
> > > I'm baffled.
> >
> > I agree, and I am also baffled because I think this discussion has
> > happened at least once already on this thread.
> >
> 
> Yes, we previously discussed this topic and Robert seems to prefer a
> time-based parameter for invalidating the slot (1)(2) as it is easier to reason in
> terms of time. The other points discussed previously were that there are tools
> that create a lot of slots and sometimes forget to clean up slots. Bharath has
> seen this in production and we now have the tool pg_createsubscriber that
> creates a slot-per-database, so if for some reason, such slots are not cleaned
> on the tool's exit, such a parameter could save the cluster. See (3)(4).
> 
> Also, we previously didn't have a good experience with XID-based threshold
> parameters like vacuum_defer_cleanup_age as mentioned by Robert (1).
> AFAICU from the previous discussion we need a time-based parameter and we
> didn't rule out xid_age based parameter as another parameter.

Yeah, I think the primary purpose of this time-based option is to invalidate dormant
replication slots that have been inactive for a long period, in which case the
slots are no longer useful.
 
Such slots can remain if a subscriber is down due to a system error or
inaccessible because of network issues. If this situation persists, it might be
more practical to recreate the subscriber rather than attempt to recover the
node and wait for it to catch up, which could be time-consuming.

Parameters like max_slot_wal_keep_size and max_slot_xid_id_age do not
differentiate between active and inactive replication slots. Some customers I
met are hesitant about using these settings, as they can sometimes invalidate
a slot unnecessarily and break the replication.


> (1) -
> https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzx
> AdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
> (2) -
> https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2
> S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
> (3) -
> https://www.postgresql.org/message-id/CALj2ACVFV%3DyUa3DXXfJLOtJxU
> M8qzC_mEECMJ2iekDGPeQLkTw%40mail.gmail.com
> (4) -
> https://www.postgresql.org/message-id/CAA4eK1L3awyzWMuymLJUm8SoF
> EQe%3DDa9KUwCcAfC31RNJ1xdJA%40mail.gmail.com

Best Regards,
Hou zj

  406. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-14T12:00:16Z

    Please find the updated v78 patches after a few off-list review rounds.

Here is a summary of changes in v78:
patch-001:
- Fixed bugs reported by Hou-san and Peter in [1] and [2].
- Fixed a race condition reported by Hou-san off-list, which could
lead to an assert failure.
This failure happens when the checkpointer sets the invalidation cause
to idle_timeout on the first attempt, but if it later finds another
process's pid active for the slot, it retries after terminating that
process. By then, inactive_since may have been updated, and it
determines the invalidation_cause as RS_INVAL_NONE and below assert
fails:

```
Assert(!(invalidation_cause_prev != RS_INVAL_NONE && terminated &&
invalidation_cause_prev != invalidation_cause));
```

- Moved the slot's idle_time calculation to the caller of
ReportSlotInvalidation().
- Improved the patch commit message for better clarity.

patch-002:
- Fixed a bug reported by Kuroda-san - "check_extension() must be done
before the CREATE EXTENSION".
- Addressed a few other comments by Peter and Kuroda-san to optimize
code and improve comments.

[1] https://www.postgresql.org/message-id/CABdArM7eeejXEgd6t4wtBiK%3DaWc%2B%2Bgt1__WwAWm-Y_5xMVskWg%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPtnWyOMvxb6mZHWFxqD-NdHuYL8Zp%3D-QasAQ3VvxauiMA%40mail.gmail.com

--
Thanks,
Nisha

  407. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Peter Smith <smithpb2250@gmail.com> — 2025-02-16T22:18:28Z

    Some review comments for v78-0001.

======
src/backend/replication/slot.c

ReportSlotInvalidation:

1.
+ int minutes;
+ int secs;

The variables 'minutes' and 'seconds' are only used by case
RS_INVAL_IDLE_TIMEOUT, so I think it would be better to make a new
code block for that case where you can declare and initialise these in
one go at that scope.

~~~

DetermineSlotInvalidationCause:

2.
+ if (SlotIsLogical(s) &&
+ /* invalid DB oid signals a shared relation */
+ (dboid == InvalidOid || dboid == s->data.database))
+ {

The comment placement in the master code was ok because then there
were different statements, but now in this patch multiple conditions
are combined, and this comment seems strangely placed.

~~~

GetSlotInvalidationCause:

3.
I understand your argument "let's not change anything unless
absolutely necessary for our patch", but in this case since half the
function is changing anyway it seems a missed opportunity to not
simplify the rest of the code "in passing" to make it consistent with
the newly added partner function GetSlotInvalidationCauseName. My
question is "if not now, then when?", because I expect a future patch
to do this might be rejected as being too trivial, so by not changing
now probably these functions are doomed to be inconsistent forever.
Anyway it is just my opinion -- leave it as-is if you wish.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  408. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-17T02:27:22Z

    On Wed, Feb 12, 2025 at 1:16 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Wednesday, February 12, 2025 11:56 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Feb 11, 2025 at 9:39 PM Nathan Bossart
> > <nathandbossart@gmail.com> wrote:
> > >
> > > On Tue, Feb 11, 2025 at 03:22:49PM +0100, Álvaro Herrera wrote:
> > > > I find this proposed patch a bit strange and I feel it needs more
> > > > explanation.
> > > >
> > > > When this thread started, Bharath justified his patches saying that
> > > > a slot that's inactive for a very long time could be problematic
> > > > because of XID wraparound.  Fine, that sounds a reasonable feature.
> > > > If you wanted to invalidate slots whose xmins were too old, I would
> > > > support that.  He submitted that as his 0004 patch then.
> > > >
> > > > However, he also chose to submit 0003 with invalidation based on a
> > > > timeout.  This is far less convincing a feature to me.  The
> > > > justification for the time out seems to be that ... it's difficult
> > > > to have a one-size-fits-all value because size of disks vary. (???)
> > > > Or something like that.  Really?  I mean -- yes, this will prevent
> > > > problems in toy databases when run in developer's laptops.  It will
> > > > not prevent any problems in production databases.  Do we really want
> > > > a setting that is only useful for toy situations rather than production?
> > > >
> > > >
> > ...
> > > >
> > > > I'm baffled.
> > >
> > > I agree, and I am also baffled because I think this discussion has
> > > happened at least once already on this thread.
> > >
> >
> > Yes, we previously discussed this topic and Robert seems to prefer a
> > time-based parameter for invalidating the slot (1)(2) as it is easier to reason in
> > terms of time. The other points discussed previously were that there are tools
> > that create a lot of slots and sometimes forget to clean up slots. Bharath has
> > seen this in production and we now have the tool pg_createsubscriber that
> > creates a slot-per-database, so if for some reason, such slots are not cleaned
> > on the tool's exit, such a parameter could save the cluster. See (3)(4).
> >
> > Also, we previously didn't have a good experience with XID-based threshold
> > parameters like vacuum_defer_cleanup_age as mentioned by Robert (1).
> > AFAICU from the previous discussion we need a time-based parameter and we
> > didn't rule out xid_age based parameter as another parameter.
>
> Yeah, I think the primary purpose of this time-based option is to invalidate dormant
> replication slots that have been inactive for a long period, in which case the
> slots are no longer useful.
>
> Such slots can remain if a subscriber is down due to a system error or
> inaccessible because of network issues. If this situation persists, it might be
> more practical to recreate the subscriber rather than attempt to recover the
> node and wait for it to catch up, which could be time-consuming.
>
> Parameters like max_slot_wal_keep_size and max_slot_xid_id_age do not
> differentiate between active and inactive replication slots. Some customers I
> met are hesitant about using these settings, as they can sometimes invalidate
> a slot unnecessarily and break the replication.
>

Alvaro, Nathan, do let us know if you would like to discuss more on
the use case for this new GUC idle_replication_slot_timeout?
Otherwise, we can proceed with this patch.

-- 
With Regards,
Amit Kapila.

  409. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-17T05:59:24Z

    On Fri, Feb 14, 2025 at 5:30 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> Here is a summary of changes in v78:
>

A few minor comments:
1.
Slots that appear idle due to a disrupted connection between
+        the publisher and subscriber are also excluded, as they are managed by
+        <link linkend="guc-wal-sender-timeout"><varname>wal_sender_timeout</varname></link>.
...

How do we exclude the above kind of slots? I think it is trying to
cover the case where walsender is not exited even after the connection
is broken between publisher and subscriber. The point is quite
confusing and adds much less value. So, we can remove it.

2.
- * Returns true when any slot have got invalidated.
+ * Returns true if there are any invalidated slots.
...

I find the existing comment more suitable for this function and easy to follow.

Apart from the above, I have changed a few other comments and minor
cosmetic cleanup.

-- 
With Regards,
Amit Kapila.

  410. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nisha Moond <nisha.moond412@gmail.com> — 2025-02-17T06:20:41Z

    On Mon, Feb 17, 2025 at 11:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Feb 14, 2025 at 5:30 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Here is a summary of changes in v78:
> >
>
> A few minor comments:
> 1.
> Slots that appear idle due to a disrupted connection between
> +        the publisher and subscriber are also excluded, as they are managed by
> +        <link linkend="guc-wal-sender-timeout"><varname>wal_sender_timeout</varname></link>.
> ...
>
> How do we exclude the above kind of slots? I think it is trying to
> cover the case where walsender is not exited even after the connection
> is broken between publisher and subscriber. The point is quite
> confusing and adds much less value. So, we can remove it.
>
> 2.
> - * Returns true when any slot have got invalidated.
> + * Returns true if there are any invalidated slots.
> ...
>
> I find the existing comment more suitable for this function and easy to follow.
>
> Apart from the above, I have changed a few other comments and minor
> cosmetic cleanup.
>

Here are the v79 patches with the above changes and comments from [1]
incorporated.

[1] https://www.postgresql.org/message-id/CAHut%2BPutqw%3D79SPh%2BEJZoS%2B98cJJvRRBmp-v6zqSRwngHey_ow%40mail.gmail.com

--
Thanks,
Nisha

  411. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Nathan Bossart <nathandbossart@gmail.com> — 2025-02-17T16:48:49Z

    On Mon, Feb 17, 2025 at 07:57:22AM +0530, Amit Kapila wrote:
> On Wed, Feb 12, 2025 at 1:16 PM Zhijie Hou (Fujitsu)
> <houzj.fnst@fujitsu.com> wrote:
>> On Wednesday, February 12, 2025 11:56 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>> > Also, we previously didn't have a good experience with XID-based threshold
>> > parameters like vacuum_defer_cleanup_age as mentioned by Robert (1).
>> > AFAICU from the previous discussion we need a time-based parameter and we
>> > didn't rule out xid_age based parameter as another parameter.

I am not sure I buy the comparison with vacuum_defer_cleanup_age.  That is
a very different feature than max_slot_xid_age, and we still have a number
of XID-based parameters (vacuum_freeze_table_age, vacuum_freeze_min_age,
vacuum_failsafe_age, the multixact versions of those parameters, and the
autovacuum versions).

>> Yeah, I think the primary purpose of this time-based option is to invalidate dormant
>> replication slots that have been inactive for a long period, in which case the
>> slots are no longer useful.
>>
>> Such slots can remain if a subscriber is down due to a system error or
>> inaccessible because of network issues. If this situation persists, it might be
>> more practical to recreate the subscriber rather than attempt to recover the
>> node and wait for it to catch up, which could be time-consuming.
>>
>> Parameters like max_slot_wal_keep_size and max_slot_xid_id_age do not
>> differentiate between active and inactive replication slots. Some customers I
>> met are hesitant about using these settings, as they can sometimes invalidate
>> a slot unnecessarily and break the replication.

Sure, an inactive-timeout feature won't break replication, but it's also
not going to be terribly effective against wraparound-related issues.  It
seems weird to me to allow an active replication slot to take priority over
imminent storage/XID issues it causes.

> Alvaro, Nathan, do let us know if you would like to discuss more on
> the use case for this new GUC idle_replication_slot_timeout?
> Otherwise, we can proceed with this patch.

I guess I'm not mortally opposed to it.  I just think we really need
proper backstops against the storage/XID issues more than we need this one,
and I don't want it to be mistaken for a solution to those problems.

-- 
nathan

  412. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-18T03:12:06Z

    On Mon, Feb 17, 2025 at 10:18 PM Nathan Bossart
<nathandbossart@gmail.com> wrote:
>
> On Mon, Feb 17, 2025 at 07:57:22AM +0530, Amit Kapila wrote:
>
> > Alvaro, Nathan, do let us know if you would like to discuss more on
> > the use case for this new GUC idle_replication_slot_timeout?
> > Otherwise, we can proceed with this patch.
>
> I guess I'm not mortally opposed to it.  I just think we really need
> proper backstops against the storage/XID issues more than we need this one,
> and I don't want it to be mistaken for a solution to those problems.
>

Fair enough. I see your point and would like to discuss the other
parameter in a separate thread. I plan to push the 0001 tomorrow after
some more review/testing unless I see any further arguments or
comments.

-- 
With Regards,
Amit Kapila.

  413. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-19T10:13:30Z

    On Tue, Feb 18, 2025 at 8:42 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Feb 17, 2025 at 10:18 PM Nathan Bossart
> <nathandbossart@gmail.com> wrote:
> >
> > On Mon, Feb 17, 2025 at 07:57:22AM +0530, Amit Kapila wrote:
> >
> > > Alvaro, Nathan, do let us know if you would like to discuss more on
> > > the use case for this new GUC idle_replication_slot_timeout?
> > > Otherwise, we can proceed with this patch.
> >
> > I guess I'm not mortally opposed to it.  I just think we really need
> > proper backstops against the storage/XID issues more than we need this one,
> > and I don't want it to be mistaken for a solution to those problems.
> >
>
> Fair enough. I see your point and would like to discuss the other
> parameter in a separate thread. I plan to push the 0001 tomorrow after
> some more review/testing unless I see any further arguments or
> comments.
>

Pushed after minor modifications.

-- 
With Regards,
Amit Kapila.

  414. Re: Introduce XID age and inactive timeout based replication slot invalidation

    Amit Kapila <amit.kapila16@gmail.com> — 2025-02-20T06:11:27Z

    On Wed, Feb 19, 2025 at 3:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Pushed after minor modifications.
>

I have closed the corresponding CF entry. Please feel free to start a
new thread for xid age based parameter.

-- 
With Regards,
Amit Kapila.