Re: Introduce XID age and inactive timeout based replication slot invalidation

Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>

From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>

To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>

Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2024-02-20T06:35:00Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Add a test for commit ac0e33136a using the injection point.
- 8a695d7998be 18.0 landed
Invalidate inactive replication slots.
- ac0e33136abc 18.0 landed
Fix incorrect slot type in BuildTupleHashTableExt
- d96d1d5152f3 18.0 cited
Allow synced slots to have their inactive_since.
- 6f132ed693b6 17.0 landed
Change last_inactive_time to inactive_since in pg_replication_slots.
- 6d49c8d4b4f4 17.0 landed
Track last_inactive_time in pg_replication_slots.
- a11f330b5584 17.0 landed
Track invalidation_reason in pg_replication_slots.
- 6ae701b4378d 17.0 landed
Add option force_initdb to PostgreSQL::Test::Cluster:init()
- ff9e1e764fcc 17.0 cited
Add a failover option to subscriptions.
- 776621a5e479 17.0 cited
Allow setting failover property in the replication command.
- 73292404370c 17.0 cited
Allow to enable failover property for replication slots via SQL API.
- c393308b69d2 17.0 cited
Track conflict_reason in pg_replication_slots.
- 007693f2a3ac 17.0 cited
Log messages for replication slot acquisition and release.
- 7c3fb505b14e 17.0 cited
Remove vacuum_defer_cleanup_age
- 1118cd37eb61 16.0 cited
Fix corruption due to vacuum_defer_cleanup_age underflowing 64bit xids
- be504a3e974d 16.0 cited
meson: Add initial version of meson based build system
- e6927270cd18 16.0 cited

Attachments

v5-0001-Track-invalidation_reason-in-pg_replication_slots.patch (application/octet-stream) patch v5-0001
v5-0002-Add-XID-based-replication-slot-invalidation.patch (application/octet-stream) patch v5-0002
v5-0003-Track-inactive-replication-slot-information.patch (application/octet-stream) patch v5-0003
v5-0004-Add-inactive_timeout-based-replication-slot-inval.patch (application/octet-stream) patch v5-0004

On Fri, Feb 9, 2024 at 1:12 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> I think "conflict" is an important topic and does contain several reasons. The
> slot "first" conflict and then leads to slot "invalidation".
>
> > They both are the same internally, so why
> > confuse the users?
>
> I don't think that would confuse the users, I do think that would be easier to
> check for conflicting slots.

I've added a separate column for invalidation reasons for now. I'll
see how others think on this as the time goes by.

> I did not look closely at the code, just played a bit with the patch and was able
> to produce something like:
>
> postgres=# select slot_name,slot_type,active,active_pid,wal_status,invalidation_reason from pg_replication_slots;
>   slot_name  | slot_type | active | active_pid | wal_status | invalidation_reason
> -------------+-----------+--------+------------+------------+---------------------
>  rep1        | physical  | f      |            | reserved   |
>  master_slot | physical  | t      |    1482441 | unreserved | wal_removed
> (2 rows)
>
> does that make sense to have an "active/working" slot "ivalidated"?

Thanks. Can you please provide the steps to generate this error? Are
you setting max_slot_wal_keep_size on primary to generate
"wal_removed"?

Attached v5 patch set after rebasing and addressing review comments.
Please review it further.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com