Horton

RE: Synchronizing slots from primary to standby

Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>

View thread View on postgr.es
From: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
To: shveta malik <shveta.malik@gmail.com>, "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Peter Smith <smithpb2250@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Bruce Momjian <bruce@momjian.us>, Ashutosh Sharma <ashu.coek88@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, Ajin Cherian <itsajin@gmail.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: 2023-11-22T04:32:36Z
Lists: pgsql-hackers 
On Tuesday, November 21, 2023 5:33 PM shveta malik <shveta.malik@gmail.com> wrote:
> 
> 
> v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the
> patches.  PFA v37_2 patches.

Thanks for updating the patches.

I'd like to discuss one issue related to the correct handling of failover flag
when executing ALTER SUBSCRIPTION SET (slot_name = 'new_slot')".

Since the command intends to use a new slot on the primary, the new slot needs
to reflect the "failover" state that the subscription currently has. If the
failoverstate of the Subscription is LOGICALREP_FAILOVER_STATE_ENABLED, then I
can reset it to LOGICALREP_FAILOVER_STATE_PENDING and allow the apply worker to
handle it the way it is handled today (just like two_phase handling).

But if the failoverstate is LOGICALREP_FAILOVER_STATE_DISABLED, the original
idea is to call walrcv_alter_slot and alter the slot from the "ALTER
SUBSCRIPTION" handling backend itself. This works if the slot is currently
disabled. But the " ALTER SUBSCRIPTION SET (slot_name = 'new_slot')" command is
supported even if the subscription is enabled. If the subscription is enabled,
then calling walrcv_alter_slot() fails because the slot is still acquired by
apply worker.

So, I am thinking do we need a new mechanism to change the failover flag to
false on an enabled subscription ? For example, we could call walrcv_alter_slot
on startup of apply worker if AllTablesyncsReady(), for both true and false
values of failover flag. This way, every time apply worker is started, it calls
walrcv_alter_slot to set the failover flag on the primary.

Or we could just document that it is user's responsibility to match the failover
property in case it changes the slot_name.

Thoughts ?

Best Regards,
Hou zj

Commits

  1. Doc: Add the new section "Logical Replication Failover".

  2. Fix the review comments and a bug in the slot sync code.

  3. Fix a test in failover slots regression test.

  4. Fix the intermittent buildfarm failures in 040_standby_failover_slots_sync.

  5. Split XLogCtl->LogwrtResult into separate struct members

  6. Ensure that the sync slots reach a consistent state after promotion without losing data.

  7. Introduce a new GUC 'standby_slot_names'.

  8. Fix BF failure introduced by commit b3f6b14cf4.

  9. Fixups for commit 93db6cbda0.

  10. Fix BF failure in commit 93db6cbda0.

  11. Add a new slot sync worker to synchronize logical slots.

  12. Improve ERROR/LOG messages added by commits ddd5f4f54a and 7a424ece48.

  13. Disable autovacuum on primary in 040_standby_failover_slots_sync test.

  14. Fix the incorrect format specifier used in commit 7a424ece48.

  15. Change the LOG level in 040_standby_failover_slots_sync.pl to DEBUG2.

  16. Add more LOG and DEBUG messages for slot synchronization.

  17. Another try to fix BF failure introduced in commit ddd5f4f54a.

  18. Fix BF introduced in commit ddd5f4f54a.

  19. Add a slot synchronization function.

  20. Clean-ups for 776621a5e4 and 7329240437.

  21. Enhance libpqrcv APIs to support slot synchronization.

  22. Add a failover option to subscriptions.

  23. Allow setting failover property in the replication command.

  24. Allow to enable failover property for replication slots via SQL API.

  25. Add macros for looping through a List without a ListCell.

  26. Track conflict_reason in pg_replication_slots.

  27. Allow upgrades to preserve the full subscription's state.

  28. Add support for incremental backup.

  29. Simplify some logic in CreateReplicationSlot()

  30. Fix uninitialized access to InitialRunningXacts during decoding after ERROR.

  31. Flexible options for CREATE_REPLICATION_SLOT.

  32. Empty search_path in logical replication apply worker and walsender.

  33. Fix oldest xmin and LSN computation across repslots after advancing

  34. Require C99 (and thus MSCV 2013 upwards).