Horton

Re: Synchronizing slots from primary to standby

Ashutosh Sharma <ashu.coek88@gmail.com>

View thread View on postgr.es
From: Ashutosh Sharma <ashu.coek88@gmail.com>
To: "Hsu, John" <hsuchen@amazon.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2022-02-04T17:51:24Z
Lists: pgsql-hackers 
On Sat, Jan 22, 2022 at 4:33 AM Hsu, John <hsuchen@amazon.com> wrote:
>
>    > I might be missing something but isn’t it okay even if the new primary
>    > server is behind the subscribers? IOW, even if two slot's LSNs (i.e.,
>    > restart_lsn and confirm_flush_lsn) are behind the subscriber's remote
>    > LSN (i.e., pg_replication_origin.remote_lsn), the primary sends only
>    > transactions that were committed after the remote_lsn. So the
>    > subscriber can resume logical replication with the new primary without
>    > any data loss.
>
>     Maybe I'm misreading, but I thought the purpose of this to make
>     sure that the logical subscriber does not have data that has not been
>     replicated to the new primary. The use-case I can think of would be
>     if synchronous_commit were enabled and fail-over occurs. If
>     we didn't have this set, isn't it possible that this logical subscriber
>     has extra commits that aren't present on the newly promoted primary?
>

This is very much possible if the new primary used to be asynchronous
standby. But, it seems like the current patch is trying to hold the
logical replication until the data has been replicated to the physical
standby when synchronous_slot_names is set. This will ensure that the
logical subscriber is never ahead of the new primary. However, AFAIU
that's not the primary use-case of this patch; instead this is to
ensure that the logical subscribers continue getting data from the new
primary when the failover occurs.

>
>     If a logical slot was dropped on the writer, should the worker drop logical
>     slots that it was previously synchronizing but are no longer present? Or
>     should we leave that to the user to manage? I'm trying to think why users
>     would want to sync logical slots to a reader but not have that be dropped
>     as well if it's no longer present.
>

AFAIU this should be taken care of by the background worker used to
synchronize the replication slot.

--
With Regards,
Ashutosh Sharma.

Commits

  1. Doc: Add the new section "Logical Replication Failover".

  2. Fix the review comments and a bug in the slot sync code.

  3. Fix a test in failover slots regression test.

  4. Fix the intermittent buildfarm failures in 040_standby_failover_slots_sync.

  5. Split XLogCtl->LogwrtResult into separate struct members

  6. Ensure that the sync slots reach a consistent state after promotion without losing data.

  7. Introduce a new GUC 'standby_slot_names'.

  8. Fix BF failure introduced by commit b3f6b14cf4.

  9. Fixups for commit 93db6cbda0.

  10. Fix BF failure in commit 93db6cbda0.

  11. Add a new slot sync worker to synchronize logical slots.

  12. Improve ERROR/LOG messages added by commits ddd5f4f54a and 7a424ece48.

  13. Disable autovacuum on primary in 040_standby_failover_slots_sync test.

  14. Fix the incorrect format specifier used in commit 7a424ece48.

  15. Change the LOG level in 040_standby_failover_slots_sync.pl to DEBUG2.

  16. Add more LOG and DEBUG messages for slot synchronization.

  17. Another try to fix BF failure introduced in commit ddd5f4f54a.

  18. Fix BF introduced in commit ddd5f4f54a.

  19. Add a slot synchronization function.

  20. Clean-ups for 776621a5e4 and 7329240437.

  21. Enhance libpqrcv APIs to support slot synchronization.

  22. Add a failover option to subscriptions.

  23. Allow setting failover property in the replication command.

  24. Allow to enable failover property for replication slots via SQL API.

  25. Add macros for looping through a List without a ListCell.

  26. Track conflict_reason in pg_replication_slots.

  27. Allow upgrades to preserve the full subscription's state.

  28. Add support for incremental backup.

  29. Simplify some logic in CreateReplicationSlot()

  30. Fix uninitialized access to InitialRunningXacts during decoding after ERROR.

  31. Flexible options for CREATE_REPLICATION_SLOT.

  32. Empty search_path in logical replication apply worker and walsender.

  33. Fix oldest xmin and LSN computation across repslots after advancing

  34. Require C99 (and thus MSCV 2013 upwards).