Horton

Re: Logical Replication of sequences

Yogesh Sharma <yogesh.sharma@catprosystems.com>

View thread
From: Yogesh Sharma <yogesh.sharma@catprosystems.com>
To: Amit Kapila <amit.kapila16@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Peter Eisentraut <peter@eisentraut.org>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Euler Taveira <euler@eulerto.com>, Michael Paquier <michael@paquier.xyz>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Hou, Zhijie/侯 志杰 <houzj.fnst@fujitsu.com>, Sawada Masahiko <sawada.mshk@gmail.com>, "Katz, Jonathan" <jkatz@amazon.com>
Date: 2024-06-04T15:26:47Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Add documentation for sequence synchronization.

  2. Remove unused assignment in CREATE PUBLICATION grammar.

  3. Add seq_sync_error_count to subscription statistics.

  4. Fix few issues in commit 5509055d69.

  5. Add sequence synchronization for logical replication.

  6. Add worker type argument to logical replication worker functions.

  7. Introduce "REFRESH SEQUENCES" for subscriptions.

  8. Refactor logical worker synchronization code into a separate file.

  9. Standardize use of REFRESH PUBLICATION in code and messages.

  10. Add "ALL SEQUENCES" support to publications.

  11. Expose sequence page LSN via pg_get_sequence_data.

  12. Resume conflict-relevant data retention automatically.

  13. Post-commit review fixes for 228c370868.

  14. Generate GUC tables from .dat file

On 6/4/24 06:57, Amit Kapila wrote:
> 1. Provide a tool to copy all the sequences from publisher to
> subscriber. The major drawback is that users need to perform this as
> an additional step during the upgrade which would be inconvenient and
> probably not as useful as some built-in mechanism.

Agree, this requires additional steps. Not a preferred approach in my
opinion. When a large set of sequences are present, it will add
additional downtime for upgrade process.

> 2. Provide a command say Alter Subscription ...  Replicate Sequences
> (or something like that) which users can perform before shutdown of
> the publisher node during upgrade. This will allow copying all the
> sequences from the publisher node to the subscriber node directly.
> Similar to previous approach, this could also be inconvenient for
> users.

This is similar to option 1 except that it is a SQL command now. Still
not a preferred approach in my opinion. When a large set of sequences are
present, it will add additional downtime for upgrade process.


> 3. Replicate published sequences via walsender at the time of shutdown
> or incrementally while decoding checkpoint record. The two ways to
> achieve this are: (a) WAL log a special NOOP record just before
> shutting down checkpointer. Then allow the WALsender to read the
> sequence data and send it to the subscriber while decoding the new
> NOOP record. (b) Similar to the previous idea but instead of WAL
> logging a new record directly invokes a decoding callback after
> walsender receives a request to shutdown which will allow pgoutput to
> read and send required sequences. This approach has a drawback that we
> are adding more work at the time of shutdown but note that we already
> waits for all the WAL records to be decoded and sent before shutting
> down the walsender during shutdown of the node.

At the time of shutdown a) most logical upgrades don't necessarily call
for shutdown b) it will still add to total downtime with large set of
sequences. Incremental option is better as it will not require a shutdown.

I do see a scenario where sequence of events can lead to loss of sequence
and generate duplicate sequence values, if subscriber starts consuming
sequences while publisher is also consuming them. In such cases, subscriber
shall not be allowed sequence consumption.



-- 
Kind Regards,
Yogesh Sharma
Open Source Enthusiast and Advocate
PostgreSQL Contributors Team @ RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com