Horton

Re: Logical Replication of sequences

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Peter Eisentraut <peter@eisentraut.org>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Euler Taveira <euler@eulerto.com>, Michael Paquier <michael@paquier.xyz>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Hou, Zhijie/侯 志杰 <houzj.fnst@fujitsu.com>, Sawada Masahiko <sawada.mshk@gmail.com>, "Katz, Jonathan" <jkatz@amazon.com>
Date: 2024-06-04T12:10:08Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Add documentation for sequence synchronization.

  2. Remove unused assignment in CREATE PUBLICATION grammar.

  3. Add seq_sync_error_count to subscription statistics.

  4. Fix few issues in commit 5509055d69.

  5. Add sequence synchronization for logical replication.

  6. Add worker type argument to logical replication worker functions.

  7. Introduce "REFRESH SEQUENCES" for subscriptions.

  8. Refactor logical worker synchronization code into a separate file.

  9. Standardize use of REFRESH PUBLICATION in code and messages.

  10. Add "ALL SEQUENCES" support to publications.

  11. Expose sequence page LSN via pg_get_sequence_data.

  12. Resume conflict-relevant data retention automatically.

  13. Post-commit review fixes for 228c370868.

  14. Generate GUC tables from .dat file

On Tue, Jun 4, 2024 at 4:53 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Jun 4, 2024 at 4:27 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > 3. Replicate published sequences via walsender at the time of shutdown
> > or incrementally while decoding checkpoint record. The two ways to
> > achieve this are: (a) WAL log a special NOOP record just before
> > shutting down checkpointer. Then allow the WALsender to read the
> > sequence data and send it to the subscriber while decoding the new
> > NOOP record. (b) Similar to the previous idea but instead of WAL
> > logging a new record directly invokes a decoding callback after
> > walsender receives a request to shutdown which will allow pgoutput to
> > read and send required sequences. This approach has a drawback that we
> > are adding more work at the time of shutdown but note that we already
> > waits for all the WAL records to be decoded and sent before shutting
> > down the walsender during shutdown of the node.
>
> Thanks. IIUC, both of the above approaches decode the sequences during
> only shutdown. I'm wondering, why not periodically decode and
> replicate the published sequences so that the decoding at the shutdown
> will not take that longer?
>

Even if we decode it periodically (say each time we decode the
checkpoint record) then also we need to send the entire set of
sequences at shutdown. This is because the sequences may have changed
from the last time we sent them.

>
> I can imagine a case where there are tens
> of thousands of sequences in a production server, and surely decoding
> and sending them just during the shutdown can take a lot of time
> hampering the overall server uptime.
>

It is possible but we will send only the sequences that belong to
publications for which walsender is supposed to send the required
data. Now, we can also imagine providing option 2 (Alter Subscription
... Replicate Sequences) so that users can replicate sequences before
shutdown and then disable the subscriptions so that there won't be a
corresponding walsender.

-- 
With Regards,
Amit Kapila.