Horton

Re: Logical Replication of sequences

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Peter Eisentraut <peter@eisentraut.org>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Euler Taveira <euler@eulerto.com>, Michael Paquier <michael@paquier.xyz>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Hou, Zhijie/侯 志杰 <houzj.fnst@fujitsu.com>, Sawada Masahiko <sawada.mshk@gmail.com>, "Katz, Jonathan" <jkatz@amazon.com>
Date: 2024-06-06T04:04:13Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Add documentation for sequence synchronization.

  2. Remove unused assignment in CREATE PUBLICATION grammar.

  3. Add seq_sync_error_count to subscription statistics.

  4. Fix few issues in commit 5509055d69.

  5. Add sequence synchronization for logical replication.

  6. Add worker type argument to logical replication worker functions.

  7. Introduce "REFRESH SEQUENCES" for subscriptions.

  8. Refactor logical worker synchronization code into a separate file.

  9. Standardize use of REFRESH PUBLICATION in code and messages.

  10. Add "ALL SEQUENCES" support to publications.

  11. Expose sequence page LSN via pg_get_sequence_data.

  12. Resume conflict-relevant data retention automatically.

  13. Post-commit review fixes for 228c370868.

  14. Generate GUC tables from .dat file

On Wed, Jun 5, 2024 at 3:17 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Jun 4, 2024 at 5:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Even if we decode it periodically (say each time we decode the
> > checkpoint record) then also we need to send the entire set of
> > sequences at shutdown. This is because the sequences may have changed
> > from the last time we sent them.
>
> Agree. How about decoding and sending only the sequences that are
> changed from the last time when they were sent? I know it requires a
> bit of tracking and more work, but all I'm looking for is to reduce
> the amount of work that walsenders need to do during the shutdown.
>

I see your point but going towards tracking the changed sequences
sounds like moving towards what we do for incremental backups unless
we can invent some other smart way.

> Having said that, I like the idea of letting the user sync the
> sequences via ALTER SUBSCRIPTION command and not weave the logic into
> the shutdown checkpoint path. As Peter Eisentraut said here
> https://www.postgresql.org/message-id/42e5cb35-4aeb-4f58-8091-90619c7c3ecc%40eisentraut.org,
> this can be a good starting point to get going.
>

Agreed.

> > > I can imagine a case where there are tens
> > > of thousands of sequences in a production server, and surely decoding
> > > and sending them just during the shutdown can take a lot of time
> > > hampering the overall server uptime.
> >
> > It is possible but we will send only the sequences that belong to
> > publications for which walsender is supposed to send the required
> > data.
>
> Right, but what if all the publication tables can have tens of
> thousands of sequences.
>

In such cases we have no option but to send all the sequences.

> > Now, we can also imagine providing option 2 (Alter Subscription
> > ... Replicate Sequences) so that users can replicate sequences before
> > shutdown and then disable the subscriptions so that there won't be a
> > corresponding walsender.
>
> As stated above, I like this idea to start with.
>

+1.


--
With Regards,
Amit Kapila.