Re: Logical Replication of sequences

vignesh C <vignesh21@gmail.com>

From: vignesh C <vignesh21@gmail.com>
To: shveta malik <shveta.malik@gmail.com>
Cc: Peter Smith <smithpb2250@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Masahiko Sawada <sawada.mshk@gmail.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Peter Eisentraut <peter@eisentraut.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Euler Taveira <euler@eulerto.com>, Michael Paquier <michael@paquier.xyz>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, "Jonathan S. Katz" <jkatz@postgresql.org>
Date: 2025-05-14T04:56:31Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →
  1. Doc: Add documentation for sequence synchronization.

  2. Remove unused assignment in CREATE PUBLICATION grammar.

  3. Add seq_sync_error_count to subscription statistics.

  4. Fix few issues in commit 5509055d69.

  5. Add sequence synchronization for logical replication.

  6. Add worker type argument to logical replication worker functions.

  7. Introduce "REFRESH SEQUENCES" for subscriptions.

  8. Refactor logical worker synchronization code into a separate file.

  9. Standardize use of REFRESH PUBLICATION in code and messages.

  10. Add "ALL SEQUENCES" support to publications.

  11. Expose sequence page LSN via pg_get_sequence_data.

  12. Resume conflict-relevant data retention automatically.

  13. Post-commit review fixes for 228c370868.

  14. Generate GUC tables from .dat file

Attachments

On Fri, 9 May 2025 at 14:28, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Sat, May 3, 2025 at 7:27 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > >
> > > Thanks for the comments, the updated patch has the changes for the same.
> >
>
> Thanks for the patches. Please find few comments:
>
>
> 1)
> patch004 commit msg:
> - Drop published sequences are removed from pg_subscription_rel.
>
> Drop -->Dropped

Modified

> 2)
> copy_sequences:
>
> LOG:  Executing query :SELECT s.schname, s.seqname, ps.*, seq.seqtypid,
>        seq.seqstart, seq.seqincrement, seq.seqmin,
>        seq.seqmax, seq.seqcycle
> FROM ( VALUES ('public', 'myseq1'), ('public', 'myseq3') ) AS s
> (schname, seqname)
> JOIN LATERAL pg_sequence_state(s.schname, s.seqname) ps ON true
> ....
>
> Do we really need to log this query? If so, shall it be DEBUG1/DEBUG2?

This is not required, removed it

> 3)
> In log, we get:
>
> ------------------
> LOG:  logical replication synchronized 9 of 9 sequences for subscription "sub1"
> WARNING:  parameters differ for the remote and local sequences
> ("public.myseq1", "public.myseq3") for subscription "sub1"
>
> LOG:  logical replication synchronized 2 of 2 sequences for subscription "sub1"
> WARNING:  parameters differ for the remote and local sequences
> ("public.myseq1", "public.myseq3") for subscription "sub1"
> ------------------
>
> This is confusing. I have 9 sequences, out of which 2 are mismatched.
> So on REFRESH I get the first message as 'synchronized 9 of 9' and
> later when it attempts to resynchronize pending ones automatically, it
> keeps on displaying  'synchronized 2 of 2'.
>
> Can we mention something like below:
> -----------------
> Unsynchronized sequences: 9, attempted in this batch: 9, succedded: 7,
> mismatched/failed:2
>
> So that if it is more than 100, say 120, it will say:
> Unsynchronized sequences: 120, attempted in this batch: 100,
> succedded: 98, mismatched/failed:2
> And in next batch:
> Unsynchronized sequences: 120, attempted in this batch: 20, succedded:
> 20, mismatched:0
>
> And while attempting to synchronize failed ones, it will say:
> Unsynchronized sequences: 2, attempted in this batch: 2, succedded: 0,
> mismatched:2
> -----------------
>
> Please feel free to change the words. The intent is to get a clear
> picture on what is happening.

Fixed this

> 4)
> Why in patch001, we have 'pg_sequence_state' with one argument while
> in 4ht patch it is changed to 2 args? Is it intentional to have it the
> current way in patch001?

This should have been in 001 itself, moved these changes to 001 patch

> 5)
> Section1:
>   <para>
>    A new <firstterm>sequence synchronization worker</firstterm> will be started
>    after executing any of the above subscriber commands, and will exit once the
>    sequences are synchronized.
>   </para>
>
> Section2:
>   <sect2 id="sequence-definition-mismatches">
>    <title>Sequence Definition Mismatches</title>
>    <warning>
>     <para>
>      During sequence synchronization, the sequence definitions of the publisher
>      and the subscriber are compared. A WARNING is logged if any differences
>      are detected.
>     </para>
>    </warning>
>
>
> None of the section mentions that the synchronization worker will keep
> on attempting to synchronize the failed/mismtached sequences until the
> differences are resolved (provided disable_on_error is not enabled).
> I think we can mention such a thing briefly in
> 'sequence-definition-mismatches' section.

Modified

The attached v20250514 version patch has the changes for the same.

Regards,
Vignesh