Horton

Re: Logical Replication of sequences

shveta malik <shveta.malik@gmail.com>

View thread
From: shveta malik <shveta.malik@gmail.com>
To: Peter Smith <smithpb2250@gmail.com>
Cc: Nisha Moond <nisha.moond412@gmail.com>, vignesh C <vignesh21@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Masahiko Sawada <sawada.mshk@gmail.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Peter Eisentraut <peter@eisentraut.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Euler Taveira <euler@eulerto.com>, Michael Paquier <michael@paquier.xyz>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, "Jonathan S. Katz" <jkatz@postgresql.org>, shveta malik <shveta.malik@gmail.com>
Date: 2025-05-21T10:41:22Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Add documentation for sequence synchronization.

  2. Remove unused assignment in CREATE PUBLICATION grammar.

  3. Add seq_sync_error_count to subscription statistics.

  4. Fix few issues in commit 5509055d69.

  5. Add sequence synchronization for logical replication.

  6. Add worker type argument to logical replication worker functions.

  7. Introduce "REFRESH SEQUENCES" for subscriptions.

  8. Refactor logical worker synchronization code into a separate file.

  9. Standardize use of REFRESH PUBLICATION in code and messages.

  10. Add "ALL SEQUENCES" support to publications.

  11. Expose sequence page LSN via pg_get_sequence_data.

  12. Resume conflict-relevant data retention automatically.

  13. Post-commit review fixes for 228c370868.

  14. Generate GUC tables from .dat file

On Tue, May 20, 2025 at 11:13 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> > Test-scenario:
> > --Created 250 sequences on both pub and sub.
> > --There were 10 sequences mismatched.
> > --Sequence replication worked as expected. Logs look better now:
> >
> > LOG:  Logical replication sequence synchronization for subscription
> > "sub1" - total unsynchronized: 250; batch #1 = 100 attempted, 97
> > succeeded, 3 mismatched
> > LOG:  Logical replication sequence synchronization for subscription
> > "sub1" - total unsynchronized: 250; batch #2 = 100 attempted, 95
> > succeeded, 5 mismatched
> > LOG:  Logical replication sequence synchronization for subscription
> > "sub1" - total unsynchronized: 250; batch #3 = 50 attempted, 48
> > succeeded, 2 mismatched
> >
>
> When there are many batches required, it seems a bit strange to repeat
> the same "total unsynchronized" over and over.
>
> Would it be better to show the total number once, and thereafter show
> the number of sequences remaining to be processed as they tick down?
>
> e.g.
> LOG:  Logical replication sequence synchronization for subscription
> "sub1" - total unsynchronized = 250
> LOG:  Logical replication sequence synchronization for subscription
> "sub1" - batch #1 = 100 attempted, 97 succeeded, 3 mismatched, 150
> remaining
> LOG:  Logical replication sequence synchronization for subscription
> "sub1" - batch #2 = 100 attempted, 95 succeeded, 5 mismatched, 50
> remaining
> LOG:  Logical replication sequence synchronization for subscription
> "sub1" - batch #3 = 50 attempted, 48 succeeded, 2 mismatched, 0
> remaining

+1 on log change suggestions.

Please find few more comments:

1)
Temporary sequences will not be replicated, shall we mention this in
doc under '29.7. Replicating Sequences'?

2)
CREATE publication pub1 for all sequences  WITH (publish = 'insert,
update, truncate');

I think it does not make sense to give 'publish' as above (or
publish_via_partition_root) for 'all sequences' publication. Shall we
display a WARNING that such will be ignored for 'all sequences' and
let the create-publication go ahead? Thoughts? Also the doc for
publish* option in the CREATE-PUBLICATION page needs to specify that
these options are not-applicable for ALL SEQUENCES publication.

3)
It will be good to move create_publication.sgml as well to the last
patch where all other doc changes are present. I was trying to find
this change in the last patch but ultimately found it in pacth002.

4)
Currently the log is:

------
LOG:  logical replication sequence synchronization worker for
subscription "sub1" has started
LOG:  Logical replication sequence synchronization for subscription
"sub1" - total unsynchronized: 1; batch #1 = 1 attempted, 0 succeeded,
1 mismatched
WARNING:  parameters differ for the remote and local sequences
("public.myseq34") for subscription "sub1"
HINT:  Alter/Re-create local sequences to have the same parameters as
the remote sequences.
WARNING:  sequence synchronization worker failed: one or more
sequences have mismatched parameters between the publisher and
subscriber
LOG:  logical replication sequence synchronization worker for
subscription "sub1" has finished
-----

Do we need both?
--WARNING:  sequence synchronization worker failed.
--LOG:  logical replication sequence synchronization worker for
subscription "sub1" has finished

This WARNING repeats previously stated information. I feel we can get
rid of it, unless there is a chance of some new error which we are
trying to display in this WARNING other than mismatched seq error?

thanks
Shveta