Horton

Re: Logical Replication of sequences

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: shveta malik <shveta.malik@gmail.com>
Cc: vignesh C <vignesh21@gmail.com>, Peter Smith <smithpb2250@gmail.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, Peter Eisentraut <peter@eisentraut.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Euler Taveira <euler@eulerto.com>, Michael Paquier <michael@paquier.xyz>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Hou, Zhijie/侯 志杰 <houzj.fnst@fujitsu.com>, "Jonathan S. Katz" <jkatz@postgresql.org>
Date: 2024-08-06T04:24:35Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Add documentation for sequence synchronization.

  2. Remove unused assignment in CREATE PUBLICATION grammar.

  3. Add seq_sync_error_count to subscription statistics.

  4. Fix few issues in commit 5509055d69.

  5. Add sequence synchronization for logical replication.

  6. Add worker type argument to logical replication worker functions.

  7. Introduce "REFRESH SEQUENCES" for subscriptions.

  8. Refactor logical worker synchronization code into a separate file.

  9. Standardize use of REFRESH PUBLICATION in code and messages.

  10. Add "ALL SEQUENCES" support to publications.

  11. Expose sequence page LSN via pg_get_sequence_data.

  12. Resume conflict-relevant data retention automatically.

  13. Post-commit review fixes for 228c370868.

  14. Generate GUC tables from .dat file

On Tue, Aug 6, 2024 at 8:49 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Aug 5, 2024 at 5:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Aug 5, 2024 at 2:36 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Fri, 2 Aug 2024 at 14:24, shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > > >
> > > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > >
> > > > > > Thanks for reporting this, these issues are fixed in the attached
> > > > > > v20240730_2 version patch.
> > > > > >
> > > >
> > > > I was reviewing the design of patch003, and I have a query. Do we need
> > > > to even start an apply worker and create replication slot when
> > > > subscription created is for 'sequences only'? IIUC, currently logical
> > > > replication apply worker is the one launching sequence-sync worker
> > > > whenever needed. I think it should be the launcher doing this job and
> > > > thus apply worker may even not be needed for current functionality of
> > > > sequence sync?
> > >
> >
> > But that would lead to maintaining all sequence-sync of each
> > subscription by launcher. Say there are 100 sequences per subscription
> > and some of them from each subscription are failing due to some
> > reasons then the launcher will be responsible for ensuring all the
> > sequences are synced. I think it would be better to handle
> > per-subscription work by the apply worker.
>
> I thought we can give that task to sequence-sync worker. Once sequence
> sync worker is started by launcher, it keeps on syncing until all the
> sequences are synced (even failed ones) and then exits only after all
> are synced; instead of apply worker starting it multiple times for
> failed sequences. Launcher to start sequence sync worker when signaled
> by 'alter-sub refresh seq'.
> But after going through details given by Vignesh in [1], I also see
> the benefits of using apply worker for this task. Since apply worker
> is already looping and doing that for table-sync, we can reuse the
> same code for sequence sync and maintenance will be easy. So looks
> okay if we go with existing apply worker design.
>

Fair enough. However, I was wondering whether apply_worker should exit
after syncing all sequences for a sequence-only subscription or should
it be there for future commands that can refresh the subscription and
add additional tables or sequences?

-- 
With Regards,
Amit Kapila.