Horton

Re: speed up a logical replica setup

vignesh C <vignesh21@gmail.com>

View thread
From: vignesh C <vignesh21@gmail.com>
To: Peter Eisentraut <peter@eisentraut.org>
Cc: Euler Taveira <euler@eulerto.com>, Amit Kapila <amit.kapila16@gmail.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, "kuroda.hayato@fujitsu.com" <kuroda.hayato@fujitsu.com>, Tomas Vondra <tomas.vondra@enterprisedb.com>, "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>, Michael Paquier <michael@paquier.xyz>, Andres Freund <andres@anarazel.de>, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Fabrízio de Royes Mello <fabriziomello@gmail.com>
Date: 2024-03-21T13:33:30Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. pg_createsubscriber: Remove obsolete comment

  2. pg_createsubscriber: Fix an unpredictable recovery wait time.

  3. Fix unstable test in 040_pg_createsubscriber.

  4. Fix the testcase introduced in commit 81d20fbf7a.

  5. Further weaken new pg_createsubscriber test on Windows.

  6. Temporarily(?) weaken new pg_createsubscriber test on Windows.

  7. Make pg_createsubscriber warn if publisher has two-phase commit enabled.

  8. Make pg_createsubscriber more wary about quoting connection parameters.

  9. pg_createsubscriber: Remove failover replication slots on subscriber

  10. pg_createsubscriber: Remove replication slot check on primary

  11. pg_createsubscriber: Only --recovery-timeout controls the end of recovery process

  12. pg_createsubscriber: creates a new logical replica from a standby server

  13. Add some const decorations

  14. Add option force_initdb to PostgreSQL::Test::Cluster:init()

  15. Remove MSVC scripts

On Thu, 21 Mar 2024 at 18:02, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 21.03.24 12:35, vignesh C wrote:
> > Here are a few suggestions:
> > 1) I felt displaying the server log to the console is not a good idea,
> > I prefer this to be logged. There were similar comments from
> > Kuroda-san at [1], Peter at [2]. The number of lines will increase
> > based on the log level set. If you don't want to use pg_upgrade style,
> > how about exposing the log file option and logging it to the specified
> > log file.
>
> Let's leave that for the next version.  We need to wrap things up for
> this release.

Ok, This can be done as an improvement.

> > 2) Currently for publication, replication-slot and subscription, we
> > will have to specify these options based on the number of databases.
> > Here if we have 100 databases we will have to specify these options
> > 100 times, it might not be user friendly. How about something like
> > what Tomas had proposed at [3] and  Amit proposed at [4]. It will be
> > better if the user has to just specify publication, replication slot
> > and subscription options only one time.
>
> Same.  Designing, implementing, discussing, and testing this cannot be
> done in the time remaining.

If we commit this we might not be able to change the way the option
behaves once the customers starts using it. How about removing these
options in the first version and adding it in the next version after
more discussion.

> > 3) Can we have an option similar to dry-run which will display the
> > configurations required in the primary and standby node something
> > like:
> > pg_createsubscriber -D data_N2/ -P "port=5431 user=postgres"  -p 9999
> > -s /home/vignesh/postgres/inst/bin/ -U postgres -d db1 -d db2
> > --suggest-config
> > Suggested optimal configurations in the primary:
> > --------------------------------------
> > wallevel = logical
> > max_replication_slots = 3
> > max_wal_senders = 3
> > ...
> > Suggested optimal configurations in the standby:
> > --------------------------------------
> > max_replication_slots = 3
> > max_wal_senders = 3
> > ...
>
> How would this be different from what --dry-run does now?

Currently dry-run will do the check and might fail on identifying a
few failures like after checking subscriber configurations. Then the
user will have to correct the configuration and re-run then fix the
next set of failures. Whereas the suggest-config will display all the
optimal configuration for both the primary and the standby in a single
shot. This is not a must in the first version, it can be done as a
subsequent enhancement.

Regards,
Vignesh