Horton

Re: speed up a logical replica setup

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Peter Eisentraut <peter@eisentraut.org>
Cc: Euler Taveira <euler@eulerto.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, "kuroda.hayato@fujitsu.com" <kuroda.hayato@fujitsu.com>, Tomas Vondra <tomas.vondra@enterprisedb.com>, "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>, Michael Paquier <michael@paquier.xyz>, Andres Freund <andres@anarazel.de>, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Fabrízio de Royes Mello <fabriziomello@gmail.com>, vignesh C <vignesh21@gmail.com>
Date: 2024-03-19T07:05:17Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. pg_createsubscriber: Remove obsolete comment

  2. pg_createsubscriber: Fix an unpredictable recovery wait time.

  3. Fix unstable test in 040_pg_createsubscriber.

  4. Fix the testcase introduced in commit 81d20fbf7a.

  5. Further weaken new pg_createsubscriber test on Windows.

  6. Temporarily(?) weaken new pg_createsubscriber test on Windows.

  7. Make pg_createsubscriber warn if publisher has two-phase commit enabled.

  8. Make pg_createsubscriber more wary about quoting connection parameters.

  9. pg_createsubscriber: Remove failover replication slots on subscriber

  10. pg_createsubscriber: Remove replication slot check on primary

  11. pg_createsubscriber: Only --recovery-timeout controls the end of recovery process

  12. pg_createsubscriber: creates a new logical replica from a standby server

  13. Add some const decorations

  14. Add option force_initdb to PostgreSQL::Test::Cluster:init()

  15. Remove MSVC scripts

On Mon, Mar 18, 2024 at 7:22 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> In check_subscriber(): All these permissions checks seem problematic
> to me.  We shouldn't reimplement our own copy of the server's
> permission checks.  The server can check the permissions.  And if the
> permission checking in the server ever changes, then we have
> inconsistencies to take care of.  Also, the error messages "permission
> denied" are inappropriate, because we are not doing the actual thing.
> Maybe we want to do a dry-run for the benefit of the user, but then we
> should do the actual thing, like try to create a replication slot, or
> whatever.  But I would rather just remove all this, it seems too
> problematic.
>

If we remove all the checks then there is a possibility that we can
fail later while creating the actual subscription. For example, if
there are not sufficient max_replication_slots, then it is bound to
fail in the later steps which would be a costlier affair because by
that time the standby would have been promoted and the user won't have
any way to move forward but to re-create standby and then use this
tool again. I think here the patch tries to mimic pg_upgrade style
checks where we do some pre-checks.

This raises a question in my mind how are we expecting users to know
all these required values and configure it properly before using this
tool?  IIUC, we are expecting that the user should figure out the
appropriate values for max_replication_slots,
max_logical_replication_workers, etc. by querying the number of
databases in the primary. Then either stop the standby to change these
parameters or use ALTER SYSTEM depending on the required parameters.
Similarly, there are some config requirements (like max_wal_senders,
max_replication_slots) for the primary which would be difficult for
users to know as these are tools are internal requirements.

The two possibilities that come to my mind are (a) pg_createsubscriber
should have some special mode/option using which the user can find out
all the required config settings, or (b) document how a user can find
the required settings. There are good chances of mistakes with option
(b).

-- 
With Regards,
Amit Kapila.