Horton

Re: speed up a logical replica setup

Euler Taveira <euler@eulerto.com>

View thread
From: "Euler Taveira" <euler@eulerto.com>
To: "kuroda.hayato@fujitsu.com" <kuroda.hayato@fujitsu.com>, "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>
Cc: "vignesh C" <vignesh21@gmail.com>, "Michael Paquier" <michael@paquier.xyz>, "Peter Eisentraut" <peter@eisentraut.org>, "Andres Freund" <andres@anarazel.de>, "Ashutosh Bapat" <ashutosh.bapat.oss@gmail.com>, "Amit Kapila" <amit.kapila16@gmail.com>, "Shlok Kyal" <shlok.kyal.oss@gmail.com>
Date: 2024-01-30T02:01:38Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. pg_createsubscriber: Remove obsolete comment

  2. pg_createsubscriber: Fix an unpredictable recovery wait time.

  3. Fix unstable test in 040_pg_createsubscriber.

  4. Fix the testcase introduced in commit 81d20fbf7a.

  5. Further weaken new pg_createsubscriber test on Windows.

  6. Temporarily(?) weaken new pg_createsubscriber test on Windows.

  7. Make pg_createsubscriber warn if publisher has two-phase commit enabled.

  8. Make pg_createsubscriber more wary about quoting connection parameters.

  9. pg_createsubscriber: Remove failover replication slots on subscriber

  10. pg_createsubscriber: Remove replication slot check on primary

  11. pg_createsubscriber: Only --recovery-timeout controls the end of recovery process

  12. pg_createsubscriber: creates a new logical replica from a standby server

  13. Add some const decorations

  14. Add option force_initdb to PostgreSQL::Test::Cluster:init()

  15. Remove MSVC scripts

Attachments

On Sun, Jan 28, 2024, at 10:10 PM, Euler Taveira wrote:
> On Fri, Jan 26, 2024, at 4:55 AM, Hayato Kuroda (Fujitsu) wrote:
>> Again, thanks for updating the patch! There are my random comments for v9.
> 
> Thanks for checking v9. I already incorporated some of the points below into
> the next patch. Give me a couple of hours to include some important points.

Here it is another patch that includes the following changes:

* rename the tool to pg_createsubscriber: it was the name with the most votes
  [1].
* fix recovery-timeout option [2]
* refactor: split setup_publisher into check_publisher (that contains only GUC
  checks) and setup_publisher (that does what the name suggests) [2]
* doc: verbose option can be specified multiple times [2]
* typedefs.list: add LogicalRepInfo [2]
* fix: register cleanup routine after the data structure (dbinfo) is assigned
  [2]
* doc: add mandatory options to synopsis [3]
* refactor: rename some options (such as publisher-conninfo to
  publisher-server) to use the same pattern as pg_rewind.
* feat: remove publications from subscriber [2]
* feat: use temporary replication slot to get consistent LSN [3]
* refactor: move subscriber setup to its own function
* refactor: stop standby server to its own function [2]
* refactor: start standby server to its own function [2]
* fix: getopt options [4]

There is a few open items in my list. I included v10-0002. I had already
included a refactor to include start/stop functions so I didn't include
v10-0003. I'll check v10-0004 tomorrow.

One open item that is worrying me is how to handle the pg_ctl timeout. This
patch does nothing and the user should use PGCTLTIMEOUT environment variable to
avoid that the execution is canceled after 60 seconds (default for pg_ctl).
Even if you set a high value, it might not be enough for cases like
time-delayed replica. Maybe pg_ctl should accept no timeout as --timeout
option. I'll include this caveat into the documentation but I'm afraid it is
not sufficient and we should provide a better way to handle this situation.

[1] https://www.postgresql.org/message-id/b315c7da-7ab1-4014-a2a9-8ab6ae26017c%40app.fastmail.com
[2] https://www.postgresql.org/message-id/TY3PR01MB98895A551923953B3DA3C7C8F5792%40TY3PR01MB9889.jpnprd01.prod.outlook.com
[3] https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2@TY3PR01MB9889.jpnprd01.prod.outlook.com
[4] https://www.postgresql.org/message-id/TY3PR01MB98891E7735141FE8760CEC4AF57E2%40TY3PR01MB9889.jpnprd01.prod.outlook.com


--
Euler Taveira
EDB   https://www.enterprisedb.com/