Horton

Re: Switching XLog source from archive to streaming when primary available

Michael Paquier <michael@paquier.xyz>

View thread
From: Michael Paquier <michael@paquier.xyz>
To: "Andrey M. Borodin" <x4mmm@yandex-team.ru>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Nathan Bossart <nathandbossart@gmail.com>, Japin Li <japinli@hotmail.com>, Ian Lawrence Barwick <barwick@gmail.com>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>, Cary Huang <cary.huang@highgo.ca>, SATYANARAYANA NARLAPURAM <satyanarlapuram@gmail.com>, pgsql-hackers@lists.postgresql.org
Date: 2025-01-07T23:47:05Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add basic TAP tests for the low-level backup method, take two

  2. Add a failover option to subscriptions.

  3. Make all Perl warnings fatal

  4. Add PostgreSQL::Test::Cluster::advance_wal

  5. Rename wal_keep_segments to wal_keep_size.

On Thu, Jan 02, 2025 at 11:12:37PM +0500, Andrey M. Borodin wrote:
> In my observation restore from archive is many orders of magnitude
> faster than streaming replication. Advanced archive tools employ
> compression (x6 to speed), download parallelism (x4), are not
> constrained be primary's network limits (x3) and disk limits, do not
> depend on complicated FEBE protocol, etc.

This is a fair argument in terms of flexibility of what can be
achieved on a file-basis, yes, because you are not bottlenecked by
the existing replication protocol and can request them ahead of time
if necessary and can decide what you want within a single
restore_command or archive_command (or module for the latter).

It may be relevant to think in terms of what could be done at protocol
level to retrieve batches of WAL segments so as the backend has a
better control on how each segment is handled in a batch, or provide
better in-core tools to achieve that with the existing two command
GUCs for restore and archiving?  Nathan has also proposed a couple of
months ago restore modules, because relying on commands can be very
fancy in terms of error handling.  And we already have the archive
module part.
--
Michael