Horton

Re: archive modules

Nathan Bossart <bossartn@amazon.com>

View thread View on postgr.es
From: "Bossart, Nathan" <bossartn@amazon.com>
To: Fujii Masao <masao.fujii@oss.nttdata.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2021-11-02T15:03:11Z
Lists: pgsql-hackers 
On 11/1/21, 9:44 PM, "Fujii Masao" <masao.fujii@oss.nttdata.com> wrote:
> What is the main motivation of this patch? I was thinking that
> it's for parallelizing WAL archiving. But as far as I read
> the patch very briefly, WAL file name is still passed to
> the archive callback function one by one.

The main motivation is provide a way to archive without shelling out.
This reduces the amount of overhead, which can improve archival rate
significantly.  It should also make it easier to archive more safely.
For example, many of the common shell commands used for archiving
won't fsync the data, but it isn't too hard to do so via C.  The
current proposal doesn't introduce any extra infrastructure for
batching or parallelism, but it is probably still possible.  I would
like to eventually add batching, but for now I'm only focused on
introducing basic archive module support.

> Are you planning to extend this mechanism to other WAL
> archiving-related commands like restore_command? I can imagine
> that those who use archive library (rather than shell) would
> like to use the same mechanism for WAL restore.

I would like to do this eventually, but my current proposal is limited
to archive_command.

> I think that it's worth adding this module into core
> rather than handling it as test module. It provides very basic
> WAL archiving feature, but (I guess) it's enough for some users.

Do you think it should go into contrib?

Nathan

Commits

  1. Disallow setting archive_library and archive_command at the same time

  2. Restore archive_command documentation

  3. Doc: improve explanation of when custom GUCs appear in pg_settings.

  4. doc: Add note about re-archiving of same WAL files in docs.

  5. In basic_archive tests, insist on wal_level='replica'.

  6. Allow archiving via loadable modules.

  7. Move the code to archive files via the shell to a separate file.