Horton

Thread

Commits

  1. Adjust documentation for syncfs().

  2. Allow using syncfs() in frontend utilities.

  1. pgsql: Allow using syncfs() in frontend utilities.

    Nathan Bossart <nathan@postgresql.org> — 2023-09-06T23:28:00Z

    Allow using syncfs() in frontend utilities.

This commit allows specifying a --sync-method in several frontend
utilities that must synchronize many files to disk (initdb,
pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade).
On Linux, users can specify "syncfs" to synchronize the relevant
file systems instead of calling fsync() for every single file.  In
many cases, using syncfs() is much faster.

As with recovery_init_sync_method, this new option comes with some
caveats.  The descriptions of these caveats have been moved to a
new appendix section in the documentation.

Co-authored-by: Justin Pryzby
Reviewed-by: Michael Paquier, Thomas Munro, Robert Haas, Justin Pryzby
Discussion: https://postgr.es/m/20210930004340.GM831%40telsasoft.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/8c16ad3b43299695f203f9157a2b27c22b9ed634

Modified Files
--------------
doc/src/sgml/config.sgml              | 12 +++---------
doc/src/sgml/filelist.sgml            |  1 +
doc/src/sgml/postgres.sgml            |  1 +
doc/src/sgml/ref/initdb.sgml          | 22 +++++++++++++++++++++
doc/src/sgml/ref/pg_basebackup.sgml   | 25 ++++++++++++++++++++++++
doc/src/sgml/ref/pg_checksums.sgml    | 22 +++++++++++++++++++++
doc/src/sgml/ref/pg_dump.sgml         | 21 ++++++++++++++++++++
doc/src/sgml/ref/pg_rewind.sgml       | 22 +++++++++++++++++++++
doc/src/sgml/ref/pgupgrade.sgml       | 23 ++++++++++++++++++++++
doc/src/sgml/syncfs.sgml              | 36 +++++++++++++++++++++++++++++++++++
src/bin/initdb/initdb.c               |  6 ++++++
src/bin/initdb/t/001_initdb.pl        | 12 ++++++++++++
src/bin/pg_basebackup/pg_basebackup.c |  7 +++++++
src/bin/pg_checksums/pg_checksums.c   |  6 ++++++
src/bin/pg_dump/pg_dump.c             |  7 +++++++
src/bin/pg_rewind/pg_rewind.c         |  8 ++++++++
src/bin/pg_upgrade/option.c           | 13 +++++++++++++
src/bin/pg_upgrade/pg_upgrade.c       |  6 ++++--
src/bin/pg_upgrade/pg_upgrade.h       |  1 +
src/fe_utils/option_utils.c           | 27 ++++++++++++++++++++++++++
src/include/fe_utils/option_utils.h   |  4 ++++
21 files changed, 271 insertions(+), 11 deletions(-)

  2. Re: pgsql: Allow using syncfs() in frontend utilities.

    Robert Haas <robertmhaas@gmail.com> — 2024-03-22T16:52:15Z

    On Wed, Sep 6, 2023 at 7:28 PM Nathan Bossart <nathan@postgresql.org> wrote:
> Allow using syncfs() in frontend utilities.
>
> This commit allows specifying a --sync-method in several frontend
> utilities that must synchronize many files to disk (initdb,
> pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade).
> On Linux, users can specify "syncfs" to synchronize the relevant
> file systems instead of calling fsync() for every single file.  In
> many cases, using syncfs() is much faster.
>
> As with recovery_init_sync_method, this new option comes with some
> caveats.  The descriptions of these caveats have been moved to a
> new appendix section in the documentation.

Hi,

I'd like to complain about this commit's addition of a new appendix. I
do understand the temptation to document caveats like this centrally
instead of in multiple places, but as I've been complaining about over
in the "documentation structure" thread, our top-level documentation
index is too big, and I feel strongly that we need to de-clutter it
rather than cluttering it further. This added a new chapter which is
just 5 sentences long. I understand that this was done because the
same issue applies to a bunch of different utilities and we didn't
want to duplicate this text in all of those places, but I feel like
this approach just doesn't scale. If we did this in every place where
we have this much text that we want to avoid duplicating, we'd soon
have hundreds of appendixes.

What I would suggest we do instead is pick one of the places where
this comes up and document it there, perhaps the
recovery_init_sync_method GUC. And then make the documentation for the
other say something like, you know those issues we documented for
recovery_init_sync_method? Well they also apply to this.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  3. Re: pgsql: Allow using syncfs() in frontend utilities.

    Peter Eisentraut <peter@eisentraut.org> — 2024-03-26T10:18:57Z

    On 22.03.24 17:52, Robert Haas wrote:
> On Wed, Sep 6, 2023 at 7:28 PM Nathan Bossart <nathan@postgresql.org> wrote:
>> Allow using syncfs() in frontend utilities.
>>
>> This commit allows specifying a --sync-method in several frontend
>> utilities that must synchronize many files to disk (initdb,
>> pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade).
>> On Linux, users can specify "syncfs" to synchronize the relevant
>> file systems instead of calling fsync() for every single file.  In
>> many cases, using syncfs() is much faster.
>>
>> As with recovery_init_sync_method, this new option comes with some
>> caveats.  The descriptions of these caveats have been moved to a
>> new appendix section in the documentation.
> 
> Hi,
> 
> I'd like to complain about this commit's addition of a new appendix.

I already complained about that at 
<https://www.postgresql.org/message-id/42804669-7063-1320-ed37-3226d5f1067d@eisentraut.org> 
and some follow-up was announced but didn't happen.  It was on my list 
to look into cleaning up during beta.

  4. Re: pgsql: Allow using syncfs() in frontend utilities.

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-26T14:52:10Z

    On Fri, Mar 22, 2024 at 12:52:15PM -0400, Robert Haas wrote:
> I'd like to complain about this commit's addition of a new appendix. I
> do understand the temptation to document caveats like this centrally
> instead of in multiple places, but as I've been complaining about over
> in the "documentation structure" thread, our top-level documentation
> index is too big, and I feel strongly that we need to de-clutter it
> rather than cluttering it further. This added a new chapter which is
> just 5 sentences long. I understand that this was done because the
> same issue applies to a bunch of different utilities and we didn't
> want to duplicate this text in all of those places, but I feel like
> this approach just doesn't scale. If we did this in every place where
> we have this much text that we want to avoid duplicating, we'd soon
> have hundreds of appendixes.

Sorry I missed this.  I explored a couple of options last year but the
discussion trailed off [0].

> What I would suggest we do instead is pick one of the places where
> this comes up and document it there, perhaps the
> recovery_init_sync_method GUC. And then make the documentation for the
> other say something like, you know those issues we documented for
> recovery_init_sync_method? Well they also apply to this.

WFM.  I'll put together a patch.

[0] https://postgr.es/m/20231009204823.GA659480%40nathanxps13

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  5. Re: pgsql: Allow using syncfs() in frontend utilities.

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-26T15:11:31Z

    On Tue, Mar 26, 2024 at 11:18:57AM +0100, Peter Eisentraut wrote:
> On 22.03.24 17:52, Robert Haas wrote:
>> I'd like to complain about this commit's addition of a new appendix.
> 
> I already complained about that at <https://www.postgresql.org/message-id/42804669-7063-1320-ed37-3226d5f1067d@eisentraut.org>
> and some follow-up was announced but didn't happen.  It was on my list to
> look into cleaning up during beta.

Sorry about this, I lost track of it.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  6. Re: pgsql: Allow using syncfs() in frontend utilities.

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-26T16:34:49Z

    On Tue, Mar 26, 2024 at 10:11:31AM -0500, Nathan Bossart wrote:
> On Tue, Mar 26, 2024 at 11:18:57AM +0100, Peter Eisentraut wrote:
>> On 22.03.24 17:52, Robert Haas wrote:
>>> I'd like to complain about this commit's addition of a new appendix.
>> 
>> I already complained about that at <https://www.postgresql.org/message-id/42804669-7063-1320-ed37-3226d5f1067d@eisentraut.org>
>> and some follow-up was announced but didn't happen.  It was on my list to
>> look into cleaning up during beta.
> 
> Sorry about this, I lost track of it.

Here's a first attempt at a patch based on Robert's suggestion from
upthread.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  7. Re: pgsql: Allow using syncfs() in frontend utilities.

    Robert Haas <robertmhaas@gmail.com> — 2024-03-27T14:41:42Z

    On Tue, Mar 26, 2024 at 12:34 PM Nathan Bossart
<nathandbossart@gmail.com> wrote:
> Here's a first attempt at a patch based on Robert's suggestion from
> upthread.

WFM.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  8. Re: pgsql: Allow using syncfs() in frontend utilities.

    Nathan Bossart <nathandbossart@gmail.com> — 2024-03-27T15:25:16Z

    On Wed, Mar 27, 2024 at 10:41:42AM -0400, Robert Haas wrote:
> On Tue, Mar 26, 2024 at 12:34 PM Nathan Bossart
> <nathandbossart@gmail.com> wrote:
>> Here's a first attempt at a patch based on Robert's suggestion from
>> upthread.
> 
> WFM.

Committed.  Again, I apologize this took so long.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  9. Re: pgsql: Allow using syncfs() in frontend utilities.

    Robert Haas <robertmhaas@gmail.com> — 2024-03-27T15:27:46Z

    On Wed, Mar 27, 2024 at 11:25 AM Nathan Bossart
<nathandbossart@gmail.com> wrote:
> Committed.  Again, I apologize this took so long.

No worries from my side; I only noticed recently. I guess Peter's been
waiting a while, though. Thanks for committing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com