Horton

Thread

Commits

  1. Remove documentation of waiting restore_command.

  2. Retire pg_standby.

  3. doc: Fix typos

  1. [PATCH] remove pg_archivecleanup and pg_standby

    Justin Pryzby <pryzby@telsasoft.com> — 2020-10-29T02:44:12Z

    Forking this thread:
https://www.postgresql.org/message-id/fd93f1c5-7818-a02c-01e5-1075ac0d4def@iki.fi

I think these are old-fashioned since 9.6 (?), so remove them for v14.

I found it confusing when re-familiarizing myself with modern streaming
replication that there are extensions which only help do things the "old way".

  2. Re: [PATCH] remove pg_archivecleanup and pg_standby

    Michael Banck <michael.banck@credativ.de> — 2020-10-29T19:40:31Z

    Hi,

Am Mittwoch, den 28.10.2020, 21:44 -0500 schrieb Justin Pryzby:
> Forking this thread:
> https://www.postgresql.org/message-id/fd93f1c5-7818-a02c-01e5-1075ac0d4def@iki.fi

Glancing over this in the context of pg_standby/pg_archivecleanup, I am
not sure Heikki's "Ditto" is about "remove pg_archivecleanup just like
pg_standby" or rather "keep the note until we get around doing something
with it". Probably the former, but see below.

> I think these are old-fashioned since 9.6 (?), so remove them for v14.

Why 9.6?

> I found it confusing when re-familiarizing myself with modern streaming
> replication that there are extensions which only help do things the "old way".

I guess not many will complain about pg_standby going away, but I am
under the impression that pg_archivecleanup is still used a lot in PITR
backup environments as a handy tool to expire WAL related to expired
base backups. I certainly saw hand-assembled shell code fail with "too
many files" and things when it tried to act on large amount of WAL.

So I think the part about it being used in archive_cleanup_command can
be probably be removed, but the part about it being useful as a stand-
alone tool, in particular this part:

|In this mode, if you specify a .partial or .backup file name, then only

|the file prefix will be used as the oldestkeptwalfile. This treatment
|of .backup file name allows you to remove all WAL files archived prior
|to a specific base backup without error.

At the very least, the commit message should give a rationale on why
pg_archivebackup is retired, and what it should be replaced with, in
case valid use-cases for it are still present.


Michael

-- 
Michael Banck
Projektleiter / Senior Berater
Tel.: +49 2166 9901-171
Fax:  +49 2166 9901-100
Email: michael.banck@credativ.de

credativ GmbH, HRB Mönchengladbach 12080
USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer

Unser Umgang mit personenbezogenen Daten unterliegt
folgenden Bestimmungen: https://www.credativ.de/datenschutz

  3. Re: [PATCH] remove pg_archivecleanup and pg_standby

    Justin Pryzby <pryzby@telsasoft.com> — 2020-11-02T18:26:21Z

    On Thu, Oct 29, 2020 at 08:40:31PM +0100, Michael Banck wrote:
> Am Mittwoch, den 28.10.2020, 21:44 -0500 schrieb Justin Pryzby:
> > Forking this thread:
> > https://www.postgresql.org/message-id/fd93f1c5-7818-a02c-01e5-1075ac0d4def@iki.fi

> > I think these are old-fashioned since 9.6 (?), so remove them for v14.
> 
> Why 9.6?

My work doesn't currently bring me in contact with replication, so I've had to
dig through release notes.  I think streaming replication was new in 9.0, and
increasingly mature throughout 9.x.  Maybe someone else will say a different
release was when streaming replication became the norm and wal shipping old.

> > I found it confusing when re-familiarizing myself with modern streaming
> > replication that there are extensions which only help do things the "old way".
> 
> I guess not many will complain about pg_standby going away, but I am
> under the impression that pg_archivecleanup is still used a lot in PITR
> backup environments as a handy tool to expire WAL related to expired
> base backups. I certainly saw hand-assembled shell code fail with "too
> many files" and things when it tried to act on large amount of WAL.

I anticipate you're right, and I'll withdraw 0002.

-- 
Justin

  4. Re: [PATCH] remove pg_archivecleanup and pg_standby

    Heikki Linnakangas <hlinnaka@iki.fi> — 2020-11-03T15:28:46Z

    On 02/11/2020 20:26, Justin Pryzby wrote:
> On Thu, Oct 29, 2020 at 08:40:31PM +0100, Michael Banck wrote:
>> Am Mittwoch, den 28.10.2020, 21:44 -0500 schrieb Justin Pryzby:
>>> Forking this thread:
>>> https://www.postgresql.org/message-id/fd93f1c5-7818-a02c-01e5-1075ac0d4def@iki.fi
> 
>>> I think these are old-fashioned since 9.6 (?), so remove them for v14.
>>
>> Why 9.6?
> 
> My work doesn't currently bring me in contact with replication, so I've had to
> dig through release notes.  I think streaming replication was new in 9.0, and
> increasingly mature throughout 9.x.  Maybe someone else will say a different
> release was when streaming replication became the norm and wal shipping old.

Removing pg_standby has been proposed a couple of times in the past. See 
https://www.postgresql.org/message-id/20170913064824.rqflkadxwpboabgw@alap3.anarazel.de 
for the latest attempt.

Masao-san, back in 2014 you mentioned "fast failover" as a feature that 
was missing from the built-in standby mode 
(https://www.postgresql.org/message-id/CAHGQGwEE_8vvpQk0ex6Qa_aXt-OSJ7OdZjX4uM_FtqKfxq5SbQ%40mail.gmail.com). 
I think that's been implemented since, with the recovery_target 
settings. Would you agree?

I'm pretty sure we can remove pg_standby by now. But if there's 
something crucial missing from the built-in facilities, we need to talk 
about implementing them.

- Heikki

  5. Re: [PATCH] remove pg_archivecleanup and pg_standby

    Robert Haas <robertmhaas@gmail.com> — 2020-11-03T16:49:02Z

    On Thu, Oct 29, 2020 at 3:40 PM Michael Banck <michael.banck@credativ.de> wrote:
> I guess not many will complain about pg_standby going away, but I am
> under the impression that pg_archivecleanup is still used a lot in PITR
> backup environments as a handy tool to expire WAL related to expired
> base backups. I certainly saw hand-assembled shell code fail with "too
> many files" and things when it tried to act on large amount of WAL.

Yeah, I see pg_archivecleanup used in customer environments all the
time. Like just this morning, for example.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  6. Re: [PATCH] remove pg_archivecleanup and pg_standby

    Michael Paquier <michael@paquier.xyz> — 2020-11-16T08:10:05Z

    On Tue, Nov 03, 2020 at 05:28:46PM +0200, Heikki Linnakangas wrote:
> Removing pg_standby has been proposed a couple of times in the past. See https://www.postgresql.org/message-id/20170913064824.rqflkadxwpboabgw@alap3.anarazel.de
> for the latest attempt.
> 
> Masao-san, back in 2014 you mentioned "fast failover" as a feature that was
> missing from the built-in standby mode (https://www.postgresql.org/message-id/CAHGQGwEE_8vvpQk0ex6Qa_aXt-OSJ7OdZjX4uM_FtqKfxq5SbQ%40mail.gmail.com).
> I think that's been implemented since, with the recovery_target settings.
> Would you agree?
> 
> I'm pretty sure we can remove pg_standby by now. But if there's something
> crucial missing from the built-in facilities, we need to talk about
> implementing them.

Reading the thread you are mentioning, it seems to me that the
statu-quo is the same, but I find rather scary that this tool is used
in exactly zero tests.

Echoing with Robert, I think that pg_archivecleanup is still useful in
many cases, so that's not something we should remove.
--
Michael

  7. Re: [PATCH] remove pg_archivecleanup and pg_standby

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2020-11-20T16:26:54Z

    On 2020-10-29 03:44, Justin Pryzby wrote:
> diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml
> index 4e833d79ef..be4292ec33 100644
> --- a/doc/src/sgml/contrib.sgml
> +++ b/doc/src/sgml/contrib.sgml
> @@ -199,6 +199,5 @@ pages.
>      part of the core <productname>PostgreSQL</productname> distribution.
>     </para>
>   
> - &pgstandby;
>    </sect1>
>   </appendix>

With this removal, that section becomes empty.  So you probably want to 
clean up or reorganize this a bit.

See https://www.postgresql.org/docs/devel/contrib-prog.html for the context.

  8. Re: [PATCH] remove pg_standby

    Justin Pryzby <pryzby@telsasoft.com> — 2020-11-21T19:41:05Z

    On Fri, Nov 20, 2020 at 05:26:54PM +0100, Peter Eisentraut wrote:
> On 2020-10-29 03:44, Justin Pryzby wrote:
> > diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml
> > index 4e833d79ef..be4292ec33 100644
> > --- a/doc/src/sgml/contrib.sgml
> > +++ b/doc/src/sgml/contrib.sgml
> > @@ -199,6 +199,5 @@ pages.
> >      part of the core <productname>PostgreSQL</productname> distribution.
> >     </para>
> > - &pgstandby;
> >    </sect1>
> >   </appendix>
> 
> With this removal, that section becomes empty.  So you probably want to
> clean up or reorganize this a bit.
> 
> See https://www.postgresql.org/docs/devel/contrib-prog.html for the context.

Oops.  I guess I'd write something like this.  If we just remove it, then
there'd no place to add a new server application, and "client applications"
would be the only subsection.

-- 
Justin

  9. Re: [PATCH] remove pg_standby

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2020-11-25T09:04:12Z

    On 2020-11-21 20:41, Justin Pryzby wrote:
> On Fri, Nov 20, 2020 at 05:26:54PM +0100, Peter Eisentraut wrote:
>> On 2020-10-29 03:44, Justin Pryzby wrote:
>>> diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml
>>> index 4e833d79ef..be4292ec33 100644
>>> --- a/doc/src/sgml/contrib.sgml
>>> +++ b/doc/src/sgml/contrib.sgml
>>> @@ -199,6 +199,5 @@ pages.
>>>       part of the core <productname>PostgreSQL</productname> distribution.
>>>      </para>
>>> - &pgstandby;
>>>     </sect1>
>>>    </appendix>
>>
>> With this removal, that section becomes empty.  So you probably want to
>> clean up or reorganize this a bit.
>>
>> See https://www.postgresql.org/docs/devel/contrib-prog.html for the context.
> 
> Oops.  I guess I'd write something like this.  If we just remove it, then
> there'd no place to add a new server application, and "client applications"
> would be the only subsection.

I have committed the typo fix.  I don't have a well-formed opinion yet 
about whether all the reservations about removing pg_standby have been 
addressed.

  10. Re: [PATCH] remove pg_standby

    Thomas Munro <thomas.munro@gmail.com> — 2021-01-27T03:13:24Z

    On Wed, Nov 25, 2020 at 10:04 PM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
> On 2020-11-21 20:41, Justin Pryzby wrote:
> > Oops.  I guess I'd write something like this.  If we just remove it, then
> > there'd no place to add a new server application, and "client applications"
> > would be the only subsection.
>
> I have committed the typo fix.  I don't have a well-formed opinion yet
> about whether all the reservations about removing pg_standby have been
> addressed.

I would like to commit this, because "waiting restore commands" have
confusing interactions with my proposed prefetching-during-recovery
patch[1].  Here's a version that fixes an error when building the docs
(there was a stray remaining <xref linkend="pgstandby"/>), and adds a
commit message.  Any objections?

Furthermore, I think we should also remove the section of the manual
that describes how to write your own "waiting restore command".
Thoughts?

[1] https://www.postgresql.org/message-id/flat/CA%2BhUKGKFeYPL9K%2BSRixcsx1%2B6HsHhqK%2BPOZyrnnZjw1jERpGcQ%40mail.gmail.com

  11. Re: [PATCH] remove pg_standby

    Michael Paquier <michael@paquier.xyz> — 2021-01-27T05:06:35Z

    On Wed, Jan 27, 2021 at 04:13:24PM +1300, Thomas Munro wrote:
> I would like to commit this, because "waiting restore commands" have
> confusing interactions with my proposed prefetching-during-recovery
> patch[1].  Here's a version that fixes an error when building the docs
> (there was a stray remaining <xref linkend="pgstandby"/>), and adds a
> commit message.  Any objections?

It looks like you are missing two references in your patch set:
$ git grep pg_standby
doc/src/sgml/high-availability.sgml:     Do not use pg_standby or
similar tools with the built-in standby mode
src/backend/access/transam/xlog.c:   * segment. Only recycle normal
files, pg_standby for example can create

The logic assumed in RemoveXlogFile() is actually a bit scary.  I have
not checked in details but it could be possible to clean up more code
in this area?

> Furthermore, I think we should also remove the section of the manual
> that describes how to write your own "waiting restore command".
> Thoughts?

Agreed.  No objections to that.
--
Michael

  12. Re: [PATCH] remove pg_standby

    Thomas Munro <thomas.munro@gmail.com> — 2021-01-27T05:32:35Z

    On Wed, Jan 27, 2021 at 6:06 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Wed, Jan 27, 2021 at 04:13:24PM +1300, Thomas Munro wrote:
> > I would like to commit this, because "waiting restore commands" have
> > confusing interactions with my proposed prefetching-during-recovery
> > patch[1].  Here's a version that fixes an error when building the docs
> > (there was a stray remaining <xref linkend="pgstandby"/>), and adds a
> > commit message.  Any objections?
>
> It looks like you are missing two references in your patch set:
> $ git grep pg_standby
> doc/src/sgml/high-availability.sgml:     Do not use pg_standby or
> similar tools with the built-in standby mode
> src/backend/access/transam/xlog.c:   * segment. Only recycle normal
> files, pg_standby for example can create

Thanks, fixed.

> The logic assumed in RemoveXlogFile() is actually a bit scary.  I have
> not checked in details but it could be possible to clean up more code
> in this area?

I think the check that it's a regular file is a good idea anyway, but
I removed the offending comment.

> > Furthermore, I think we should also remove the section of the manual
> > that describes how to write your own "waiting restore command".
> > Thoughts?
>
> Agreed.  No objections to that.

Thanks!

  13. Re: [PATCH] remove pg_standby

    Fujii Masao <masao.fujii@oss.nttdata.com> — 2021-01-27T08:08:56Z

    
On 2021/01/27 14:32, Thomas Munro wrote:
> On Wed, Jan 27, 2021 at 6:06 PM Michael Paquier <michael@paquier.xyz> wrote:
>> On Wed, Jan 27, 2021 at 04:13:24PM +1300, Thomas Munro wrote:
>>> I would like to commit this, because "waiting restore commands" have
>>> confusing interactions with my proposed prefetching-during-recovery
>>> patch[1].  Here's a version that fixes an error when building the docs
>>> (there was a stray remaining <xref linkend="pgstandby"/>), and adds a
>>> commit message.  Any objections?

I agree with this direction (i.e, remove pg_standby). BTW last month when I gave the talk about possible retire of pg_standby at PostgreSQL Unconference Tokyo, no one in audience complained about that retire.

But one question is; shouldn't we follow "usual" way to retire the feature instead of dropping that immediately? That is, mark pg_standby as obsolete, announce that pg_standby will be dropped after several releases, and then drop pg_standby. This seems safe because there might be some users. While it's been marked as obsolete, maybe WAL prefetch feature doesn't work with pg_standby, but we can live with that because it's obsolete.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

  14. Re: [PATCH] remove pg_standby

    Michael Paquier <michael@paquier.xyz> — 2021-01-28T07:36:39Z

    On Wed, Jan 27, 2021 at 05:08:56PM +0900, Fujii Masao wrote:
> But one question is; shouldn't we follow "usual" way to retire the
> feature instead of dropping that immediately? That is, mark
> pg_standby as obsolete, announce that pg_standby will be dropped
> after several releases, and then drop pg_standby. This seems safe
> because there might be some users. While it's been marked as
> obsolete, maybe WAL prefetch feature doesn't work with pg_standby,
> but we can live with that because it's obsolete.

Thanks.  FWIW, at this stage, my take is just to move on and remove
it.  If we mark that as obsolete, it will stay around forever while
annoying future development.
--
Michael

  15. Re: [PATCH] remove pg_standby

    Thomas Munro <thomas.munro@gmail.com> — 2021-01-28T22:13:03Z

    On Thu, Jan 28, 2021 at 8:36 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Wed, Jan 27, 2021 at 05:08:56PM +0900, Fujii Masao wrote:
> > But one question is; shouldn't we follow "usual" way to retire the
> > feature instead of dropping that immediately? That is, mark
> > pg_standby as obsolete, announce that pg_standby will be dropped
> > after several releases, and then drop pg_standby. This seems safe
> > because there might be some users. While it's been marked as
> > obsolete, maybe WAL prefetch feature doesn't work with pg_standby,
> > but we can live with that because it's obsolete.
>
> Thanks.  FWIW, at this stage, my take is just to move on and remove
> it.  If we mark that as obsolete, it will stay around forever while
> annoying future development.

I agree.  Also, this thing is entirely separate from the server, so a
hypothetical user who really wants to upgrade to 14 but keep using
pg_standby a bit longer could always use the version that shipped with
13.

  16. Re: [PATCH] remove pg_standby

    Thomas Munro <thomas.munro@gmail.com> — 2021-01-29T01:19:22Z

    On Fri, Jan 29, 2021 at 11:13 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Thu, Jan 28, 2021 at 8:36 PM Michael Paquier <michael@paquier.xyz> wrote:
> > On Wed, Jan 27, 2021 at 05:08:56PM +0900, Fujii Masao wrote:
> > > But one question is; shouldn't we follow "usual" way to retire the
> > > feature instead of dropping that immediately? That is, mark
> > > pg_standby as obsolete, announce that pg_standby will be dropped
> > > after several releases, and then drop pg_standby. This seems safe
> > > because there might be some users. While it's been marked as
> > > obsolete, maybe WAL prefetch feature doesn't work with pg_standby,
> > > but we can live with that because it's obsolete.
> >
> > Thanks.  FWIW, at this stage, my take is just to move on and remove
> > it.  If we mark that as obsolete, it will stay around forever while
> > annoying future development.
>
> I agree.  Also, this thing is entirely separate from the server, so a
> hypothetical user who really wants to upgrade to 14 but keep using
> pg_standby a bit longer could always use the version that shipped with
> 13.

And, pushed.