Horton

Thread

Commits

  1. Remove server support for the previous base backup protocol.

  2. Remove server support for old BASE_BACKUP command syntax.

  3. Modify pg_basebackup to use a new COPY subprotocol for base backups.

  4. Flexible options for BASE_BACKUP.

  1. is the base backup protocol used by out-of-core tools?

    Robert Haas <robertmhaas@gmail.com> — 2022-02-08T16:26:41Z

    Hi,

Commit 0ba281cb4bf9f5f65529dfa4c8282abb734dd454 added a new syntax for
the BASE_BACKUP command, and commit
cc333f32336f5146b75190f57ef587dff225f565 added a new COPY subprotocol
for taking base backups. In both cases, I preserved backward
compatibility. However, nothing in PostgreSQL itself cares about this,
because if you try to run an older version of pg_basebackup against a
v15 server, it's going to fail anyway:

pg_basebackup: error: incompatible server version 15devel

From that point of view, there's no downside to removing from the
server the old syntax for BASE_BACKUP and the old protocol for taking
backups. We can't remove anything from pg_basebackup, because it is
our practice to make new versions of pg_basebackup work with old
versions of the server. But the reverse is not true: an older
pg_basebackup will categorically refuse to work with a newer server
version. Therefore keeping the code for this stuff around in the
server has no value ... unless there is out-of-core code that (a) uses
the BASE_BACKUP command and (b) wouldn't immediately adopt the new
syntax and protocol anyway. If there is, we might want to keep the
backward-compatibility code around in the server for a few releases.
If not, we should probably nuke that code to simplify things and
reduce the maintenance burden.

Patches for the nuking are attached. If nobody writes back, I'm going
to assume that means nobody cares, and commit these some time before
feature freeze. If one or more people do write back, then my plan is
to see what they have to say and proceed accordingly.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  2. Re: is the base backup protocol used by out-of-core tools?

    Justin Pryzby <pryzby@telsasoft.com> — 2022-02-08T18:03:15Z

    On Tue, Feb 08, 2022 at 11:26:41AM -0500, Robert Haas wrote:
> Patches for the nuking are attached. If nobody writes back, I'm going
> to assume that means nobody cares, and commit these some time before
> feature freeze. If one or more people do write back, then my plan is
> to see what they have to say and proceed accordingly.

I created this and added that for visibility and so it's not forgotten.
ISTM that could be done post-freeze, although I don't know if that's useful or
important.

https://wiki.postgresql.org/wiki/PostgreSQL_15_Open_Items

-- 
Justin

  3. Re: is the base backup protocol used by out-of-core tools?

    Robert Haas <robertmhaas@gmail.com> — 2022-02-08T18:39:12Z

    On Tue, Feb 8, 2022 at 1:03 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> I created this and added that for visibility and so it's not forgotten.
> ISTM that could be done post-freeze, although I don't know if that's useful or
> important.
>
> https://wiki.postgresql.org/wiki/PostgreSQL_15_Open_Items

Thanks. I feel like this is in the department of things that are
unlikely to be an issue for anyone, but could be. But the set of
people who could care is basically just backup tool authors, so
hopefully they'll notice this thread, or that list.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  4. Re: is the base backup protocol used by out-of-core tools?

    David Steele <david@pgmasters.net> — 2022-02-08T18:52:32Z

    On 2/8/22 12:39, Robert Haas wrote:
> On Tue, Feb 8, 2022 at 1:03 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
>> I created this and added that for visibility and so it's not forgotten.
>> ISTM that could be done post-freeze, although I don't know if that's useful or
>> important.
>>
>> https://wiki.postgresql.org/wiki/PostgreSQL_15_Open_Items
> 
> Thanks. I feel like this is in the department of things that are
> unlikely to be an issue for anyone, but could be. But the set of
> people who could care is basically just backup tool authors, so
> hopefully they'll notice this thread, or that list.

I'm aware of several tools that use pg_basebackup but they are using the 
command-line tool, not the server protocol directly.

Personally, I'm in favor of simplifying the code on the server side. If 
anyone is using the protocol directly (which I kind of doubt) I imagine 
they would want to take advantage of all the new features that have been 
added.

Regards,
-David

  5. Re: is the base backup protocol used by out-of-core tools?

    Robert Haas <robertmhaas@gmail.com> — 2022-02-08T18:59:16Z

    On Tue, Feb 8, 2022 at 1:52 PM David Steele <david@pgmasters.net> wrote:
> I'm aware of several tools that use pg_basebackup but they are using the
> command-line tool, not the server protocol directly.

Good to know.

> Personally, I'm in favor of simplifying the code on the server side. If
> anyone is using the protocol directly (which I kind of doubt) I imagine
> they would want to take advantage of all the new features that have been
> added.

I agree, and I'm glad you do too, but I thought it was worth asking.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  6. Re: is the base backup protocol used by out-of-core tools?

    Michael Paquier <michael@paquier.xyz> — 2022-02-09T02:21:41Z

    On Tue, Feb 08, 2022 at 11:26:41AM -0500, Robert Haas wrote:
> From that point of view, there's no downside to removing from the
> server the old syntax for BASE_BACKUP and the old protocol for taking
> backups. We can't remove anything from pg_basebackup, because it is
> our practice to make new versions of pg_basebackup work with old
> versions of the server. But the reverse is not true: an older
> pg_basebackup will categorically refuse to work with a newer server
> version. Therefore keeping the code for this stuff around in the
> server has no value ... unless there is out-of-core code that (a) uses
> the BASE_BACKUP command and (b) wouldn't immediately adopt the new
> syntax and protocol anyway. If there is, we might want to keep the
> backward-compatibility code around in the server for a few releases.
> If not, we should probably nuke that code to simplify things and
> reduce the maintenance burden.

This line of arguments looks sensible from here, so +1 for this
cleanup in the backend as of 15~.  I am not sure if we should worry
about out-of-core tools that use replication commands, either, and the
new grammar is easy to adapt to.

FWIW, one backup tool maintained by NTT is pg_rman, which does not use
the replication protocol AFAIK:
https://github.com/ossc-db/pg_rman
Perhaps Horiguchi-san or Fujita-san have an opinion on that.
--
Michael

  7. Re: is the base backup protocol used by out-of-core tools?

    Ian Lawrence Barwick <barwick@gmail.com> — 2022-02-09T02:42:18Z

    2022年2月9日(水) 11:21 Michael Paquier <michael@paquier.xyz>:
>
> On Tue, Feb 08, 2022 at 11:26:41AM -0500, Robert Haas wrote:
> > From that point of view, there's no downside to removing from the
> > server the old syntax for BASE_BACKUP and the old protocol for taking
> > backups. We can't remove anything from pg_basebackup, because it is
> > our practice to make new versions of pg_basebackup work with old
> > versions of the server. But the reverse is not true: an older
> > pg_basebackup will categorically refuse to work with a newer server
> > version. Therefore keeping the code for this stuff around in the
> > server has no value ... unless there is out-of-core code that (a) uses
> > the BASE_BACKUP command and (b) wouldn't immediately adopt the new
> > syntax and protocol anyway. If there is, we might want to keep the
> > backward-compatibility code around in the server for a few releases.
> > If not, we should probably nuke that code to simplify things and
> > reduce the maintenance burden.
>
> This line of arguments looks sensible from here, so +1 for this
> cleanup in the backend as of 15~.  I am not sure if we should worry
> about out-of-core tools that use replication commands, either, and the
> new grammar is easy to adapt to.

FWIW the out-of-core utility I have some involvement in and which uses base
backup functionality, uses this by executing pg_basebackup, so no problem
there. But even if it used the replication protocol directly, it'd
just be a case of
adding another bit of Pg version-specific code handling; I certainly wouldn't
expect core to hold itself back for my convenience.

Regards

Ian Barwick

-- 
EnterpriseDB: https://www.enterprisedb.com

  8. Re: is the base backup protocol used by out-of-core tools?

    Kyotaro Horiguchi <horikyota.ntt@gmail.com> — 2022-02-09T08:14:43Z

    At Wed, 9 Feb 2022 11:21:41 +0900, Michael Paquier <michael@paquier.xyz> wrote in 
> On Tue, Feb 08, 2022 at 11:26:41AM -0500, Robert Haas wrote:
> > From that point of view, there's no downside to removing from the
> > server the old syntax for BASE_BACKUP and the old protocol for taking
> > backups. We can't remove anything from pg_basebackup, because it is
> > our practice to make new versions of pg_basebackup work with old
> > versions of the server. But the reverse is not true: an older
> > pg_basebackup will categorically refuse to work with a newer server
> > version. Therefore keeping the code for this stuff around in the
> > server has no value ... unless there is out-of-core code that (a) uses
> > the BASE_BACKUP command and (b) wouldn't immediately adopt the new
> > syntax and protocol anyway. If there is, we might want to keep the
> > backward-compatibility code around in the server for a few releases.
> > If not, we should probably nuke that code to simplify things and
> > reduce the maintenance burden.
> 
> This line of arguments looks sensible from here, so +1 for this
> cleanup in the backend as of 15~.  I am not sure if we should worry
> about out-of-core tools that use replication commands, either, and the
> new grammar is easy to adapt to.
> 
> FWIW, one backup tool maintained by NTT is pg_rman, which does not use
> the replication protocol AFAIK:
> https://github.com/ossoc-db/pg_rman
> Perhaps Horiguchi-san or Fujita-san have an opinion on that.

# Oh, the excessive 'o' perplexed me:p

Thanks for pining.  AFAICS, as you see, pg_rman doesn't talk
basebackup protocol (nor even pg_basebackup command) as it supports
inremental backup.  So there's no issue about the removal of old
syntax on our side.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  9. Re: is the base backup protocol used by out-of-core tools?

    Robert Haas <robertmhaas@gmail.com> — 2022-02-09T13:01:41Z

    On Wed, Feb 9, 2022 at 3:14 AM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
> Thanks for pining.  AFAICS, as you see, pg_rman doesn't talk
> basebackup protocol (nor even pg_basebackup command) as it supports
> inremental backup.  So there's no issue about the removal of old
> syntax on our side.

Cool. Since the verdict here seems pretty unanimous and we've heard
from a good number of backup tool authors, I'll give this another day
or so and then commit, unless objections emerge before then.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  10. Re: is the base backup protocol used by out-of-core tools?

    Bernd Helmle <mailings@oopsware.de> — 2022-02-09T13:50:58Z

    Am Mittwoch, dem 09.02.2022 um 08:01 -0500 schrieb Robert Haas:
> On Wed, Feb 9, 2022 at 3:14 AM Kyotaro Horiguchi
> <horikyota.ntt@gmail.com> wrote:
> > Thanks for pining.  AFAICS, as you see, pg_rman doesn't talk
> > basebackup protocol (nor even pg_basebackup command) as it supports
> > inremental backup.  So there's no issue about the removal of old
> > syntax on our side.
> 
> Cool. Since the verdict here seems pretty unanimous and we've heard
> from a good number of backup tool authors, I'll give this another day
> or so and then commit, unless objections emerge before then.
> 

A while ago i worked on a tool which talks the streaming protocol
directly. In the past there were already additions to the protocol
(like MANIFEST), so to support those optional features and multiple
major versions you already have to abstract the code to deal with that.

Having said that, i don't have any objections either to remove the old
protocol support.

Thanks
	Bernd

  11. Re: is the base backup protocol used by out-of-core tools?

    Magnus Hagander <magnus@hagander.net> — 2022-02-09T20:05:02Z

    On Wed, Feb 9, 2022 at 2:02 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Wed, Feb 9, 2022 at 3:14 AM Kyotaro Horiguchi
> <horikyota.ntt@gmail.com> wrote:
> > Thanks for pining.  AFAICS, as you see, pg_rman doesn't talk
> > basebackup protocol (nor even pg_basebackup command) as it supports
> > inremental backup.  So there's no issue about the removal of old
> > syntax on our side.
>
> Cool. Since the verdict here seems pretty unanimous and we've heard
> from a good number of backup tool authors, I'll give this another day
> or so and then commit, unless objections emerge before then.

Late to the game, but here's another +1.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

  12. Re: is the base backup protocol used by out-of-core tools?

    Robert Haas <robertmhaas@gmail.com> — 2022-02-10T20:28:05Z

    On Wed, Feb 9, 2022 at 3:05 PM Magnus Hagander <magnus@hagander.net> wrote:
> Late to the game, but here's another +1.

Not that late, thanks for chiming in.

Looks pretty unanimous, so I have committed the patches.

-- 
Robert Haas
EDB: http://www.enterprisedb.com