Horton

Re: trying again to get incremental backup

Michael Banck <mbanck@gmx.net>

View thread View on postgr.es
From: Michael Banck <mbanck@gmx.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2024-10-28T21:10:58Z
Lists: pgsql-hackers 
Hi,

So I am a bit confused about the status of the tar format support, and
after re-reading the thread (or at least grepping it for ' tar '), this
wasn't really much discussed here either.

On Wed, Jun 14, 2023 at 02:46:48PM -0400, Robert Haas wrote:
> - We only know how to operate on directories, not tar files. I thought
> about that when working on pg_verifybackup as well, but I didn't do
> anything about it. It would be nice to go back and make that tool work
> on tar-format backups, and this one, too.

I believe "that tool" is pg_verifybackup, while "this one" is
pg_combinebackup? However, what's up with pg_basebackup itself with
respect to tar format incremental backups?

AFAICT (see below), pg_basebackup -Ft --incremental=foo/backup_manifest
happily creates an incremental backup in tar format; however,
pg_combinebackup will not be able to restore it? If that is the case,
shouldn't there be a bigger warning in the documentation about this, or
maybe pg_basebackup should refuse to make incremental tar-format backups
in the first place?

Am I missing something here? It will be obvious to users after the first
failure (to try to restore) that this will not work, and hopefully
everybody tests a restore before they put a backup solution into
production (or even better, wait until this whole feature is included in
a wholesale solution), but I wonder whether somebody might trip over
this after all and be unhappy. If one reads the pg_combinebackup
documentation carefully it kinda becomes obvious that it does occupy
itself with tar format backups, but it is not spelt out explicitly
either.

|postgres@mbanck-lin-1:~$ pg_basebackup -c fast -Ft -D backup/backup_full
|postgres@mbanck-lin-1:~$ pg_basebackup -c fast -Ft -D backup/backup_incr_1 --incremental=backup/backup_full/backup_manifest
|postgres@mbanck-lin-1:~$ echo $?
|0
|postgres@mbanck-lin-1:~$ du -h backup/
|44M	backup/backup_incr_1
|4,5G	backup/backup_full
|4,5G	backup/
|postgres@mbanck-lin-1:~$ tar tf backup/backup_incr_1/base.tar | grep INCR | head
|base/1/INCREMENTAL.3603
|base/1/INCREMENTAL.2187
|base/1/INCREMENTAL.13418
|base/1/INCREMENTAL.3467
|base/1/INCREMENTAL.2615_vm
|base/1/INCREMENTAL.2228
|base/1/INCREMENTAL.3503
|base/1/INCREMENTAL.2659
|base/1/INCREMENTAL.2607_vm
|base/1/INCREMENTAL.4164
|postgres@mbanck-lin-1:~$ /usr/lib/postgresql/17/bin/pg_combinebackup backup/backup_full/ backup/backup_incr_1/ -o backup/combined
|pg_combinebackup: error: could not open file "backup/backup_incr_1//PG_VERSION": No such file or directory


Michael

Commits

  1. Minor fixes to pg_combinebackup and its documentation.

  2. Fix defects in PrepareForIncrementalBackup.

  3. Add WALSummarizerLock to wait_event_names.txt

  4. Initialize variable to placate compiler.

  5. Replace nonsense comment with a relevant one.

  6. Fix numerous typos in incremental backup commits.

  7. Add support for incremental backup.

  8. Add a new WAL summarizer process.

  9. Move src/bin/pg_verifybackup/parse_manifest.c into src/common.

  10. Fix brown paper bag bug in 5c47c6546c413d5eb51c1626070a807026e6139d.

  11. Rename pg_verifybackup's JsonManifestParseContext callback functions.

  12. Rename JsonManifestParseContext callbacks.

  13. Change how a base backup decides which files have checksums.

  14. Change struct tablespaceinfo's oid member from 'char *' to 'Oid'

  15. Refactor parse_filename_for_nontemp_relation to parse more.

  16. During online checkpoints, insert XLOG_CHECKPOINT_REDO at redo point.

  17. In basebackup.c, refactor to create read_file_data_into_buffer.

  18. In basebackup.c, refactor to create verify_page_checksum.

  19. Report syncscan position at end of scan.

  20. Exclude additional directories in pg_basebackup

  21. Add new JSON processing functions and parser API.