Horton

Re: refactoring basebackup.c

Robert Haas <robertmhaas@gmail.com>

View thread
From: Robert Haas <robertmhaas@gmail.com>
To: Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>
Cc: Mark Dilger <mark.dilger@enterprisedb.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>, tushar <tushar.ahuja@enterprisedb.com>
Date: 2021-11-05T15:50:01Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Document BaseBackupSync and BaseBackupWrite wait events.

  2. Support long distance matching for zstd compression

  3. Fix possible NULL-pointer-deference in backup_compression.c.

  4. Allow parallel zstd compression when taking a base backup.

  5. Make PostgreSQL::Test::Cluster::run_log() return a useful value.

  6. Fix a few goofs in new backup compression code.

  7. Replace BASE_BACKUP COMPRESSION_LEVEL option with COMPRESSION_DETAIL.

  8. Add 'basebackup_to_shell' contrib module.

  9. Allow extensions to add new backup targets.

  10. Change HAVE_LIBLZ4 and HAVE_LIBZSTD tests to USE_LZ4 and USE_ZSTD.

  11. pg_basebackup: Clean up some bogus file extension tests.

  12. pg_basebackup: Avoid unclean failure with server-compression and -D -.

  13. Fix LZ4 tests for remaining buffer space.

  14. Add support for zstd base backup compression.

  15. pg_basebackup: Allow client-side LZ4 (de)compression.

  16. Add suport for server-side LZ4 base backup compression.

  17. Add min() and max() aggregates for xid8.

  18. Remove superfluous variable.

  19. pg_basebackup: Cleaner handling when compression is multiply specified.

  20. Allow server-side compression to be used with -Fp.

  21. pg_basebackup: Fix a couple of recently-introduced bugs.

  22. Tidy up a few cosmetic issues related to pg_basebackup.

  23. Server-side gzip compression.

  24. Unbreak pg_basebackup/t/010_pg_basebackup.pl on msys

  25. Suppress variable-set-but-not-used warning from clang 13.

  26. Extend the options of pg_basebackup to control compression

  27. Support base backup targets.

  28. Modify pg_basebackup to use a new COPY subprotocol for base backups.

  29. Document that tar archives are now properly terminated.

  30. Fix thinko in bbsink_throttle_manifest_contents.

  31. Have the server properly terminate tar archives.

  32. Minimal fix for unterminated tar archive problem.

  33. Introduce 'bbstreamer' abstraction to modularize pg_basebackup.

  34. Introduce 'bbsink' abstraction to modularize base backup code.

  35. Refactor basebackup.c's _tarWriteDir() function.

  36. Flexible options for CREATE_REPLICATION_SLOT.

  37. Flexible options for BASE_BACKUP.

Attachments

On Tue, Nov 2, 2021 at 10:32 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Meanwhile, I think it's probably OK for me to go ahead and commit
> 0001-0003 from my patches at this point, since it seems we have pretty
> good evidence that the abstraction basically works, and there doesn't
> seem to be any value in holding off and maybe having to do a bunch
> more rebasing.

I went ahead and committed 0001 and 0002, but got nervous about
proceeding with 0003. For those who may not have been following along
closely, what was 0003 and is now 0001 introduces a new COPY
subprotocol for taking backups. That probably needs to be documented
and as of now the patch does not do that, but the bigger question is
what to do about backward compatibility. I wrote the patch in such a
way that, post-patch, the server can do backups either the way that we
do them now, or the new way that it introduces, but I'm wondering if I
should rip that out and just support the new way only. If you run a
newer pg_basebackup against an older server, it will work, and still
does with the patch. If, however, you run an older pg_basebackup
against a newer server, it complains. For example running a pg13
pg_basebackup against a pg14 cluster produces this:

pg_basebackup: error: incompatible server version 14.0
pg_basebackup: removing data directory "pgstandby"

Now for all I know there is out-of-core software out there that speaks
the replication protocol and can take base backups using it and would
like it to continue working as it does today, and that's easy for me
to do, because that's the way the patch works. But on the other hand
since the patch adapts the in-core tools to use the new method when
talking to a new server, we wouldn't have test coverage for the old
method any more, which might possibly make it annoying to maintain.
But then again that is a problem we could leave for the future, and
rip it out then rather than now. I'm not sure which way to jump.
Anyone else have thoughts?

-- 
Robert Haas
EDB: http://www.enterprisedb.com