Re: backup manifests
Rushabh Lathia <rushabh.lathia@gmail.com>
Commits
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Try to avoid compiler warnings in optimized builds.
- 05021a2c0cd2 13.0 landed
-
Fix option related issues in pg_verifybackup.
- 0a89e93bfaa6 13.0 landed
-
Add index term for backup manifest in documentation.
- 4db819ba4039 13.0 landed
-
Code review for backup manifest.
- a2ac73e7be7a 13.0 landed
-
Document the backup manifest file format.
- 149f2ae88ab0 13.0 landed
-
Fix typo in pg_validatebackup documentation.
- c4f82a779d26 13.0 landed
-
Exclude backup_manifest file that existed in database, from BASE_BACKUP.
- 1ec50a81ec0a 13.0 landed
-
Msys2 tweaks for pg_validatebackup corruption test
- c3e4cbaab936 13.0 landed
-
Fix resource management bug with replication=database.
- 3e0d80fd8d3d 13.0 cited
-
Be more careful about time_t vs. pg_time_t in basebackup.c.
- db1531cae009 13.0 cited
-
pg_validatebackup: Fix 'make clean' to remove tmp_check.
- 9f8f881caa0f 13.0 landed
-
pg_validatebackup: Also use perl2host in TAP tests.
- 460314db08e8 13.0 landed
-
Generate backup manifests for base backups, and validate them.
- 0d8c9c1210c4 13.0 landed
-
Add checksum helper functions.
- c12e43a2e0d4 13.0 landed
-
pg_waldump: Add a --quiet option.
- ac44367efbef 13.0 landed
-
Catversion bump for b9b408c48724
- afb5465e0cfc 13.0 cited
-
pg_basebackup: Refactor code for reading COPY and tar data.
- 431ba7bebf13 13.0 landed
-
Use a ResourceOwner to track buffer pins in all cases.
- 3cb646264e8c 12.0 cited
-
Use ARMv8 CRC instructions where available.
- f044d71e331d 11.0 cited
-
Logical replication support for initial data copy
- 7c4f52409a8c 10.0 cited
-
Use Intel SSE 4.2 CRC instructions where available.
- 3dc2d62d0486 9.5.0 cited
-
Switch to CRC-32C in WAL and other places.
- 5028f22f6eb0 9.5.0 cited
-
Remove support for 64-bit CRC.
- 404bc51cde9d 9.5.0 cited
-
Change CRCs in WAL records from 64bit to 32bit for performance reasons.
- 21fda22ec46d 8.1.0 cited
Attachments
- 0001-Reduce-code-duplication-and-eliminate-weird-macro-tr.patch (text/x-patch) patch 0001
- 0004-Documentation-for-backup-manifest-and-manifest-check.patch (text/x-patch) patch 0004
- 0003-Make-checksum-optional-in-pg_basebackup-review-comme.patch (text/x-patch) patch 0003
- 0002-POC-of-backup-manifest-with-file-names-sizes-timesta.patch (text/x-patch) patch 0002
- 0005-Allow-user-to-choose-a-checksum-algorithm-for-manife.patch (text/x-patch) patch 0005
On Thu, Dec 5, 2019 at 12:17 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Dec 4, 2019 at 1:01 PM Rushabh Lathia <rushabh.lathia@gmail.com>
> wrote:
> > As per the discussion on the thread, here is the patch which
> >
> > a) Make checksum for manifest file optional.
> > b) Allow user to choose a particular algorithm.
> >
> > Currently with the WIP patch SHA256 and CRC checksum algorithm
> > supported. Patch also changed the manifest file format to append
> > the used algorithm name before the checksum, this way it will be
> > easy to validator to know which algorithm to used.
> >
> > Ex:
> > ./db/bin/pg_basebackup -D bksha/ --manifest-with-checksums=SHA256
> >
> > $ cat bksha/backup_manifest | more
> > PostgreSQL-Backup-Manifest-Version 1
> > File backup_label 226 2019-12-04 17:46:46 GMT
> SHA256:7cf53d1b9facca908678ab70d93a9e7460cd35cedf7891de948dcf858f8a281a
> > File pg_xact/0000 8192 2019-12-04 17:46:46 GMT
> SHA256:8d2b6cb1dc1a6e8cee763b52d75e73571fddce06eb573861d44082c7d8c03c26
> >
> > ./db/bin/pg_basebackup -D bkcrc/ --manifest-with-checksums=CRC
> > PostgreSQL-Backup-Manifest-Version 1
> > File backup_label 226 2019-12-04 17:58:40 GMT CRC:343138313931333134
> > File pg_xact/0000 8192 2019-12-04 17:46:46 GMT CRC:363538343433333133
> >
> > Pending TODOs:
> > - Documentation update
> > - Code cleanup
> > - Testing.
> >
> > I will further continue to work on the patch and meanwhile feel free to
> provide
> > thoughts/inputs.
>
> + initilize_manifest_checksum(&cCtx);
>
> Spelling.
>
>
Fixed.
-
>
> Spurious.
>
> + case MC_CRC:
> + INIT_CRC32C(cCtx->crc_ctx);
>
> Suggest that we do CRC -> CRC32C throughout the patch. Someone might
> conceivably want some other CRC variant, mostly likely 64-bit, in the
> future.
>
>
Make sense, done.
+final_manifest_checksum(ChecksumCtx *cCtx, char *checksumbuf)
>
> finalize
>
>
Done.
printf(_(" --manifest-with-checksums\n"
> - " do calculate checksums for manifest files\n"));
> + " calculate checksums for manifest files
> using provided algorithm\n"));
>
> Switch name is wrong. Suggest --manifest-checksums.
> Help usually shows that an argument is expected, e.g.
> --manifest-checksums=ALGORITHM or
> --manifest-checksums=sha256|crc32c|none
>
>
Fixed.
This seems to apply over some earlier version of the patch. A
> consolidated patch, or the whole stack, would be better.
>
Here is the whole stack of patches.
Thanks,
Rushabh Lathia
www.EnterpriseDB.com