Horton

Re: backup manifests

David Steele <david@pgmasters.net>

View thread
From: David Steele <david@pgmasters.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Rushabh Lathia <rushabh.lathia@gmail.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Jeevan Chalke <jeevan.chalke@enterprisedb.com>, vignesh C <vignesh21@gmail.com>
Date: 2019-11-22T19:02:12Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Try to avoid compiler warnings in optimized builds.

  2. Fix option related issues in pg_verifybackup.

  3. Add index term for backup manifest in documentation.

  4. Code review for backup manifest.

  5. Document the backup manifest file format.

  6. Fix typo in pg_validatebackup documentation.

  7. Exclude backup_manifest file that existed in database, from BASE_BACKUP.

  8. Msys2 tweaks for pg_validatebackup corruption test

  9. Fix resource management bug with replication=database.

  10. Be more careful about time_t vs. pg_time_t in basebackup.c.

  11. pg_validatebackup: Fix 'make clean' to remove tmp_check.

  12. pg_validatebackup: Also use perl2host in TAP tests.

  13. Generate backup manifests for base backups, and validate them.

  14. Add checksum helper functions.

  15. pg_waldump: Add a --quiet option.

  16. Catversion bump for b9b408c48724

  17. pg_basebackup: Refactor code for reading COPY and tar data.

  18. Use a ResourceOwner to track buffer pins in all cases.

  19. Use ARMv8 CRC instructions where available.

  20. Logical replication support for initial data copy

  21. Use Intel SSE 4.2 CRC instructions where available.

  22. Switch to CRC-32C in WAL and other places.

  23. Remove support for 64-bit CRC.

  24. Change CRCs in WAL records from 64bit to 32bit for performance reasons.

On 11/22/19 1:24 PM, Robert Haas wrote:
> On Tue, Nov 19, 2019 at 4:34 PM David Steele <david@pgmasters.net> wrote:
>> On 11/19/19 5:00 AM, Rushabh Lathia wrote:
>>> My colleague Suraj did testing and noticed the performance impact
>>> with the checksums.   On further testing, he found that specifically with
>>> sha its more of performance impact.
>>
>> We have found that SHA1 adds about 3% overhead when the backup is also
>> compressed (gzip -6), which is what most people want to do.  This
>> percentage goes down even more if the backup is being transferred over a
>> network or to an object store such as S3.
> 
> I don't really understand why your tests and Suraj's tests are showing
> such different results, or how compression plays into it. I tried
> running shasum -a$N lineitem-big.csv on my laptop, where that file
> contains ~70MB of random-looking data whose source I no longer
> remember. Here are the results by algorithm: SHA1, ~25 seconds; SHA224
> or SHA256, ~52 seconds; SHA384 and SHA512, ~39 seconds. Aside from the
> interesting discovery that the algorithms with more bits actually run
> faster on this machine, this seems to show that there's only about a
> ~2x difference between the SHA1 that you used and that I (pretty much
> arbitrarily) used. But Rushabh and Suraj are reporting 43-54%
> overhead, and even if you divide that by two it's a lot more than 3%.
> 
> One possible explanation is that the compression is really slow, and
> so it makes the checksum overhead a smaller percentage of the total.
> Like, if you've already slowed down the backup by 8x, then 24%
> overhead turns into 3% overhead! But I assume that's not the real
> explanation here. 

That's the real explanation here.  Hash calculations run at the same
speed, they just become a smaller portion of the *total* time once
compression (gzip -6) is added.  With something like lz4 hashing will
obviously be a big percentage of the total.

Also consider how much extra latency you get from copying over a
network.  My 3% did not include that but realistically most backups are
running over a network (hopefully).

>> That said, making SHA256 optional seems reasonable.  We decided not to
>> make our SHA1 checksums optional to reduce the test matrix and because
>> parallelism largely addressed performance concerns.
> 
> Just to be clear, I really don't have any objection to using SHA1
> instead of SHA256, or anything else for that matter. I picked the one
> to use out of a hat for the purpose of having a POC quickly; I didn't
> have any intention to insist on that as the final selection. It seems
> likely that anything we pick here will eventually be considered
> obsolete, so I think we need to allow for configurability, but I don't
> have a horse in the game as far as an initial selection goes.

We decided that SHA1 was good enough and there was no need to go up to
SHA256.  What we were interested in was collision rates and what the
chance of getting a false positive were based on the combination of
path, size, and hash.  With SHA1 the chance of a collision was literally
astronomically low (as in the universe would probably end before it
happened, depending on whether you are an expand forever or contract
proponent).

> Except - and this gets back to the previous point - I don't want to
> slow down backups by 40% by default. I wouldn't mind slowing them down
> 3% by default, but 40% is too much overhead. I think we've gotta
> either the overhead of using SHA way down or not use SHA by default.

Maybe -- my take is that the measurements, an uncompressed backup to the
local filesystem, are not a very realistic use case.

However, I'm still fine with leaving the user the option of checksums or
no.  I just wanted to point out that CRCs have their limits so maybe
that's not a great option unless it is properly caveated and perhaps not
the default.

Regards,
-- 
-David
david@pgmasters.net