Horton

Re: backup manifests

Robert Haas <robertmhaas@gmail.com>

View thread
From: Robert Haas <robertmhaas@gmail.com>
To: David Steele <david@pgmasters.net>
Cc: Andres Freund <andres@anarazel.de>, Noah Misch <noah@leadboat.com>, Stephen Frost <sfrost@snowman.net>, Amit Kapila <amit.kapila16@gmail.com>, Suraj Kharage <suraj.kharage@enterprisedb.com>, tushar <tushar.ahuja@enterprisedb.com>, Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>, Rushabh Lathia <rushabh.lathia@gmail.com>, Tels <nospam-pg-abuse@bloodgate.com>, Andrew Dunstan <andrew.dunstan@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Jeevan Chalke <jeevan.chalke@enterprisedb.com>, vignesh C <vignesh21@gmail.com>
Date: 2020-04-03T20:49:19Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Try to avoid compiler warnings in optimized builds.

  2. Fix option related issues in pg_verifybackup.

  3. Add index term for backup manifest in documentation.

  4. Code review for backup manifest.

  5. Document the backup manifest file format.

  6. Fix typo in pg_validatebackup documentation.

  7. Exclude backup_manifest file that existed in database, from BASE_BACKUP.

  8. Msys2 tweaks for pg_validatebackup corruption test

  9. Fix resource management bug with replication=database.

  10. Be more careful about time_t vs. pg_time_t in basebackup.c.

  11. pg_validatebackup: Fix 'make clean' to remove tmp_check.

  12. pg_validatebackup: Also use perl2host in TAP tests.

  13. Generate backup manifests for base backups, and validate them.

  14. Add checksum helper functions.

  15. pg_waldump: Add a --quiet option.

  16. Catversion bump for b9b408c48724

  17. pg_basebackup: Refactor code for reading COPY and tar data.

  18. Use a ResourceOwner to track buffer pins in all cases.

  19. Use ARMv8 CRC instructions where available.

  20. Logical replication support for initial data copy

  21. Use Intel SSE 4.2 CRC instructions where available.

  22. Switch to CRC-32C in WAL and other places.

  23. Remove support for 64-bit CRC.

  24. Change CRCs in WAL records from 64bit to 32bit for performance reasons.

On Fri, Apr 3, 2020 at 3:53 PM Robert Haas <robertmhaas@gmail.com> wrote:
> 2. Also, a bunch of machines were super-unhappy with
> 003_corruption.pl, failing with this sort of thing:
>
> pg_basebackup: error: could not get COPY data stream: ERROR:  symbolic
> link target too long for tar format: file name "pg_tblspc/16387",
> target "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/src/bin/pg_validatebackup/tmp_check/tmp_test_7w0w"
>
> Apparently, this is a known problem and the solution is to use
> TestLib::tempdir_short instead of TestLib::tempdir, so I pushed a fix
> to make it do that.

By and large, the buildfarm is a lot happier now, but fairywren
(Windows / Msys Server 2019 / 2 gcc 7.3.0 x86_64) failed like this:

# Postmaster PID for node "master" is 198420
error running SQL: 'psql:<stdin>:3: ERROR:  directory
"/tmp/9peoZHrEia" does not exist'
while running 'psql -XAtq -d port=51493 host=127.0.0.1
dbname='postgres' -f - -v ON_ERROR_STOP=1' with sql 'CREATE TABLE x1
(a int);
INSERT INTO x1 VALUES (111);
CREATE TABLESPACE ts1 LOCATION '/tmp/9peoZHrEia';
CREATE TABLE x2 (a int) TABLESPACE ts1;
INSERT INTO x1 VALUES (222);
' at /home/pgrunner/bf/root/HEAD/pgsql.build/../pgsql/src/test/perl/PostgresNode.pm
line 1531.
### Stopping node "master" using mode immediate

I wondered why this should be failing on this machine when none of the
other places where tempdir_short is used are similarly failing. The
answer appears to be that most of the TAP tests that use tempdir_short
just do this:

my $tempdir_short = TestLib::tempdir_short;

...and then ignore that variable completely for the rest of the
script.  That's not ideal, and we should probably remove those calls
to avoid giving that it's actually used for something. The two TAP
tests that actually do something with it - apart from the one I just
added - are pg_basebackup's 010_pg_basebackup.pl and pg_ctl's
001_start_stop.pl. However, both of those are skipped on Windows.
Also, PostgresNode.pm itself uses it, but only when UNIX sockets are
used, so again not on Windows. So it sorta looks to me like we no
preexisting tests that meaningfully exercise TestLib::tempdir_short on
Windows.

Given that, I suppose I should consider myself lucky if this ends up
working on *any* of the Windows critters, but given the implementation
I'm kinda surprised we have a problem. That function is just:

sub tempdir_short
{

        return File::Temp::tempdir(CLEANUP => 1);
}

And File::Temp's documentation says that the temporary directory is
picked using File::Spec's tmpdir(), which says that it knows about
different operating systems and will DTRT on Unix, Mac, OS2, Win32,
and VMS. Yet on fairywren it is apparently DTWT. I'm not sure why.

Any ideas?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company