Horton

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

P <apraveen@pivotal.io>

View thread View on postgr.es
From: Asim R P <apraveen@pivotal.io>
To: Paul Guo <pguo@pivotal.io>, Anastasia Lubennikova <a.lubennikova@postgrespro.ru>, Alexandra Wang <leiwang@pivotal.io>
Cc: Thomas Munro <thomas.munro@gmail.com>, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>, PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Date: 2019-09-19T11:59:59Z
Lists: pgsql-hackers

Attachments

On Thu, Aug 22, 2019 at 6:44 PM Paul Guo <pguo@pivotal.io> wrote:
>
> Thanks. I updated the patch to v5. It passes install-check testing and
recovery testing.
>

This patch contains one more bug, in addition to what Anastasia has found.
If the test case in the patch is tweaked slightly, as follows, the standby
crashes due to PANIC.

--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -147,8 +147,6 @@ $node_standby->start;
 $node_master->poll_query_until(
        'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');

-$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
-
 # Make sure to perform restartpoint after tablespace creation
 $node_master->wait_for_catchup($node_standby, 'replay',

 $node_master->lsn('replay'));
@@ -156,7 +154,8 @@ $node_standby->safe_psql('postgres', 'CHECKPOINT');

 # Do immediate shutdown ...
 $node_master->safe_psql('postgres',
-                                               q[ALTER DATABASE db1 SET
TABLESPACE ts2;
+                                               q[CREATE DATABASE db1
TABLESPACE ts1;
+                                                 ALTER DATABASE db1 SET
TABLESPACE ts2;
                                                  DROP TABLESPACE ts1;]);
 $node_master->wait_for_catchup($node_standby, 'replay',

 $node_master->lsn('replay'));

Notice the create additional create database in the above change.  That
causes the same tablespace directory (ts1) logged twice in the list of
missing directories.  At the end of crash recovery, there is one unmatched
entry in the missing dirs list and the standby PANICs.

Please find attached a couple of tests that are built on top of what was
already written by Paul, Kyotaro.  The patch includes a test to demonstrate
the above mentioned failure and a test case that my friend Alexandra wrote
to implement the archive recovery scenario noted by Anastasia.

In order to fix the test failures, we need to distinguish between a missing
database directory and a missing tablespace directory.  And also add logic
to forget missing directories during tablespace drop.  I am working on it.

Asim

Commits

  1. Improve recently-added test reliability

  2. Fix new recovery test for log_error_verbosity=verbose case

  3. Fix test instability

  4. Fix replay of create database records on standby

  5. Allow "in place" tablespaces.

  6. Fix get_dirent_type() for Windows junction points.

  7. Revert "Fix replay of create database records on standby"

  8. Add end-to-end testing of pg_basebackup's tar-format output.

  9. Make DROP DATABASE command generate less WAL records.

  10. Consolidate methods for translating a Perl path to a Windows path.