[PoC] pg_upgrade: allow to upgrade publisher node
Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: 'Julien Rouhaud' <rjuju123@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>
Date: 2023-04-04T07:00:01Z
Lists: pgsql-hackers
Attachments
- 0001-pg_upgrade-Add-include-replication-slot-option.patch (application/octet-stream) patch 0001
Dear hackers, (CC: Amit and Julien) This is a fork thread of Julien's thread, which allows to upgrade subscribers without losing changes [1]. I briefly implemented a prototype for allowing to upgrade publisher node. IIUC the key lack was that replication slots used for logical replication could not be copied to new node by pg_upgrade command, so this patch allows that. This feature can be used when '--include-replication-slot' is specified. Also, I added a small test for the typical case. It may be helpful to understand. Pg_upgrade internally executes pg_dump for dumping a database object from the old. This feature follows this, adds a new option '--slot-only' to pg_dump command. When specified, it extracts needed info from old node and generate an SQL file that executes pg_create_logical_replication_slot(). The notable deference from pre-existing is that restoring slots are done at the different time. Currently pg_upgrade works with following steps: ... 1. dump schema from old nodes 2. do pg_resetwal several times to new node 3. restore schema to new node 4. do pg_resetwal again to new node ... The probem is that if we create replication slots at step 3, the restart_lsn and confirmed_flush_lsn are set to current_wal_insert_lsn at that time, whereas pg_resetwal discards the WAL file. Such slots cannot extracting changes. To handle the issue the resotring is seprarated into two phases. At the first phase restoring is done at step 3, excepts replicatin slots. At the second phase replication slots are restored at step 5, after doing pg_resetwal. Before upgrading a publisher node, all the changes gerenated on publisher must be sent and applied on subscirber. This is because restart_lsn and confirmed_flush_lsn of copied replication slots is same as current_wal_insert_lsn. New node resets the information which WALs are really applied on subscriber and restart. Basically it is not problematic because before shutting donw the publisher, its walsender processes confirm all data is replicated. See WalSndDone() and related code. Currently physical slots are ignored because this is out-of-scope for me. I did not any analysis about it. [1]: https://www.postgresql.org/message-id/flat/20230217075433.u5mjly4d5cr4hcfe%40jrouhaud Best Regards, Hayato Kuroda FUJITSU LIMITED
Commits
-
Fix issues in binary_upgrade_logical_slot_has_caught_up().
- 0bf62460bb9e 17.0 landed
-
Fix a random failure in 003_logical_slots.pl.
- 63c5df126abb 17.0 landed
-
Fix a test in 003_logical_slots.
- 3e36e48d8efe 17.0 landed
-
Fix uninitialized slot array access during the upgrade.
- a7db71ed2787 17.0 landed
-
Fix the test 003_logical_slots.
- 8af917be6bad 17.0 landed
-
Commit b195e6d482 forgot to update meson.build.
- 8949b978ff49 17.0 landed
-
Use shorter file names in the upgrade logical slots test.
- b195e6d482b8 17.0 landed
-
Migrate logical slots to the new node during an upgrade.
- 29d0a77fa660 17.0 landed
-
Flush logical slots to disk during a shutdown checkpoint if required.
- e0b2eed047df 17.0 landed
-
Prevent possibility of panics during shutdown checkpoint.
- c6c333436491 10.0 cited