RE: [PoC] pg_upgrade: allow to upgrade publisher node
Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: 'Amit Kapila' <amit.kapila16@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Bruce Momjian <bruce@momjian.us>, Julien Rouhaud <rjuju123@gmail.com>, vignesh C <vignesh21@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, "Zhijie Hou
(Fujitsu)" <houzj.fnst@fujitsu.com>
Date: 2023-08-18T03:12:19Z
Lists: pgsql-hackers
Dear Amit,
> > I have not used other plugins, but forcibly renamed the shared object file.
> > I would test by plugins like wal2json[1] if more cases are needed.
> >
> > 1. created logical replication slots on old node
> > SELECT * FROM pg_create_logical_replication_slot('test', 'test_decoding')
> > 2. stopped the old nde
> > 3. forcibly renamed the so file. I used following script:
> > sudo mv /path/to/test_decoding.so /path/to//test\"_decoding.so
> > 4. executed pg_upgrade and failed. Outputs what I got were:
> >
> > ```
> > Checking for presence of required libraries fatal
> >
>
> Your test sounds reasonable but there is no harm in testing wal2json
> or some other plugin just to mimic the actual production scenario.
> Additionally, it would give us better coverage for the patch by
> testing out-of-core plugins for some other tests as well.
I've tested by using wal2json, decoder_raw[1], and my small decoder. The results were
the same: pg_upgrade correctly raised an ERROR. Following demo shows the case for wal2json.
In this test, the plugin was installed only on the old node and a slot was created.
Below shows the created slot:
```
(Old)=# SELECT slot_name, plugin FROM pg_replication_slots
slot_name | plugin
-----------+----------
test | wal2json
(1 row)
```
And I confirmed that the plugin worked well via pg_logical_slot_get_changes()
(This was needed to move forward the confirmed_flush_lsn)
```
(Old)=# INSERT INTO foo VALUES (1)
INSERT 0 1
(Old)=# SELECT * FROM pg_logical_slot_get_changes('test', NULL, NULL);
lsn | xid | data
----------+-----+-------------------------------------------------------------------------------------------------------------
---------------------
0/63C8A8 | 731 | {"change":[{"kind":"insert","schema":"public","table":"foo","columnnames":["id"],"columntypes":["integer"],"
columnvalues":[1]}]}
(1 row)
```
Then the pg_upgrade was executed but failed, same as the previous example.
```
Checking for presence of required libraries fatal
Your installation references loadable libraries that are missing from the
new installation. You can add these libraries to the new installation,
or remove the functions using them from the old installation. A list of
problem libraries is in the file:
data_N3/pg_upgrade_output.d/20230818T030006.675/loadable_libraries.txt
Failure, exiting
```
In the loadable_libraries.txt, it mentioned that wal2json was not installed to new directory.
```
could not load library "wal2json": ERROR: could not access file "wal2json": No such file or directory
In database: postgres
```
Note that upgrade was done if the plugin was installed to new binary too.
Acknowledgement: Thank you Michael and Euler for creating great plugins!
[1]: https://github.com/michaelpq/pg_plugins
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Commits
-
Fix issues in binary_upgrade_logical_slot_has_caught_up().
- 0bf62460bb9e 17.0 landed
-
Fix a random failure in 003_logical_slots.pl.
- 63c5df126abb 17.0 landed
-
Fix a test in 003_logical_slots.
- 3e36e48d8efe 17.0 landed
-
Fix uninitialized slot array access during the upgrade.
- a7db71ed2787 17.0 landed
-
Fix the test 003_logical_slots.
- 8af917be6bad 17.0 landed
-
Commit b195e6d482 forgot to update meson.build.
- 8949b978ff49 17.0 landed
-
Use shorter file names in the upgrade logical slots test.
- b195e6d482b8 17.0 landed
-
Migrate logical slots to the new node during an upgrade.
- 29d0a77fa660 17.0 landed
-
Flush logical slots to disk during a shutdown checkpoint if required.
- e0b2eed047df 17.0 landed
-
Prevent possibility of panics during shutdown checkpoint.
- c6c333436491 10.0 cited