Logical Replication of sequences
Amit Kapila <amit.kapila16@gmail.com>
Commits
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Doc: Add documentation for sequence synchronization.
- 55cefadde874 19 (unreleased) landed
-
Remove unused assignment in CREATE PUBLICATION grammar.
- bfb7419b0bbe 19 (unreleased) landed
-
Add seq_sync_error_count to subscription statistics.
- f6a4c498dcf6 19 (unreleased) landed
-
Fix few issues in commit 5509055d69.
- 5a4eba558aa7 19 (unreleased) landed
-
Add sequence synchronization for logical replication.
- 5509055d6956 19 (unreleased) landed
-
Add worker type argument to logical replication worker functions.
- 3e8e05596a02 19 (unreleased) landed
-
Introduce "REFRESH SEQUENCES" for subscriptions.
- f0b3573c3aac 19 (unreleased) landed
-
Refactor logical worker synchronization code into a separate file.
- 41c674d2e31e 19 (unreleased) landed
-
Standardize use of REFRESH PUBLICATION in code and messages.
- 2436b8c047ff 19 (unreleased) landed
-
Add "ALL SEQUENCES" support to publications.
- 96b378497346 19 (unreleased) landed
-
Expose sequence page LSN via pg_get_sequence_data.
- b93172ca59f4 19 (unreleased) landed
-
Resume conflict-relevant data retention automatically.
- 0d48d393d465 19 (unreleased) cited
-
Post-commit review fixes for 228c370868.
- 1f7e9ba3ac4e 19 (unreleased) cited
-
Generate GUC tables from .dat file
- 63599896545c 19 (unreleased) cited
In the past, we have discussed various approaches to replicate sequences by decoding the sequence changes from WAL. However, we faced several challenges to achieve the same, some of which are due to the non-transactional nature of sequences. The major ones were: (a) correctness of the decoding part, some of the problems were discussed at [1][2][3] (b) handling of sequences especially adding certain sequences automatically (e.g. sequences backing SERIAL/BIGSERIAL columns) for built-in logical replication is not considered in the proposed work [1] (c) there were some performance concerns in not so frequent scenarios [4] (see performance issues), we can probably deal with this by making sequences optional for builtin logical replication It could be possible that we can deal with these and any other issues with more work but as the use case for this feature is primarily major version upgrades it is not clear that we want to make such a big change to the code or are there better alternatives to achieve the same. This time at pgconf.dev (https://2024.pgconf.dev/), we discussed alternative approaches for this work which I would like to summarize. The various methods we discussed are as follows: 1. Provide a tool to copy all the sequences from publisher to subscriber. The major drawback is that users need to perform this as an additional step during the upgrade which would be inconvenient and probably not as useful as some built-in mechanism. 2. Provide a command say Alter Subscription ... Replicate Sequences (or something like that) which users can perform before shutdown of the publisher node during upgrade. This will allow copying all the sequences from the publisher node to the subscriber node directly. Similar to previous approach, this could also be inconvenient for users. 3. Replicate published sequences via walsender at the time of shutdown or incrementally while decoding checkpoint record. The two ways to achieve this are: (a) WAL log a special NOOP record just before shutting down checkpointer. Then allow the WALsender to read the sequence data and send it to the subscriber while decoding the new NOOP record. (b) Similar to the previous idea but instead of WAL logging a new record directly invokes a decoding callback after walsender receives a request to shutdown which will allow pgoutput to read and send required sequences. This approach has a drawback that we are adding more work at the time of shutdown but note that we already waits for all the WAL records to be decoded and sent before shutting down the walsender during shutdown of the node. Any other ideas? I have added the members I remember that were part of the discussion in the email. Please feel free to correct me if I have misunderstood or missed any point we talked about. Thoughts? [1] - https://www.postgresql.org/message-id/e4145f77-6f37-40e0-a770-aba359c50b93%40enterprisedb.com [2] - https://www.postgresql.org/message-id/CAA4eK1Lxt%2B5a9fA-B7FRzfd1vns%3DEwZTF5z9_xO9Ms4wsqD88Q%40mail.gmail.com [3] - https://www.postgresql.org/message-id/CAA4eK1KR4%3DyALKP0pOdVkqUwoUqD_v7oU3HzY-w0R_EBvgHL2w%40mail.gmail.com [4] - https://www.postgresql.org/message-id/12822961-b7de-9d59-dd27-2e3dc3980c7e%40enterprisedb.com -- With Regards, Amit Kapila.