Horton

Re: row filtering for logical replication

Ajin Cherian <itsajin@gmail.com>

View thread
From: Ajin Cherian <itsajin@gmail.com>
To: Peter Smith <smithpb2250@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Euler Taveira <euler@eulerto.com>, Greg Nancarrow <gregn4422@gmail.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, vignesh C <vignesh21@gmail.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, Dilip Kumar <dilipbalaut@gmail.com>, Rahila Syed <rahilasyed90@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Önder Kalacı <onderkalaci@gmail.com>, japin <japinli@hotmail.com>, Michael Paquier <michael@paquier.xyz>, David Steele <david@pgmasters.net>, Craig Ringer <craig@2ndquadrant.com>, Amit Langote <amitlangote09@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-12-08T11:52:28Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Release cache tuple when no longer needed

  2. Add some additional tests for row filters in logical replication.

  3. Fix one of the tests introduced in commit 52e4f0cd47.

  4. Allow specifying row filters for logical replication of tables.

  5. Move scanint8() to numutils.c

  6. Replace Test::More plans with done_testing

  7. Reduce relcache access in WAL sender streaming logical changes

  8. Small cleanups related to PUBLICATION framework code

  9. Add a view to show the stats of subscription workers.

  10. Allow publishing the tables of schema.

  11. Doc: improve documentation of CREATE/ALTER SUBSCRIPTION.

  12. Add PublicationTable and PublicationRelInfo structs

  13. Remove unused argument "txn" in maybe_send_schema().

  14. Add prepare API support for streaming transactions in logical replication.

  15. Unify PostgresNode's new() and get_new_node() methods

  16. Use l*_node() family of functions where appropriate

  17. Add support for prepared transactions to built-in logical replication.

  18. Restore the portal-level snapshot after procedure COMMIT/ROLLBACK.

  19. Rename a parse node to be more general

  20. Remove unused column atttypmod from initial tablesync query

  21. SEARCH and CYCLE clauses

Attachments

On Tue, Dec 7, 2021 at 5:36 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> We were mid-way putting together the next v45* when your latest
> attachment was posted over the weekend. So we will proceed with our
> original plan to post our v45* (tomorrow).
>
> After v45* is posted we will pause to find what are all the
> differences between your unified patch and our v45* patch set. Our
> intention is to integrate as many improvements as possible from your
> changes into the v46* etc that will follow tomorrow’s v45*. On some
> points, we will most likely need further discussion.


Posting an update for review comments, using contributions majorly
from Peter Smith.
I've also included changes based on Euler's combined patch, specially
changes to documentation and test cases.
I have left out Hou-san's 0005, in this patch-set. Hou-san will
provide a rebased update based on this.

This patch addresses the following review comments:


On Wed, Nov 24, 2021 at 8:52 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Few comments:
> 1) I'm not sure if we will be able to throw a better error message in
> this case "ERROR:  missing FROM-clause entry for table "t4"", if
> possible you could change it.

Fixed this.

On Thu, Dec 2, 2021 at 7:40 PM vignesh C <vignesh21@gmail.com> wrote:
>
> 1) Both testpub5a and testpub5c publication are same, one of them can be removed
> +SET client_min_messages = 'ERROR';
> +CREATE PUBLICATION testpub5a FOR TABLE testpub_rf_tbl1 WHERE (a > 1)
> WITH (publish="insert");
> +CREATE PUBLICATION testpub5b FOR TABLE testpub_rf_tbl1;
> +CREATE PUBLICATION testpub5c FOR TABLE testpub_rf_tbl1 WHERE (a > 3)
> WITH (publish="insert");
> +RESET client_min_messages;
> +\d+ testpub_rf_tbl1
> +DROP PUBLICATION testpub5a, testpub5b, testpub5c;

Fixed

On Fri, Dec 3, 2021 at 6:16 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
>
> doc/src/sgml/ref/create_subscription.sgml
> (2) Refer to Notes
>
> Perhaps a link to the Notes section should be used here, as follows:
>
> -          copied. Refer to the Notes section below.
> +          copied. Refer to the <xref
> linkend="sql-createsubscription-notes"/> section below.

Fixed

> 1) Typo in patch comment
> "Specifially"

Fixed

> src/backend/catalog/pg_publication.c
> 2) bms_replident comment
> Member "Bitmapset  *bms_replident;" in rf_context should have a
> comment, maybe something like "set of replica identity col indexes".
>

Fixed

> 3) errdetail message
> In rowfilter_walker(), the "forbidden" errdetail message is loaded
> using gettext() in one instance, but just a raw formatted string in
> other cases. Shouldn't they all consistently be translated strings?
>

Fixed

> (i)
> if (slot == NULL || TTS_EMPTY(slot))
> can be replaced with:
> if (TupIsNull(slot))
>

Fixed

> (ii) In the above case (where values and nulls are palloc'd),
> shouldn't the values and nulls be pfree()d at the end of the function?
>
>

Fixed, changed it into fixed arrays

On Thu, Dec 2, 2021 at 2:32 PM tanghy.fnst@fujitsu.com
<tanghy.fnst@fujitsu.com> wrote:
>>
> 1. This is an example in publication doc, but in fact it's not allowed. Should we
> change this example?
>

Fixed.

> 2. A typo in 0002 patch.
>
> + * drops such a user-defnition or if there is any other error via its function,
>
> "user-defnition" should be "user-definition".

Fixed

On Fri, Dec 3, 2021 at 12:59 AM Euler Taveira <euler@eulerto.com> wrote:
>
> ExprState cache logic is basically all the same as before (including
> all the OR combining), but there are now 4x ExprState caches keyed and
> separated by the 4x different pubactions.
>
> row filter is not applied for TRUNCATEs so it is just 3 operations.
>

Fixed

regards,
Ajin Cherian
Fujitsu Australia