Horton

Re: row filtering for logical replication

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Rahila Syed <rahilasyed90@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>, Greg Nancarrow <gregn4422@gmail.com>, Euler Taveira <euler@eulerto.com>, Peter Smith <smithpb2250@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Önder Kalacı <onderkalaci@gmail.com>, japin <japinli@hotmail.com>, Michael Paquier <michael@paquier.xyz>, David Steele <david@pgmasters.net>, Craig Ringer <craig@2ndquadrant.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Amit Langote <amitlangote09@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-07-23T10:16:23Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Release cache tuple when no longer needed

  2. Add some additional tests for row filters in logical replication.

  3. Fix one of the tests introduced in commit 52e4f0cd47.

  4. Allow specifying row filters for logical replication of tables.

  5. Move scanint8() to numutils.c

  6. Replace Test::More plans with done_testing

  7. Reduce relcache access in WAL sender streaming logical changes

  8. Small cleanups related to PUBLICATION framework code

  9. Add a view to show the stats of subscription workers.

  10. Allow publishing the tables of schema.

  11. Doc: improve documentation of CREATE/ALTER SUBSCRIPTION.

  12. Add PublicationTable and PublicationRelInfo structs

  13. Remove unused argument "txn" in maybe_send_schema().

  14. Add prepare API support for streaming transactions in logical replication.

  15. Unify PostgresNode's new() and get_new_node() methods

  16. Use l*_node() family of functions where appropriate

  17. Add support for prepared transactions to built-in logical replication.

  18. Restore the portal-level snapshot after procedure COMMIT/ROLLBACK.

  19. Rename a parse node to be more general

  20. Remove unused column atttypmod from initial tablesync query

  21. SEARCH and CYCLE clauses

On Fri, Jul 23, 2021 at 2:27 PM Rahila Syed <rahilasyed90@gmail.com> wrote:
>
> On Fri, Jul 23, 2021 at 8:36 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Fri, Jul 23, 2021 at 8:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >
>> > On Thu, Jul 22, 2021 at 8:06 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> > >
>> > > On Thu, Jul 22, 2021 at 5:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>> > > >
>> > > > On Tue, Jul 20, 2021 at 4:33 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> > > > >
>> > > > > On Tue, Jul 20, 2021 at 3:43 PM Tomas Vondra
>> > > > > <tomas.vondra@enterprisedb.com> wrote:
>> > > > > >
>> > > > > > Do we log the TOAST-ed values that were not updated?
>> > > > >
>> > > > > No, we don't, I have submitted a patch sometime back to fix that [1]
>> > > > >
>> > > >
>> > > > That patch seems to log WAL for key unchanged columns. What about if
>> > > > unchanged non-key columns? Do they get logged as part of the new tuple
>> > > > or is there some other way we can get those? If not, then we need to
>> > > > probably think of restricting filter clause in some way.
>> > >
>> > > But what sort of restrictions? I mean we can not put based on data
>> > > type right that will be too restrictive,
>> > >
>> >
>> > Yeah, data type restriction sounds too restrictive and unless the data
>> > is toasted, the data will be anyway available. I think such kind of
>> > restriction should be the last resort but let's try to see if we can
>> > do something better.
>> >
>> > > other option is only to allow
>> > > replica identity keys columns in the filter condition?
>> > >
>> >
>> > Yes, that is what I had in mind because if key column(s) is changed
>> > then we will have data for both old and new tuples. But if it is not
>> > changed then we will have it probably for the old tuple unless we
>> > decide to fix the bug you mentioned in a different way in which case
>> > we might either need to log it for the purpose of this feature (but
>> > that will be any way for HEAD) or need to come up with some other
>> > solution here. I think we can't even fetch such columns data during
>> > decoding because we have catalog-only historic snapshots here. Do you
>> > have any better ideas?
>> >
>>
>> BTW, I wonder how pglogical can handle this because if these unchanged
>> toasted values are not logged in WAL for the new tuple then how the
>> comparison for such columns will work? Either they are forcing WAL in
>> some way or don't allow WHERE clause on such columns or maybe they
>> have dealt with it in some other way unless they are unaware of this
>> problem.
>>
>
> The column comparison for row filtering happens before the unchanged toast
> columns are filtered. Unchanged toast columns are filtered just before writing the tuple
> to output stream.
>

To perform filtering, you need to use the tuple from WAL and that
tuple doesn't seem to have unchanged toast values, so how can we do
filtering? I think it is a good idea to test this once.

-- 
With Regards,
Amit Kapila.