Horton

Re: row filtering for logical replication

Euler Taveira <euler@eulerto.com>

View thread
From: "Euler Taveira" <euler@eulerto.com>
To: "Peter Eisentraut" <peter.eisentraut@enterprisedb.com>, "Rahila Syed" <rahilasyed90@gmail.com>
Cc: Önder Kalacı <onderkalaci@gmail.com>, japin <japinli@hotmail.com>, "Michael Paquier" <michael@paquier.xyz>, "David Steele" <david@pgmasters.net>, "Craig Ringer" <craig@2ndquadrant.com>, "Tomas Vondra" <tomas.vondra@2ndquadrant.com>, "Amit Langote" <amitlangote09@gmail.com>, "PostgreSQL Hackers" <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-29T01:23:28Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Release cache tuple when no longer needed

  2. Add some additional tests for row filters in logical replication.

  3. Fix one of the tests introduced in commit 52e4f0cd47.

  4. Allow specifying row filters for logical replication of tables.

  5. Move scanint8() to numutils.c

  6. Replace Test::More plans with done_testing

  7. Reduce relcache access in WAL sender streaming logical changes

  8. Small cleanups related to PUBLICATION framework code

  9. Add a view to show the stats of subscription workers.

  10. Allow publishing the tables of schema.

  11. Doc: improve documentation of CREATE/ALTER SUBSCRIPTION.

  12. Add PublicationTable and PublicationRelInfo structs

  13. Remove unused argument "txn" in maybe_send_schema().

  14. Add prepare API support for streaming transactions in logical replication.

  15. Unify PostgresNode's new() and get_new_node() methods

  16. Use l*_node() family of functions where appropriate

  17. Add support for prepared transactions to built-in logical replication.

  18. Restore the portal-level snapshot after procedure COMMIT/ROLLBACK.

  19. Rename a parse node to be more general

  20. Remove unused column atttypmod from initial tablesync query

  21. SEARCH and CYCLE clauses

Attachments

On Thu, Mar 25, 2021, at 8:15 AM, Peter Eisentraut wrote:
> I have committed the 0001 patch.
> 
> Attached are a few fixup patches that I recommend you integrate into 
> your patch set.  They address backward compatibility with PG13, and a 
> few more stylistic issues.
> 
> I suggest you combine your 0002, 0003, and 0004 patches into one.  They 
> can't be used separately, and for example the psql changes in patch 0003 
> already appear as regression test output changes in 0002, so this 
> arrangement isn't useful.  (0005 can be kept separately, since it's 
> mostly for debugging right now.)
I appreciate your work on it. I split into psql and pg_dump support just
because it was developed after the main patch. I expect them to be combined
into the main patch (0002) before committing it. This new patch set integrates
them into the main patch.

I totally forgot about the backward compatibility support. Good catch.  While
inspecting the code again, I did a small fix into the psql support. I added an
else as shown below so the query always returns the same number of columns and
we don't possibly have an issue while using a column number that is out of
range in PQgetisnull() a few lines later.

            if (pset.sversion >= 140000)
                appendPQExpBuffer(&buf,
                                  ", pg_get_expr(pr.prqual, c.oid)");
            else
                appendPQExpBuffer(&buf,
                                  ", NULL");

While testing the replication between v14 -> v10, I realized that even if the
tables in the publication have row filters, the data synchronization code won't
evaluate the row filter expressions. That's because the subscriber (v10) is
responsible to assemble the COPY command (possibly adding row filters) for data
synchronization and there is no such code in released versions. I added a new
sentence into copy_data parameter saying that row filters won't be used if
version is prior than 14. I also include this info into the commit message.

At this time, I didn't include the patch that changes the log_min_messages in
the row filter regression test. It was part of this patch set for testing
purposes only.

I don't expect the patch that measures row filter performance to be included
but I'm including it again in case someone wants to inspect the performance
numbers.


--
Euler Taveira
EDB   https://www.enterprisedb.com/