Horton

Re: Minimal logical decoding on standbys

bdrouvotAWS <bdrouvot@amazon.com>

View thread
From: "Drouvot, Bertrand" <bdrouvot@amazon.com>
To: <fabriziomello@gmail.com>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>, Andres Freund <andres@anarazel.de>, Ibrar Ahmed <ibrar.ahmad@gmail.com>, Amit Khandekar <amitdkhan.pg@gmail.com>, tushar <tushar.ahuja@enterprisedb.com>, "[pgdg] Robert Haas" <robertmhaas@gmail.com>, Rahila Syed <rahila.syed@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-09-20T10:17:11Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Reduce the log level in 035_standby_logical_decoding.pl.

  2. 035_standby_logical_decoding: Add missing waits for replication

  3. For cascading replication, wake physical and logical walsenders separately

  4. Handle logical slot conflicts on standby

  5. Support invalidating replication slots due to horizon and wal_level

  6. Prevent use of invalidated logical slot in CreateDecodingContext()

  7. Replace replication slot's invalidated_at LSN with an enum

  8. Pass down table relation into more index relation functions

  9. Assert only valid flag bits are passed to visibilitymap_set()

  10. Remove unused _bt_delitems_delete() argument.

  11. Add xl_btree_delete optimization.

Hi,

On 9/17/21 10:32 PM, Fabrízio de Royes Mello wrote:
>
> On Wed, Sep 15, 2021 at 8:36 AM Drouvot, Bertrand <bdrouvot@amazon.com 
> <mailto:bdrouvot@amazon.com>> wrote:
> >
> > Another rebase attached.
> >
> > The patch proposal to address Andre's walsender corner cases is 
> still a dedicated commit (as i think it may be easier to discuss).
> >
>
> Did one more battery of tests and everything went well...

Thanks for looking at it!

>
> But doing some manually tests:
>
> 1. Setup master/replica (wal_level=logical, hot_standby_feedback=on, etc)
> 2. Initialize the master instance: "pgbench -i -s10 on master"
> 3. Terminal1: execute "pgbench -c20 -T 2000"
> 4. Terminal2: create the logical replication slot:
>
> 271480 (replica) fabrizio=# select * from 
> pg_create_logical_replication_slot('test_logical', 'test_decoding');
> -[ RECORD 1 ]-----------
> slot_name | test_logical
> lsn       | 1/C7C59E0
>
> Time: 37658.725 ms (00:37.659)
>
>
> Even with activity on primary the creation of the logical replication 
> slot took ~38s. Can we do something related to it or should we need to 
> clarify even more the documentation?
>
For the logical slot creation on the standby, as we can not do WAL 
writes, we have to wait for xl_running_xact to be logged on the primary 
and be replayed on the standby.

So we are somehow dependent on the checkpoints on the primary and 
LOG_SNAPSHOT_INTERVAL_MS.

If we want to get rid of this, what i could think of is the standby 
having to ask the primary to log a standby snapshot (until we get one we 
are happy with).

Or, we may just want to mention in the doc:

+     For a logical slot to be created, it builds a historic snapshot, 
for which
+     information of all the currently running transactions is essential. On
+     primary, this information is available, but on standby, this 
information
+     has to be obtained from primary. So, creating a logical slot on 
standby
+     may take a noticeable time.

Instead of:

+     For a logical slot to be created, it builds a historic snapshot, 
for which
+     information of all the currently running transactions is essential. On
+     primary, this information is available, but on standby, this 
information
+     has to be obtained from primary. So, slot creation may wait for some
+     activity to happen on the primary. If the primary is idle, creating a
+     logical slot on standby may take a noticeable time.

What do you think?

Thanks

Bertrand