Horton

Re: Minimal logical decoding on standbys

Amit Khandekar <amitdkhan.pg@gmail.com>

View thread
From: Amit Khandekar <amitdkhan.pg@gmail.com>
To: tushar <tushar.ahuja@enterprisedb.com>
Cc: Petr Jelinek <petr.jelinek@2ndquadrant.com>, Andres Freund <andres@anarazel.de>, Robert Haas <robertmhaas@gmail.com>, Craig Ringer <craig@2ndquadrant.com>, Petr Jelinek <petr@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2019-06-20T09:58:10Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Reduce the log level in 035_standby_logical_decoding.pl.

  2. 035_standby_logical_decoding: Add missing waits for replication

  3. For cascading replication, wake physical and logical walsenders separately

  4. Handle logical slot conflicts on standby

  5. Support invalidating replication slots due to horizon and wal_level

  6. Prevent use of invalidated logical slot in CreateDecodingContext()

  7. Replace replication slot's invalidated_at LSN with an enum

  8. Pass down table relation into more index relation functions

  9. Assert only valid flag bits are passed to visibilitymap_set()

  10. Remove unused _bt_delitems_delete() argument.

  11. Add xl_btree_delete optimization.

Attachments

I am yet to work on Andres's latest detailed review comments, but I
thought before that, I should submit a patch for the below reported
issue because I was almost ready with the fix. Now I will start to
work on Andres's comments, for which I will reply separately.

On Fri, 1 Mar 2019 at 13:33, tushar <tushar.ahuja@enterprisedb.com> wrote:
>
> Hi,
>
> While testing  this feature  found that - if lots of insert happened on
> the master cluster then pg_recvlogical is not showing the DATA
> information  on logical replication slot which created on SLAVE.
>
> Please refer this scenario -
>
> 1)
> Create a Master cluster with wal_level=logcal and create logical
> replication slot -
>   SELECT * FROM pg_create_logical_replication_slot('master_slot',
> 'test_decoding');
>
> 2)
> Create a Standby  cluster using pg_basebackup ( ./pg_basebackup -D
> slave/ -v -R)  and create logical replication slot -
> SELECT * FROM pg_create_logical_replication_slot('standby_slot',
> 'test_decoding');
>
> 3)
> X terminal - start  pg_recvlogical  , provide port=5555 ( slave
> cluster)  and specify slot=standby_slot
> ./pg_recvlogical -d postgres  -p 5555 -s 1 -F 1  -v --slot=standby_slot
> --start -f -
>
> Y terminal - start  pg_recvlogical  , provide port=5432 ( master
> cluster)  and specify slot=master_slot
> ./pg_recvlogical -d postgres  -p 5432 -s 1 -F 1  -v --slot=master_slot
> --start -f -
>
> Z terminal - run pg_bench  against Master cluster ( ./pg_bench -i -s 10
> postgres)
>
> Able to see DATA information on Y terminal  but not on X.
>
> but same able to see by firing this below query on SLAVE cluster -
>
> SELECT * FROM pg_logical_slot_get_changes('standby_slot', NULL, NULL);
>
> Is it expected ?

Actually it shows up records after quite a long time. In general,
walsender on standby is sending each record after significant time (1
sec), and pg_recvlogical shows all the inserted records only after the
commit, so for huge inserts, it looks like it is hanging forever.

In XLogSendLogical(), GetFlushRecPtr() was used to get the flushed
point. On standby, GetFlushRecPtr() does not give a valid value, so it
was wrongly determined that the sent record is beyond flush point, as
a result of which, WalSndCaughtUp was set to true, causing
WalSndLoop() to sleep for some duration after every record. This is
why pg_recvlogical appears to be hanging forever in case of huge
number of rows inserted.

Fix : Use GetStandbyFlushRecPtr() if am_cascading_walsender.
Attached patch v8.


-- 
Thanks,
-Amit Khandekar
EnterpriseDB Corporation
The Postgres Database Company