Re: Minimal logical decoding on standbys

bdrouvotAWS <bdrouvot@amazon.com>

From: "Drouvot, Bertrand" <bdrouvot@amazon.com>

To: Andres Freund <andres@anarazel.de>

Cc: Ibrar Ahmed <ibrar.ahmad@gmail.com>, Amit Khandekar <amitdkhan.pg@gmail.com>, <fabriziomello@gmail.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, tushar <tushar.ahuja@enterprisedb.com>, "[pgdg] Robert Haas" <robertmhaas@gmail.com>, Rahila Syed <rahila.syed@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>

Date: 2021-09-08T10:08:43Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Reduce the log level in 035_standby_logical_decoding.pl.
- 3034dc56ef4b 16.0 landed
035_standby_logical_decoding: Add missing waits for replication
- 57411c82ce86 16.0 landed
For cascading replication, wake physical and logical walsenders separately
- e101dfac3a53 16.0 landed
Handle logical slot conflicts on standby
- 26669757b6a7 16.0 landed
Support invalidating replication slots due to horizon and wal_level
- be87200efd93 16.0 landed
Prevent use of invalidated logical slot in CreateDecodingContext()
- 4397abd0a2af 16.0 landed
Replace replication slot's invalidated_at LSN with an enum
- 15f8203a5975 16.0 landed
Pass down table relation into more index relation functions
- 61b313e47eb9 16.0 landed
Assert only valid flag bits are passed to visibilitymap_set()
- a88a18b1250b 16.0 landed
Remove unused _bt_delitems_delete() argument.
- dc43492e46c7 14.0 cited
Add xl_btree_delete optimization.
- d2e5e20e5711 13.0 cited

Attachments

v23-0006-Logical-Decoding-On-Standby-WalSender-Corner-Case.patch (text/plain) patch v23-0006

Hi Andres,

On 8/6/21 1:27 PM, Drouvot, Bertrand wrote:
> Hi,
>
> On 8/2/21 6:01 PM, Andres Freund wrote:
>> While working on this I found a, somewhat substantial, issue:
>>> If so, do you already have in mind a way to handle this? (I thought you
>>> already had in mind a way to handle it so the question)
>> Yes. I think we need to add a condition variable to be able to wait for
>> WAL positions to change. Either multiple condition variables (one for
>> the flush position, one for the replay position), or one that just
>> changes more often. That way one can wait for apply without a race
>> condition.
>>
> Thanks for the feedback.
>
> Wouldn't a condition variable on the replay position be enough? I 
> don't get why the proposed one on the flush position is needed.

Please find enclosed a patch proposal to address those corner cases.

I think (but may be wrong) that the condition variable on the flush 
position would be needed only for the walsender(s) on non Standby node, 
that's why:

  * I made use of a condition variable on the replay position only.

  * The walsender waits on it in WalSndWaitForWal() only if recovery is
    in progress.

For simplicity to discuss those corner cases, this is a dedicated patch 
that can be applied on top of v23 patches shared previously.

Thanks

Bertrand

>
>>> if not, what kind of additional  tests would you like to see?
>> A few catalog rows being removed (e.g. due to DELETE and then VACUUM
>> *without* full) and a standby without hot_standby_feedback catching
>> that.
>
> Test added in v23 attached.
>
> Thanks
>
> Bertrand
>