Horton

Re: Minimal logical decoding on standbys

Bertrand Drouvot <bertranddrouvot.pg@gmail.com>

View thread
From: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>, Thomas Munro <thomas.munro@gmail.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, Ibrar Ahmed <ibrar.ahmad@gmail.com>, Amit Khandekar <amitdkhan.pg@gmail.com>, fabriziomello@gmail.com, tushar <tushar.ahuja@enterprisedb.com>, Rahila Syed <rahila.syed@2ndquadrant.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2023-01-26T17:56:10Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Reduce the log level in 035_standby_logical_decoding.pl.

  2. 035_standby_logical_decoding: Add missing waits for replication

  3. For cascading replication, wake physical and logical walsenders separately

  4. Handle logical slot conflicts on standby

  5. Support invalidating replication slots due to horizon and wal_level

  6. Prevent use of invalidated logical slot in CreateDecodingContext()

  7. Replace replication slot's invalidated_at LSN with an enum

  8. Pass down table relation into more index relation functions

  9. Assert only valid flag bits are passed to visibilitymap_set()

  10. Remove unused _bt_delitems_delete() argument.

  11. Add xl_btree_delete optimization.

Attachments

Hi,

On 1/24/23 3:31 PM, Drouvot, Bertrand wrote:
> Hi,
> 
> On 1/24/23 6:20 AM, Drouvot, Bertrand wrote:
>> Hi,
>>
>> On 1/24/23 1:46 AM, Andres Freund wrote:
>>> Hi,
>>>
>>> On 2023-01-19 10:43:27 +0100, Drouvot, Bertrand wrote:
>>>> Sure, so with:
>>>>
>>>> 1) hot_standby_feedback set to off on the standby
>>>> 2) create 2 logical replication slots on the standby and activate one
>>>> 3) Invalidate the logical slots on the standby with VACUUM FULL on the primary
>>>> 4) change hot_standby_feedback to on on the standby
>>>>
>>>> If:
>>>>
>>>> 5) pg_reload_conf() on the standby, then on the primary we get a catalog_xmin
>>>> for the physical slot that the standby is attached to:
>>>>
>>>> postgres=# select slot_type,xmin,catalog_xmin  from pg_replication_slots ;
>>>>   slot_type | xmin | catalog_xmin
>>>> -----------+------+--------------
>>>>   physical  |  822 |          748
>>>> (1 row)
>>>
>>> How long did you wait for this to change? 
>>
>> Almost instantaneous after pg_reload_conf() on the standby.
>>
>>> I don't think there's anything right
>>> now that'd force a new hot-standby-feedback message to be sent to the primary,
>>> after slots got invalidated.
>>>
>>> I suspect that if you terminated the walsender connection on the primary,
>>> you'd not see it anymore either?
>>>
>>
>> Still there after the standby is shutdown but disappears when the standby is re-started.
>>
>>> If that isn't it, something is broken in InvalidateObsolete...
>>>
> 
> Yeah, you are right: ReplicationSlotsComputeRequiredXmin() is missing for the
> logical slot invalidation case (and ReplicationSlotsComputeRequiredXmin() also
> needs to take care of it).
> 
> I'll provide a fix in the next revision along with the TAP tests comments addressed.
> 

Please find attached V43 addressing the comments related to the TAP tests (in 0004 at that time) that have been done in [1] .

Remarks:

- The C helper function to call LogStandbySnapshot() is not done yet.
- While working on it, I discovered that the new isCatalogRel field was not populated in gistXLogDelete(): fixed in V43.
- The issue described above is also fixed so that a standby restart or a reload would produce the same behavior
on the primary physical slot (aka catalog_xmin is empty if logical slots are invalidated).
- A test with pg_recvlogical started before the standby promotion has been added.
- A test for conflict due to row removal via on-access pruning has been added.
- I'm struggling to create a test for btree killtuples as there is a need for rows removal on the table (that could produce a conflict too):
Do you've a scenario in mind for this one? (and btw in what kind of WAL record should the conflict be detected in such a case? xl_btree_delete?)

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

[1]: https://www.postgresql.org/message-id/20230106034036.2m4qnn7ep7b5ipet%40awork3.anarazel.de