Re: [HACKERS] Restricting maximum keep segments by repslots

Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>

From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: pgsql-hackers@lists.postgresql.org
Cc: thomas.munro@enterprisedb.com, sk@zsrv.org, michael.paquier@gmail.com, andres@anarazel.de, peter.eisentraut@2ndquadrant.com
Date: 2018-06-26T07:26:59Z
Lists: pgsql-hackers

Attachments

Hello. This is the reabased version of slot-limit feature.

This patch limits maximum WAL segments to be kept by replication
slots. Replication slot is useful to avoid desync with replicas
after temporary disconnection but it is dangerous when some of
replicas are lost. The WAL space can be exhausted and server can
PANIC in the worst case. This can prevent the worst case having a
benefit from replication slots using a new GUC variable
max_slot_wal_keep_size.

This is a feature mentioned in the documentation.

https://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION-SLOTS

> In lieu of using replication slots, it is possible to prevent the
> removal of old WAL segments using wal_keep_segments, or by
> storing the segments in an archive using
> archive_command. However, these methods often result in retaining
> more WAL segments than required, whereas replication slots retain
> only the number of segments known to be needed. An advantage of
> these methods is that they bound the space requirement for
> pg_wal; there is currently no way to do this using replication
> slots.

The previous patche files doesn't have version number so I let
the attached latest version be v2.


v2-0001-Add-WAL-releaf-vent-for-replication-slots.patch
  The body of the limiting feature

v2-0002-Add-monitoring-aid-for-max_replication_slots.patch
  Shows the status of WAL rataining in pg_replication_slot view

v2-0003-TAP-test-for-the-slot-limit-feature.patch
  TAP test for this feature

v2-0004-Documentation-for-slot-limit-feature.patch
  Documentation, as the name.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Save slot's restart_lsn when invalidated due to size

  2. Fix checkpoint signalling

  3. Check slot->restart_lsn validity in a few more places

  4. Allow users to limit storage reserved by replication slots

  5. Remove header noise from test_decoding test

  6. Rework WAL-reading supporting structs

  7. Flip argument order in XLogSegNoOffsetToRecPtr